Venue: NAACL 2021

# Why Do Document-Level Polarity Classifiers Fail?

Karen S. Martins

CS Department

UFMG

Belo Horizonte, MG, Brazil

karensm@dcc.ufmg.br &Pedro O.S. Vaz-de-Melo

CS Department

UFMG

Belo Horizonte, MG, Brazil

olmo@dcc.ufmg.br &Rodrygo L. T. Santos

CS Department

UFMG

Belo Horizonte, MG, Brazil

rodrygo@dcc.ufmg.br

###### Abstract

Machine learning solutions are often criticized for the lack of explanation of their successes and failures. Understanding which instances are misclassified and why is essential to improve the learning process. This work helps to fill this gap by proposing a methodology to characterize, quantify and measure the impact of _hard instances_ in the task of polarity classification of movie reviews. We characterize such instances into two categories: _neutrality_, where the text does not convey a clear polarity, and _discrepancy_, where the polarity of the text is the opposite of its true rating. We quantify the number of _hard instances_ in polarity classification of movie reviews and provide empirical evidence about the need to pay attention to such problematic instances, as they are much harder to classify, for both machine and human classifiers. To the best of our knowledge, this is the first systematic analysis of the impact of _hard instances_ in polarity detection from well-formed textual reviews.

+
Footnote †: 2021 Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1782–1794

June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Document-level polarity classification is the task of classifying the polarity of a whole opinionated message Pozzi et al. (2016). For instance, given a movie review, the system determines whether the review text expresses an overall positive, negative, or neutral opinion about the movie. Although polarity classification naturally suits to analyze consumer opinions about products and services Gui et al. (2017), it is also well suited to various types of applications, such as to infer votes in elections Goldberg et al. (2007), civilian sentiment during terrorism scenarios Cheong and Lee (2011), citizens' perception of government agencies Arunachalam and Sarkar (2013) and recommendation systems Zhang (2015).

Supervised machine learning is one of the most common and successful approaches for polarity classification, but even state-of-the-art methods fail to correctly classify a substantial portion of the instances, from \(10\%\) to \(20\%\), depending on the dataset Ribeiro et al. (2016). The problem with this approach is that if the data is not representative and reliable, the model is unlikely to perform well. One source of unreliability is data noise, which can be categorized into _class noise_ and _attribute noise_Gupta and Gupta (2019). Class noise occurs when the training data contains instances that are wrongly labeled. Attribute noise occurs when the training data contains one or more attributes with wrong, incomplete or missing values. In the case of textual data, such noise usually comes in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, and abbreviations Agarwal et al. (2007); Michel and Neubig (2018); Lourentzou et al. (2019). Nevertheless, for both cases, the noise can be eliminated from the data by correcting the labels (for class noise) or the problematic text (for attribute noise).

A more problematic source of data unreliability in polarity classification tasks comes from well written text that, for some reason, does not convey its class clearly. Literature calls such instances _hard instances_, which are those that are intrinsically hard to correctly label or classify Smith and Martinez (2011); Beigman Klebanov and Beigman (2014). Differently from noisy instances, _hard instances_ cannot be corrected, so the only solution is to identify and remove them from the training data. Also, _hard instances_ are not equivalent to outliers, as they do not differ significantly from other observations and may represent a significant portion of the data Smith et al. (2014). For example, in a polarity classification task, a positive movie review that describes at least as many negative as positive points of the film can be a _hard instance_. To the best of our knowledge, no study exists that characterizes such instances and quantifies their impact on document-level polarity classification tasks.

Thus, we propose a methodology to characterize,

quantify and measure the impact of _hard instances_ in polarity classification tasks and demonstrate its usefulness in the task of movie review polarity classification. To this end, we collected \(415,867\)_positive_ and _negative_ movie reviews from Metacritic. One advantage of Metacritic is that the meaning of ratings is clearly stated to the users when a review is being submitted: positive ratings range between \(61\%\) and \(100\%\), neutral range between \(40\%\) and \(60\%\), and negative between \(0\%\) and \(39\%\). Because of that, class noise and biases should be rare, that is, a user who liked (disliked) a movie will very unlikely give a negative (positive) rating to it. Thus, classification errors will mostly be due to _hard instances_, which we assign into two disjoint categories: _neutral_ and _discrepant_. A _neutral_ review does not have a clear polarity and a _discrepant_ review has a human-perceived polarity that is different from its associated rating. This categorization is complete, i.e., every instance that, for a human, does not reveal its class clearly falls into one (and only one) of these two types of _hard instances_.

_Neutral_ and _discrepant_ reviews are characterized by a well-defined human classifier that uses human reasoning to infer the class of the example. When the class assigned by the human classifier is incorrect, we label the review as _discrepant_, i.e., the human-perceived polarity of the text is different from its associated rating. When the human classifier is not confident about its prediction, we label the review as _neutral_. We labeled \(1,200\) reviews and found \(198\)_neutral_ and \(64\)_discrepant_ reviews. We tested state-of-the-art machine classifiers on these reviews and results revealed that _hard instances_ can significantly decrease their performances. In short, the main contributions are:

* A simple and reproducible methodology based on a well-defined human classifier to characterize and identify _hard instances_ on polarity classification tasks (Section 3);
* A thorough analysis of the impact of _hard instances_ in the task of movie review polarity classification (Section 5.2);
* Publicly available datasets of movie reviews describing the expected amounts of five classes of _hard instances_ (Section 5.1).

As an additional contribution, we show how far are state-of-the-art machine classifiers from human performance in the task of movie review polarity classification.

## 2 Related Work

In supervised machine learning, class and attribute noise can increase learning complexity and, consequently, reduce classification accuracy Zhu and Wu (2004). Class noise is considered to be more harmful than attribute noise Frenay and Verleysen (2014), but it is easier to detect Van Hulse et al. (2007). Thus, class noise is more often addressed in the literature Gupta and Gupta (2019), where several studies analyzed its impact in classification tasks and how to address it Natarajan et al. (2013); Hendrycks et al. (2018); Liu et al. (2017); Rehbein and Ruppenhofer (2017); Jindal et al. (2019). In NLP, attribute noise are unintended errors in text, which can come from failures in automatic character recognition processes Vinciarelli (2005) or naturally while writing the text in the form of errors in language rules, such as typos, grammatical errors, improper punctuation, irrational capitalization and abbreviations Agarwal et al. (2007); Contractor et al. (2010); Dey and Haque (2009); Florian et al. (2010); Michel and Neubig (2018). In short, noise are unintentional and undesirable errors in the text that can (and should) be eliminated from the data.

Conversely, _hard instances_ are noise-free and cannot be corrected, only eliminated from the data Smith and Martinez (2011). In addition, they differ from outliers because their feature representation vectors may be similar to others from regular instances Smith and Martinez (2011). Nevertheless, _hard instances_ are more prone to class noise. In fact, Beigman Klebanov and Beigman (2009) defined _hard instances_ in the context of label annotations, under the assumption that items that are easy are reliably annotated, whereas items that are hard display confusion and disagreement among the annotators. Later, Beigman Klebanov and Beigman (2014) showed that the presence of _hard instances_ in the training data misleads the machine learner on easy, clear-cut cases. The definition of Smith et al. (2014) is similar to ours: _hard instances_ are simply those that "should be misclassified" by machine learning methods. The authors introduced _hardness measures_ based on the outputs of an ensemble of classifiers to identify such instances and showed that classifiers are often uncertain about their classes. Following the same idea, Krymolowski (2002) argues that easy instances are correctly classified by all or most classifiers. On the other hand, _hard instances_ are missed by most of them 

In this work, we propose a _human classifier_ composed by human annotators to identify _hard instances_. Our definition unifies the ones of Beigman Klebanov and Beigman (2009, 2014) and Smith et al. (2014). Similarly to Beigman Klebanov and Beigman (2009, 2014), we define _hard instances_ as those in which the _human classifier_ is uncertain or wrong about their true labels. However, different from these studies, which quantify the impact _hard instances_ have on training, our goal is to provide a methodology to quantify the expected amount of _hard instances_ in data and the impact they have on classifiers in production and testing. Also, and similarly to Smith et al. (2014), _hard instances_ are divided into "instances that should be misclassified", which we call _discrepant_, and "border points", which we call _neutral_. To the best of our knowledge, we are the first to propose a methodology to characterize and quantify the impact of _hard instances_ in unstructured textual data for polarity classification tasks.

Regarding the effect of _hard instances_ in sentiment and polarity classification tasks, Bermingham and Smeaton (2010) showed that it is easier to classify sentiment in short documents (e.g. tweets) than in longer ones, as short documents have less non-relevant information. also, Valdivia et al. (2019) showed that ratings in TripAdvisor reviews are not strongly correlated with sentiment scores given by sentiment analysis methods and proposed a unified index that aggregates both polarities. Barnes et al. (2019) collected a subset of sentences that an ensemble of state-of-the-art sentiment classifiers misclassified and annotated them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, among others. In our work, we analyze manually identified _hard instances_ (as opposed to instances misclassified by a machine classifier). As a result, compared to these works, we have a more precise (e.g., a misclassified instance is not necessarily hard) and complete (e.g., not all _hard instances_ are misclassified) ground-truth.

## 3 Methodology

Problem Setting.In this work, we focus on the problem of polarity detection of movie reviews, but all the methods can be applied to any document-level polarity classification task. More formally, in a dataset \({\cal D}=(X,Y)\) composed by a set of textual movie reviews \(X\) and their corresponding binary ratings \(Y\), each review \(x_{i}\in X\) is associated with a score (or rating) \(y_{i}\in Y\) that can be either (_positive_) or \(1\) (negative). For the aims of this paper, it is important that \({\cal D}\) does not contain any movie reviews that have been explicitly associated with a neutral score by their author, e.g. a score of \(50\) on _Metacritic_. By doing this, we isolate _hard instances_ from explicit neutral reviews, avoiding class noise and biases.

Our methodology is composed by a human classifier \(f_{H}\), which identifies _hard instances_, and a machine classifier \(f_{M}\), which is tested on _hard_ and _regular_ instances. A classifier is defined as a function \(f(x_{i})\) that receives a textual movie review \(x_{i}\) as input and returns its polarity \(\hat{y}_{i}\in\{0,1\}\). We use the human classifier to assign a label \(l_{i}\) to a large sample of movie reviews \(x_{i}\) to indicate whether \(x_{i}\) is a _hard instance_ or not. This label can be one (and only one) of a set \(L\) of manually defined labels that indicate that the instance is _regular_ or a type of _hard instance_. With that, we will be able to quantify the impact of _hard instances_ on machine classifiers and provide explanations about why they occur and how to avoid them in order to improve machine classifiers' accuracy. More specifically, for a machine classifier \(f_{M}\) and for all labels \(l\in L\), _regular_ included, we will calculate the probabilities \(P(l_{i}=l|y_{i}\neq\hat{y}_{i})\) and \(P(y_{i}=\hat{y}_{i}|l_{i}=l)\).

Types of Hard Instances.A strong premise of this work is that the dataset \({\cal D}\) has no (or negligible) class noise, i.e., all polarity scores \(y_{i}\in Y\) reflect the real opinion of the reviewer. To guarantee that, one needs to construct \({\cal D}\) using movie reviews from systems like _Metacritic_ or _Rotten Tomatoes_, which have well defined meanings for the scores, which are always visible to the reviewers. Thus, every time the polarity of text \(x_{i}\) is inconsistent with its true score \(y_{i}\), we assume that \(x_{i}\) is a _hard instance_. More specifically, we define two possible hypotheses explaining the hardness of the text \(x_{i}\), i.e., two disjoint types of _hard instances_: (1) the text does not have a clear polarity, namely _neutrality_, and (2) the text has a clear polarity, but its score \(y_{i}\) is the opposite one, namely _discrepancy_.

A movie review \(x_{i}\) is a _hard instance_ of type _neutrality_ when its polarity is not clear. We define three labels for _neutral hard instances_: _mixed_ (text has mixed opinions), _factual_ (text is purely factual) and _contextual_ (polarity needs context). The _mixed_ label considers reviews that describes both positive and negative points about the movie without having the overall opinion clearly stated. One real exam 

ple is: _"as dumb as the film is, the actors escape relatively unscathed."_ The _factual_ label defines non-opinionated reviews that describes only facts about the movie, such as: _"it is a movie about the World War II and its consequences on the lives of those who survived."_ The label _contextual_ characterizes reviews where context is needed to understand its polarity, including those containing irony and sarcasm. One real example is: _"ultimately, Collin's film is one of forgiveness and that's not the usual way great tragedies end."_ Finally, the label _hard_undefined_ is given to reviews where the reasons for the lack of polarity are not clear.

The second type of _hard instance_, namely _discrepancy_, is given to reviews where the polarity of its text \(x_{i}\) is the opposite of the polarity of its score \(y_{i}\). For this type, we define a single label: _discrepant_ (polarity of text and score are discrepant). As an example, consider a highly acclaimed movie of a prestigious director, such as Martin Scorsese. Now, consider a reviewer who liked this movie, but unlike the vast majority of critics, found many points that prevent her from giving it a perfect score. Thus, the text will mostly be about its negative points to justify why she is not giving the expected perfect score. Consequently, the text review will appear negative although the score is positive. The following textual review has a clear negative polarity although its score \(y_{i}\) is positive: _"Thoroughly predictable from start to finish."_ For more examples, see Tables 4 and 5 in the Appendix.

Human Classifier.A fundamental building block of our methodology is the human classifier \(f_{H}\). Human classifiers are often considered to be the upper bound in terms of performance of classification tasks Stallkamp et al. (2012); Ciresan et al. (2012); Geirhos et al. (2018), which means that when it makes a prediction error, machine classifiers will most likely also miss. Moreover, when a human classifier working on its full capacity makes a mistake, and the class label is correct (i.e. no class noise), then what caused the error is most likely a _hard instance_Beigman Klebanov and Beigman (2014). We use this premise to define the two types of _hard instances_ discussed in the previous section.

In the task of polarity classification of movie reviews, a human classifier mistake can be due to two causes: **(C1)** the text of the review \(x_{i}\) is not clear about its polarity \(y_{i}\), or **(C2)** the score \(y_{i}\) is different from the (clearly) perceived polarity of \(x_{i}\). In other words, the human classifier \(f_{H}\) can be characterized by two binary features when executing this task: whether it is confident about its prediction **(F1)** and whether it correctly classified the polarity of the review \(x_{i}\) **(F2)**. Thus, when it makes a mistake, if it was not confident, an error of type **C1** occurs, and when it was confident, an error of type **C2** occurs. The first one **(C1)** is associated with a _hard instance_ of type _neutrality_, whereas the second one **(C2)** is associated with a _hard instance_ of type _discrepancy_. Also, while the second only occurs when the human classifier \(f_{H}\) makes a mistake, the first occurs every time \(f_{H}\) is not confident, i.e., it is **independent** of the prediction \(\hat{y}_{i}\).

With the aforementioned rationale, we are ready to propose a well-defined human classifier \(f_{H}\) to identify _hard instances_ in movie reviews. First, and in order to construct a robust classifier, \(f_{H}\) is an ensemble composed by three independent human classifiers \(f_{h1}\), \(f_{h2}\) and \(f_{h3}\). In other words, we will use three annotators to label a movie review \(x_{i}\) in terms of its polarity and hardness1. Each annotator \(j\in\{1,2,3\}\) is asked to classify the reviews in two levels. First, they are asked to make a prediction \(\hat{y}^{j}_{i}\), i.e., to classify the polarity of the review \(x_{i}\) as _positive_ or _negative_. Second, they are asked to indicate whether they are _confident_ or not about their classification \(\hat{y}^{j}_{i}\). We denote the confidence of annotator \(j\) on review \(x_{i}\) by \(c^{j}_{i}\in\{0,1\}\), where \(c^{j}_{i}=1\) if \(j\) is confident and \(c^{j}_{i}=0\) otherwise. If \(c^{j}_{i}=0\), then we assume that \(x_{i}\) does not contain sufficient information for \(j\) to infer its polarity, that is, \(x_{i}\) is a _hard instance_ of type _neutrality_. So, annotator \(j\) is asked to choose one label \(l^{j}_{i}\) that fits best to the _neutrality_ of \(x_{i}\), which can be either _mixed_, _factual_ or _contextual_. On the other hand, if \(c^{j}_{i}=1\), then \(l^{j}_{i}\) is set to _regular_. This process is illustrated in Figure 1. Of course, each annotator \(j\) is independent and cannot see the others' responses.

Footnote 1: In practice, any number of annotators can be used, including just one.

At the end of this process, for each instance \(x_{i}\), we will have three annotation triples \((\hat{y}^{j}_{i},c^{j}_{i},l^{j}_{i})\), where \(\hat{y}^{j}_{i}\in\{0,1\}\) (_positive_ or _negative_), \(c^{j}_{i}\in\{0,1\}\) (_not confident_ or _confident_) and \(l^{j}_{i}\in\mathcal{L}_{N}=\{\text{ {mixed}, factual, contextual, regular }\}\). Assuming that all annotators are equally skilled, we aggregate these annotations using majority voting to set the outputs of our human classifier \(f_{H}\). For the polarity \(\hat{y}_{i}\) and the confidence \(c_{i}\), the aggregation is straightforward, as described in Equations 1 and 2: 

\[\hat{y}_{i}=\begin{cases}0&\textbf{if }\sum_{j=1}^{3}\hat{y}_{i}^{j}\leq 1\\ 1,&\text{otherwise},\end{cases} \tag{1}\]

\[c_{i}=\begin{cases}0&\textbf{if }\sum_{j=1}^{3}c_{i}^{j}\leq 1\\ 1,&\text{otherwise}.\end{cases} \tag{2}\]

Setting the final _hard instance_ label \(l_{i}\) of review \(x_{i}\) is more involved. Let \(\mathcal{L}_{i}=[l_{i}^{1},l_{i}^{2},l_{i}^{3}]\) be the **list** of labels \(l_{i}^{j}\) given by the annotators to review \(x_{i}\) (e.g. \(\mathcal{L}_{1}=\textit{[mixed]}\), _mixed_, _regular_\(]\)) and \(N(l,\mathcal{L}_{i})\) the number of elements of \(\mathcal{L}_{i}\) that are equal to label \(l\) (e.g. \(N(\textit{mixed},\mathcal{L}_{1})=2\)). Then, \(l_{i}\) is the majority vote if at least two annotators (the majority) gave that label to \(x_{i}\) and, if not, \(l_{i}\) is set to _hard_undefined_, indicating no consensus. This process is formally described by Equation 3:

\[l_{i}=\begin{cases}\operatorname*{arg\,max}_{l\in\mathcal{L}_{N}}N(l, \mathcal{L}_{i})&\textbf{if }N(l,\mathcal{L}_{i})\geq 2\\ \textit{hard\_undefined},&\text{otherwise}.\end{cases} \tag{3}\]

Finally, when the human classifier is confident about its classification of \(x_{i}\) (\(c_{i}=1\)), but it makes a mistake (\(\hat{y}_{i}\neq y_{i}\)), we update the label \(l_{i}\) of \(x_{i}\) to _discrepant_. It is easy to see that this update step will be executed only if \(l_{i}\) was previously set to _regular_, i.e., it will not overwrite a _neutrality_ label. Equation 4 defines the _discrepancy_ update step:

\[l_{i}=\textit{discrepant}~{}~{}\textbf{if }\hat{y}_{i}\neq y_{i}~{}~{}\textbf{and }c_{i}=1. \tag{4}\]

## 4 Experimental Setup

Data Set.We collected movie reviews from Metacritic,2 which can be authored by _regular users_ and _experts_, i.e., people working in the movie industry or important communication channels (e.g. _The New York Times_). In case of _experts_, the review provided by Metacritic is actually a short summary of the original review and, as we show in Section 5, this can be a problem for polarity classifiers. Also, each _experts_ review is associated with a score ranging from \(0\) to \(100\), where scores from \(0\) to \(39\) are _negative_, from \(40\) to \(60\) are _neutral_, and from \(61\) to \(100\) are _positive_. Differently, _regular users_ reviews are produced by any person that has an account and are associated with a score ranging from \(0\) to \(10\), where scores between \(0\) and \(3\) are _negative_, between \(4\) and \(6\) are _neutral_, and over \(7\) are _positive_. As previously mentioned, the meaning of each rating is clearly conveyed to users in the Metacritic website. Thus, class noise and biases should be rare in the dataset.

Footnote 2: [https://www.metacritic.com/movie](https://www.metacritic.com/movie)

In total, we collected \(415,867\) reviews for \(8,170\) different movies, where \(227,348\) of those are from _regular users_ and \(188,519\) from _experts_. Our data collection was executed using the following steps. First, we collected the most popular _experts_ from the website, as provided by Metacritic. Then, we generated a list of all movies reviewed by the top \(10\) experts. From this list, which contains \(8,170\) movies, we collected all reviews from _experts_ and _regular users_ that were posted until August, \(2018\). For the purpose of this work, we avoided reviews that do not have a clear polarity (_neutral_ reviews), i.e., we only considered _positive_ and _negative_ reviews. Hence, we selected a clean and unambiguous dataset. Reviews from _experts_ are usually shorter than from _regular users_, containing an average of \(26\) words (std. dev. of \(13\)) against an average of \(100\) words (std. dev. of \(129\)) for reviews by _regular users_. In addition, we observed that _experts_ use a more elaborate language. Because of these differences, we will condition our analyses on the type of user (_experts_ or _regular users_) and score polarity (_positive_ or _negative_).

Machine Classifiers.To evaluate the impact of _hard instances_ on machine classifiers, we selected three state-of-the-art models with reported success in the task of polarity detection of movie reviews: BERT Devlin et al. (2019), CNN-GRU Wang et al. (2016) and C-LSTM Zhou et al. (2015). C-LSTM utilizes a Convolutional Neural Network (CNN) to extract a sequence of higher-level phrase representations, which are then fed into a Long Short-Term Memory (LSTM) unit to obtain the sentence representation. CNN-GRU connects a character-aware CNN with a character-aware Gated Recurrent Unit (GRU) to learn long sequence semantics. These two networks are initialized with pre-trained Word2vec vectors from Google News

Figure 1: Confidence diagram.

 

Dataset and have their final representations connected to a dense layer. BERT uses a masked language model (MLM) to pre-train deep bidirectional representations from unlabeled text that considers both the left and right context of sentences and words. In this work, we used an architecture composed by BERT embeddings pre-trained with data from Wikipedia connected with a dense layer. For all architectures, the output \(\hat{y}_{i}\) is given by a sigmoid function. For implementation and code details, please see the Appendix.

## 5 Results

### Number of Hard Instances

The first question we need to answer is: how many _hard instances_ exist in movie reviews? In the context of our Metacritic dataset \(\mathcal{D}\), the answer to this question can be influenced by two factors: (1) the type of user and (2) the polarity of their rating. Thus, the following results are conditioned on whether the authors are _experts_ or _regular users_ and whether the reviews are _positive_ or _negative_. Because of that, we sampled a collection \(D_{H}\) of \(800\) movie reviews from \(\mathcal{D}\) that is both balanced in terms of user type and score polarity, i.e., this collection has \(200\) reviews for each of the four combinations of user type and score polarity.

In order to quantify the number of _hard instances_ in \(D_{H}\), we use our proposed human classifier \(f_{H}\) described in Section 3 to label every review \(x_{i}\in D_{H}\). Recall that \(f_{H}\) assigns a polarity \(\hat{y}_{i}\in\) {_positive_, _negative_ } to \(x_{i}\) and, more important to our purpose here, a label \(l_{i}\), which can be either _regular_ (instance is not a _hard instance_), _discrepant_ (the polarity of the text is different from the score polarity), or one of the four _neutrality_ labels: _mixed_ (text has mixed opinions), _factual_ (text is purely factual), _contextual_ (polarity needs context) and _hard_undefined_ (reasons are unclear). Also, let \(u_{i}\in\) {_expert_, _regular user_ } be the user type of the author of review \(x_{i}\). Our goal with the following results is to estimate the probability \(P(l_{i}=l~{}|~{}y_{i}=y,u_{i}=u)\) for the four combinations of score polarity \(y\) and user type \(u\).

In Table 1, we show the number and proportion of movie reviews that are or are not _hard instances_ for _experts_. From the \(400\) labeled reviews, almost one quarter (\(92\)) are _hard instances_. From those, note that _neutral_ reviews are more common than _discrepant_ ones, but while the first is equally present in both _positive_ and _negative_ reviews, _discrepant_ instances are significantly more present in _positive_ reviews. In such cases, the author gave a positive score to the movie, but its review demonstrates the opposite sentiment. This often occurs when the _expert_ is using the review to justify a good, but far from perfect score, to a critically acclaimed movie. As for the _neutral_ reviews, the most predominant type is _contextual_ (\(6.8\%\)), followed by _mixed_ (\(4.3\%\)) and _factual_ (\(4.3\%\)). Also, _contextual_ instances are more common in _negative_ reviews, when _experts_ often use figures of speech (e.g. irony) together with external knowledge to create humour. Finally, _factual_ instances are more present in _positive_ reviews, where the _experts_ simply describe some characteristic of the movie that impressed them without explicitly saying that.

Also, in Table 1 we show the number and proportion of movie reviews that are or are not _hard instances_ for _regular users_. First, note that the number of reviews that are _hard instances_ significantly decreased in comparison with the ones written by _experts_. From the \(400\) labeled reviews, only \(36(9\%)\) are _hard instances_, of which \(31\) are _neutral_ and only \(5\) are _discrepant_. Different from what was verified for _experts_, the most predominant label for _regular users_ was _mixed_, which occurred significantly more in _positive_ reviews. For the other labels, their occurrences were fairly balanced between _negative_ and _positive_ reviews. We observed that _regular users_ use a much more direct and simple language to state their opinions than _experts_. Because of that, most of the _hard instances_ are concentrated in cases where the author lists both the negative and positive aspects of the movie without stating their final opinions about the movie, which

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**label (\(l_{i}\))** & _positive_ & _negative_ & **total** \\ \hline \hline \multicolumn{4}{|c|}{_experts_} \\ \hline _regular_ & \(146(36.5\%)\) & \(162(40.5\%)\) & \(\mathbf{77}\%\) \\ \hline _discrepant_ & \(20(5\%)\) & \(3(0.8\%)\) & \(\mathbf{5.8}\%\) \\ \hline _neutral_ & \(34(8.5\%)\) & \(35(8.8\%)\) & \(\mathbf{17.3}\%\) \\ \hline \hline _mixed_ & \(10(2.5\%)\) & \(7(1.8\%)\) & \(\mathbf{4.3}\%\) \\ \hline _factual_ & \(14(3.5\%)\) & \(3(0.8\%)\) & \(\mathbf{4.3}\%\) \\ \hline _contextual_ & \(7(1.8\%)\) & \(20(5\%)\) & \(\mathbf{6.8}\%\) \\ \hline _underlined_ & \(3(0.8\%)\) & \(5(1.3\%)\) & \(\mathbf{2}\%\) \\ \hline _regular users_ & & & \\ \hline _regular_ & \(177(44.3\%)\) & \(187(46.8\%)\) & \(\mathbf{91}\%\) \\ \hline _discrepant_ & \(3(0.8\%)\) & \(2(0.5\%)\) & \(\mathbf{1.3}\%\) \\ \hline _neutral_ & \(20(5\%)\) & \(11(2.8\%)\) & \(\mathbf{7.8}\%\) \\ \hline _mixed_ & \(16(4\%)\) & \(7(1.8\%)\) & \(\mathbf{5.8}\%\) \\ \hline _factual_ & \(1(0.3\%)\) & \(2(0.5\%)\) & \(\mathbf{0.8}\%\) \\ \hline _contextual_ & \(0(0\%)\) & \(1(0.3\%)\) & \(\mathbf{0.3}\%\) \\ \hline _undefined_ & \(3(0.8\%)\) & \(1(0.3\%)\) & \(1\%\) \\ \hline \end{tabular}
\end{table}
Table 1: Number of _hard instances_ in reviews.

 

is the definition of _mixed_.

A note about the human classifier.Because we used three human annotators in \(f_{H}\) and a majority vote function, only two annotators were used initially. The third annotator was called to classify \(x_{i}\) if, and only if, the first two had any kind of disagreement, i.e., a disagreement regarding the polarity \(y_{i}\), the confidence \(c_{i}\), or label \(l_{i}\). For the first two annotators, they agreed on \(91.13\%\) of the polarity scores, on \(90.5\%\) of their confidence levels and on \(88\%\) of their labels. Regarding the third annotator, only \(1.5\%\) of the instances were not in total agreement with at least one of the other annotators. The Cohen's kappa coefficient for the first two annotators was \(0.82\) in relation to polarity scores, \(0.58\) regarding their confidence levels and \(0.49\) regarding their attribute noise labels.

### Impact of Hard Instances

In this section, we quantify the impact of _hard instances_ in machine classifiers. Also, by putting these results in perspective with what was achieved by the human classifier, we hope to provide an accurate assessment on how distant machine classifiers are with respect to human performance. We guide our analyses by the following questions:

1. What are the probabilities of a correct and a misclassification given the label \(l\)? In other words, we want to estimate the probabilities \(P(\hat{y}_{i}=y_{i}~{}|~{}l_{i}=l)\) and \(P(\hat{y}_{i}\neq y_{i}~{}|~{}l_{i}=l)\) for all labels \(l\in L\).
2. What are the probabilities of label \(l\) given that the classifier was correct and that it made a mistake? In other words, we want to estimate the probabilities \(P(l_{i}=l~{}|~{}\hat{y}_{i}\neq y_{i})\) and \(P(l_{i}=l~{}|~{}\hat{y}_{i}=y_{i})\) for all labels \(l\in L\).

To address these questions, we test the three classifiers described in Section 4 in the labeled dataset \(D_{H}\) (see Section 5.1), which contains 800 reviews. Because this dataset is completely balanced, we created two balanced training datasets, one containing solely reviews from _experts_, namely \(D^{experts}_{T}\), and another containing solely reviews from _regular users_, namely \(D^{users}_{T}\). Each dataset contains \(8,796\) reviews, \(4,398\) of each polarity. Again, this dataset is solely used to train the machine classifiers. Because these classifiers are sensitive to initialization parameters, we trained and tested them 5 times and the corresponding error bars are shown in Figure 2. Finally, recall that \(y_{i}\) refers to the author's original polarity score (gold polarity) and \(\hat{y}_{i}\) refers to the polarity predicted by the classifiers, including the human classifier.

Figure 2 shows the classification error (with their respective error bars) for all classifiers in \(D_{H}\). The classification error is simply the proportion of instances that were misclassified. Each bar is also colored according to the labels' proportion in the misclassified instances. For each classifier, the left (right) bar shows the error with respect to _positive_ (_negative_) instances. In general, the human classifier was the one that achieved the smallest error, followed by BERT and C-LSTM. Also, the errors are always higher for _experts_, as these reviews have significantly less words (see Section 4) and more _hard instances_ (see Section 5.1). The latter is also one of the main reasons for the error being almost always higher for _positive_ instances than for _negative_ instances. For _expert_ reviews, while _negative_ instances always have more _regular_ instances, _positive_ instances have almost twice more _hard instances_, particularly _discrepant_ ones. For _regular user_ reviews, _positive_ instances also have more _hard instances_, but the difference in terms of _neutral_ reviews is more significant. Note that, for both user types, this difference in the instances misclassified by the human classifier is striking.

[FIGURE

Figure 2: Classification error for all classifiers.

 

For a more precise assessment of the impact of _hard instances_, we show in Table 2 the accuracy of the classifiers considering instances of each label separately. In other words, these results provide estimates for the probabilities of our first question, \(P(\hat{y}_{i}=y_{i}\mid l_{i}=l)\) and \(P(\hat{y}_{i}\neq y_{i}\mid l_{i}=l)\). First, note that for all classifiers the accuracy significantly degrades in _neutral_ instances and get even worse in _discrepant_ instances. Recall that a _discrepant_ review is a review where the human classifier was sure about its polarity, but the originally assigned polarity is the opposite. Thus, by definition, the human classifier accuracy on _discrepant_ reviews is zero. For _neutral_ instances, the human classifier always outperforms the machine classifiers. However, the machine classifiers are not always tricked by _discrepant_ reviews as the human classifier is, although their performances are not better than a coin toss. Considering the specific _neutral_ labels, note that BERT achieves human level performance for _contextual_, which is coherent with the nature of this classifier, given that its embeddings are supposed to carry much more contextual information in comparison with the embeddings used in C-LSTM and CNN-GRU. The most inconclusive results refer to _hard_undefined_, which is also the label with the least instances, \(12\) out of \(800\).

To answer our second question, related to the probabilities \(P(l_{i}=l\mid\hat{y}_{i}\neq y_{i})\) and \(P(l_{i}=l\mid\hat{y}_{i}=y_{i})\), we sample an additional dataset \(D^{error}_{H}\) to be labeled by our human classifier \(f_{H}\). First, we run the BERT classifier, which was the one that achieved the best results, on two new balanced sets of reviews extracted from \(\mathcal{D}\), one containing \(2,752\) reviews from _experts_ and the other \(2,752\) reviews from _regular users_. Again, we used the same BERT classifiers that were trained for generating the results in Figure 2, one for each user type. After running BERT, we construct \(D^{error}_{H}\) by sampling \(100\) misclassified and \(100\) correctly classified instances authored by each user type, for a total of \(400\) reviews. Then, we run \(f_{H}\) on \(D^{error}_{H}\) to have a more accurate estimate of \(P(l_{i}=l\mid\hat{y}_{i}\neq y_{i})\) and \(P(l_{i}=l\mid\hat{y}_{i}=y_{i})\).

Table 3 shows the percentages of each label for correctly and incorrectly classified instances, which provide estimates for the probabilities of \(P(l_{i}=l\mid\hat{y}_{i}\neq y_{i})\) and \(P(l_{i}=l\mid\hat{y}_{i}=y_{i})\). For both _experts_ and _regular users_, it is much more likely to find _neutral_ and _discrepant_ reviews in misclassified instances. In other words, one easy way to find _hard instances_ in movie reviews is to run BERT and sample from misclassified instances. Our estimates for the probabilities of finding a misclassified _hard instance_ is \(0.64\) for _experts_ and \(0.56\) for _regular users_. In other words, more than \(50\%\) of our sampled misclassified instances are _hard instances_. Recall from Table 1 that we found only \(23\%\) of _hard instances_ in reviews from _experts_ and only \(9\%\) in reviews from _regular users_ in our first balanced sample \(D_{H}\). The most striking difference is for _discrepant_ reviews, where the number of instances increased by one order of magnitude in misclassified instances. Regarding the _neutral_ labels, our results reveal that we are at least twice as likely to find _contextual_ instances in misclassified _expert_ reviews and _mixed_ instances in misclassified _regular users_ reviews. Therefore, to find _hard instances_ with high probability, we propose to train and run BERT in the data (without filtering anything) and, from the misclassified instances, run the human classifier to identify them.

We investigated misclassified _regular_ instances and found two patterns that explain the errors. First, reviews that have positive and negative points, but

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \cline{2-5}  & C-LSTM & CNN-GRU & BERT & Human \\ \hline _regular_ & 0.91 & 0.91 & 0.94 & **1** \\ \hline _discrepant_ & **0.55** & 0.52 & 0.45 & 0 \\ \hline _neutral_ & 0.76 & 0.73 & 0.76 & **0.78** \\ \hline _mixed_ & 0.75 & 0.72 & **0.76** & 0.75 \\ \hline _factual_ & 0.78 & 0.77 & 0.71 & **0.80** \\ \hline _contextual_ & 0.67 & 0.64 & **0.79** & **0.79** \\ \hline _undefined_ & **0.97** & 0.90 & 0.77 & 0.83 \\ \hline \end{tabular}
\end{table}
Table 2: Accuracy of the classifiers considering only instances of a particular label.

\begin{table}
\begin{tabular}{|l|c|c|} \hline
**label (\(l\),)** & \(\mathbf{\check{y}_{i}=y_{i}}\) & \(\mathbf{\check{y}_{i}\neq y_{i}}\) \\ \hline \hline \multicolumn{3}{|c|}{_experts_} \\ _regular_ & 96 (\(78\%\)) & 28 (\(36\%\)) \\ \hline _discrepant_ & 1 (\(1\%\)) & 19 (\(25\%\)) \\ \hline _neutral_ & 26 (\(21\%\)) & 30 (\(39\%\)) \\ \hline \hline _mixed_ & 10 (\(8\%\)) & 9 (\(11\%\)) \\ \hline _factual_ & 6 (\(5\%\)) & 3 (\(4\%\)) \\ \hline _contextual_ & 8 (\(6\%\)) & 11 (\(14\%\)) \\ \hline _undefined_ & 2 (\(2\%\)) & 7 (\(9\%\)) \\ \hline \hline \multicolumn{3}{|c|}{_regular users_} \\ _regular_ & 111 (\(86\%\)) & 31 (\(44\%\)) \\ \hline _discrepant_ & 2 (\(2\%\)) & 14 (\(19\%\)) \\ \hline _neutral_ & 16 (\(12\%\)) & 26 (\(37\%\)) \\ \hline \hline _mixed_ & 13 (\(10\%\)) & 15 (\(21\%\)) \\ \hline _factual_ & 0 (\(0\%\)) & 1 (\(1\%\)) \\ \hline _contextual_ & 1 (\(1\%\)) & 6 (\(8\%\)) \\ \hline _undefined_ & 1 (\(1\%\)) & 5 (\(6\%\)) \\ \hline \end{tabular}
\end{table}
Table 3: Percentage of labels in correct (\(\hat{y}_{i}=y_{i}\)) and incorrect (\(\hat{y}_{i}\neq y_{i}\)) predictions by BERT.

 

where humans can easily identify what side has the most weight. Second, reviews that have some "irony" that is clear to humans, but is created using words with the opposite polarity of the final score \(y_{i}\). For examples, see Table 6 in the Appendix. We conjecture that these instances can be correctly classified with extra training and more modern (and complex) architectures. On the other hand, we feel that dealing with _hard instances_ is not that simple, where more guided and focused approaches are probably needed, such as the one proposed by Valdivia et al. (2019). They proposed an approach to combine reviews with scores for an aggregated polarity, which can be a good idea to deal with _hard instances_.

Overview of our results.Our first goal was to quantify the expected amount of _hard instances_ in misclassifications, which is \(\approx 56\%\) for _regular users_ and \(\approx 64\%\) for _experts_. Note that even though the reviews for these users are intrinsically different, the values are similar. The second goal was to quantitatively show how different the two types of _hard instances_ are. Table 1 shows that _neutral_ instances are common, and Table 3 shows they might have a significant presence even in correctly classified instances. Contrastingly, _discrepant_ instances are rare, particularly among correctly classified instances. Given that our ultimate goal was to quantify and explain the reasons behind misclassifications, from Table 3 we can say that most of the mistakes (\(\approx 60\%\)) occur because of _neutral_ (\(\approx 38\%\)) and _discrepant_ (\(\approx 22\%\)) instances.

## 6 Conclusion

In this work, we propose a methodology to characterize, quantify and measure the impact of _hard instances_ in the task of polarity classification of movie reviews. We characterized such instances into two disjoint categories: _neutrality_ and _discrepancy_. We provided empirical evidence about the need to pay attention to such instances, as they are much harder to be classified, for both machine and human classifiers.

The main hypothesis of this work is that _hard instances_ can make polarity classifiers fail. To demonstrate this hypothesis, we provided two well defined types of _hard instances_, which are based on human reasoning, and a methodology to find them in labeled data. With that, one can quantify how many instances of those types there are in their data, which can shed light on why and when classifiers fail. We collected a noise-free (no _class noise_) and well separated (no _neutral_ polarity) dataset and showed that even in such a dataset most of the mistakes made by a state of the art classifier, namely BERT, are in our defined _hard instances_. Observe in Table 3 that more than \(50\%\) of our sampled misclassified instances are _hard instances_ (_discrepant_ or _neutral_).

Our methodology works for every type of supervised classification task. Because our proposed labels are defined from the perspective of a classifier fully capable of human-reasoning, they are easy to interpret and can be generalized to every classification task (e.g. polarity, image, song genre, topic) that humans are able to do. After employing our methodology, it will be possible to differentiate mistakes that come from _hard instances_, which are those even humans cannot classify with confidence (or at all), and mistakes that could be solved by improving the classifier architecture. In short, our proposed methodology can help quantify and explain why classifiers are making mistakes.

We made the dataset containing the labels publicly available3 so it can be used as a standard benchmark for robustness to _hard instances_ in polarity classification tasks, and to potentially foster research on models, datasets and evaluation metrics tailored for this problem.

Footnote 3: [https://github.com/karenstemartins/NAACL2021](https://github.com/karenstemartins/NAACL2021)

#### Acknowledgements

This work is supported by the authors' individual grants from FAPEMIG, CAPES and CNPq. We also thank all the reviewers for their thoughtful comments which helped to improve this work.

## References

* Agarwal et al. (2007) S. Agarwal, S. Godbole, D. Punjani, and S. Roy. 2007. How much noise is too much: A study in automatic text classification. In _Seventh IEEE International Conference on Data Mining (ICDM 2007)_, pages 3-12.
* Arunachalam and Sarkar (2013) Ravi Arunachalam and Sandipan Sarkar. 2013. The new eye of government: Citizen sentiment analysis in social media. In _Proceedings of the IJCNLP 2013 workshop on natural language processing for social media (SocialNLP)_, pages 23-28.
* Barnes et al. (2019) Jeremy Barnes, Lilja Ovrelid, and Erik Velldal. 2019. Sentiment Analysis Is Not Solved! Assessing and 

Probing Sentiment Classification. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 12-23, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Klebanov and Beigman (2009) Beata Beigman Klebanov and Eyal Beigman. 2009. From Annotator Agreement to Noise Models. _Computational Linguistics_, 35(4):495-503.
* Proceedings of the Conference_, 2:390-396.
* Bermingham and Smeaton (2010) Adam Bermingham and Alan F. Smeaton. 2010. Classifying sentiment in microblogs: Is brevity an advantage? In _Proceedings of the 19th ACM International Conference on Information and Knowledge Management_, CIKM '10, page 1833-1836, New York, NY, USA. Association for Computing Machinery.
* Cheong and Lee (2011) Marc Cheong and Vincent C S Lee. 2011. A microblogging-based approach to terrorism informatics: Exploration and chronicling civilian sentiment and response to terrorism events via Twitter. _Information Systems Frontiers_, 13(1):45-59.
* Ciresan et al. (2012) D. Ciresan, U. Meier, and J. Schmidhuber. 2012. Multicolumn deep neural networks for image classification. In _2012 IEEE Conference on Computer Vision and Pattern Recognition_, pages 3642-3649. IEEE.
* Contractor et al. (2010) Danish Contractor, Tanveer A. Faruquie, and L. Venkata Subramaniam. 2010. Unsupervised cleansing of noisy text. In _Coling 2010: Posters_, pages 189-196. Coling 2010 Organizing Committee.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Dey and Haque (2009) Lipika Dey and S. K. Mirajul Haque. 2009. Studying the effects of noisy text on text mining applications. In _Proceedings of The Third Workshop on Analytics for Noisy Unstructured Text Data_, AND '09, page 107-114, New York, NY, USA. Association for Computing Machinery.
* Elaraby and Abdul-Mageed (2018) Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. Deep models for Arabic dialect identification on benchmarked data. In _Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)_, pages 263-274. Association for Computational Linguistics.
* Florian et al. (2010) Radu Florian, John Pitrelli, Salim Roukos, and Imed Zitouni. 2010. Improving mention detection robustness to noisy input. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 335-345, Cambridge, MA. Association for Computational Linguistics.
* Frenay and Verleysen (2014) Benoit Frenay and Michel Verleysen. 2014. Classification in the Presence of Label Noise: A Survey. _IEEE Transactions on Neural Networks and Learning Systems_, 25(5):845-869.
* Geirhos et al. (2018) Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schutt, Matthias Bethge, and Felix A. Wichmann. 2018. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems 31_, pages 7538-7550. Curran Associates, Inc.
* Goldberg et al. (2007) Andrew B Goldberg, Xiaojin Zhu, and Stephen Wright. 2007. Dissimilarity in graph-based semi-supervised classification. In _Artificial Intelligence and Statistics_, pages 155-162.
* Gui et al. (2017) Lin Gui, Yu Zhou, Ruifeng Xu, Yulan He, and Qin Lu. 2017. Learning representations from heterogeneous network for sentiment classification of product reviews. _Knowledge-Based Systems_, 124:34-45.
* 474. The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia.
* Hendrycks et al. (2018) Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data to train deep networks on labels corrupted by severe noise. In _Advances in neural information processing systems_, pages 10456-10465.
* Jurdall et al. (2019) Ishan Jurdall, Daniel Pressel, Brian Lester, and Matthew Nokleby. 2019. An effective label noise model for dnn text classification. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_, pages 3246-3256. Association for Computational Linguistics.
* Krymolowski (2002)Yuval Krymolowski. 2002. Distinguishing easy and hard instances. In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Liu et al. (2017) Tianyu Liu, Kexiang Wang, Baobao Chang, and Zhifang Sui. 2017. A Soft-label Method for Nonsidement Distantly Supervised Relation Extraction. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1790-1795. Association for Computational Linguistics 

Ismini Lourentzou, Kabir Manghnani, and ChengXiang Zhai. 2019. Adapting sequence to sequence models for text normalization in social media. _Proceedings of the International AAAI Conference on Web and Social Media_, 13(01):335-345.
* Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Nishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 4765-4774. Curran Associates, Inc.
* Michel and Neubig (2018) Paul Michel and Graham Neubig. 2018. MTNT: A Testbed for Machine Translation of Noisy Text. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 543-553.
* Volume 1_, NIPS'13, page 1196-1204. Curran Associates Inc.
* Pozzi et al. (2016) Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu. 2016. _Sentiment analysis in social networks_. Morgan Kaufmann.
* Rehbein and Ruppenhofer (2017) Ines Rehbein and Josef Ruppenhofer. 2017. Detecting annotation noise in automatically labelled data. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1160-1170. Association for Computational Linguistics.
* a benchmark comparison of state-of-the-practice sentiment analysis methods. _EPJ Data Science_, 5(1):23.
* Smith and Martinez (2011) Michael R. Smith and Tony Martinez. 2011. Improving classification accuracy by identifying and removing instances that should be misclassified. In _The 2011 International Joint Conference on Neural Networks_, pages 2690-2697. IEEE.
* Smith et al. (2014) Michael R. Smith, Tony Martinez, and Christophe Giraud-Carrier. 2014. An instance level analysis of data complexity. _Machine Learning_, 95(2):225-256.
* Stallkamp et al. (2012) J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. _Neural Networks_, 32:323-332.
* Valdivia et al. (2019) Ana Valdivia, Emiliya Hrabova, Iti Chaturvedi, M. Victoria Luzon, Luigi Troiano, Erik Cambria, and Francisco Herrera. 2019. Inconsistencies on TripAdvisor reviews: A unified index between users and Sentiment Analysis Methods. _Neurocomputing_, 353:3-16.
* Van Hulse et al. (2007) Jason D. Van Hulse, Taghi M. Khoshgoftaar, and Haiying Huang. 2007. The pairwise attribute noise detection algorithm. _Knowledge and Information Systems_, 11(2):171-190.
* Vinciarelli (2005) A. Vinciarelli. 2005. Noisy text categorization. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 27(12):1882-1895.
* Wang et al. (2016) Xingyou Wang, Weijie Jiang, and Zhiyong Luo. 2016. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 2428-2437. The COLING 2016 Organizing Committee.
* WSDM '15_, pages 435-440. ACM Press.
* Zhou et al. (2015) Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis C. M. Lau. 2015. A C-LSTM neural network for text classification. _CoRR_, abs/1511.08630.
* Zhu and Wu (2004) Xingquan Zhu and Xindong Wu. 2004. Class Noise vs. Attribute Noise: A Quantitative Study. _Artificial Intelligence Review_, 22(3):177-210.

## Appendix A Appendix

### Examples of Reviews

Tables 4 and 5 show some examples of _hard instances_ labeled by our human classifier. All the code and data is publicly available at [https://github.com/karenstemearntins/NAACL2021](https://github.com/karenstemearntins/NAACL2021).

### Regular Reviews

Table 6 shows real examples of misclassified _regular_ reviews with their original polarities given by their authors. The first and last review contain

\begin{table}
\begin{tabular}{|l|c|} \hline
**class label** & \\ (\(l_{i}\)) & **Example** \\ \hline _discrepant_ & “Figgis’s film doesn’t match its reach.” \\  & (Positive) \\ \hline _mixed_ & “Pleasant but dull formula film.” (Negative) \\ \hline _factual_ & “Without trivialzing the disease, the film challenges AIDS’ stigma (albeit for heterosexuals) at a moment when it was still considered a death sentence.” (Positive) \\ \hline _contextual_ & “Dishevelled triple pieced together with the good intentions.” (Negative) \\ \hline _undefined_ & “More interesting as history, re-written, than as the moral graphic this true story became.” (Positive) \\ \hline \end{tabular}
\end{table}
Table 4: Examples of _hard instances_ from _experts_.

 

some "irony" that is clear to humans, but they are created using words with their opposite polarity of the final score. The second review contains positive and negative points of the movie, but humans can easily identify what side has the most weight.

We further investigate these patterns by using SHAP Lundberg and Lee (2017), which is a game theoretic approach to explain the output of deep learning models and designed to understand the most important words for the machine classifiers. Figure 3 shows the result for the last review in Table 6. The words are plotted in descending order according with their importance. Note that, all listed words have a positive polarity when they are analyzed separately. As a result, their combination contributes to the classifier misclassify the review.

### Machine Classifier

#### a.3.1 Third Part Material

The code of the three machine classifiers used in this work are publicly available in the Internet. CNN-GRU and BERT were published by their authors and C-LSTM by researchers who used this method in their work Elaraby and Abdul-Mageed (2018). We made small modifications in the codes so they are able to process our movie reviews data. We also created a log module to register all the results and changed the final output layer to a sigmoid function, since our problem is a binary classification. We also made BERT use the Keras library just to facilitate our comparisons, but this is not a necessary step to reproduce our results. The link to each repository is listed bellow:

* C-LSTM: [https://github.com/EngSalem/TextClassification_Off_the_shelf](https://github.com/EngSalem/TextClassification_Off_the_shelf);
* CNN-GRU: [https://github.com/ultimate010/crnn](https://github.com/ultimate010/crnn);
* BERT: [https://github.com/google-research/bert](https://github.com/google-research/bert);

#### a.3.2 Model Training

To train the machine classifiers, we randomly generated two balanced partitions of our data with the same size, one for _experts_ and other for _regular users_. Each training dataset contains \(4,398\)_positive_ and \(4,398\)_negative_ reviews, for a total of \(8,796\) reviews. It is important to note that these datasets do not contain any review labeled by the human classifier. After that, we performed a 5-fold cross-validation to choose the best hyperparameters for our data. The set of hyperparameter configurations we tested were the same used in the original articles Wang et al. (2016), Zhou et al. (2015) and Devlin et al. (2019). Since the BERT architecture

\begin{table}
\begin{tabular}{|c|p{142.3pt}|} \hline \multicolumn{2}{|c|}{**Examples**} \\ \hline “Michael Bay may think that special effects can substitute for good acting and a good story, but that does not fly around here.” (Negative) \\ \hline ”From the first moment of Superman till the very end scene Lex luthor this is a true comic book movie adaption. True there are few CGI errors, but ”Nothing is perfect in this world” and this is just a movie.” (Positive) \\ \hline ”The trailer was promising to me; I expected it to be a really good movie, but instead it was ”meh”. I didn’t really like Cruz; it was heaurwarming how Lightning McQueon made a tribute to Doc at the end, but the trailer made it seem action packed; it wasn’t as good as I expected.” (Negative) \\ \hline  & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 6: Examples of misclassified _regular_ reviews.

Figure 3: SHAP plot for the last review in Table 6.

 

is very simple, it has only a single hyperparameter, the batch size, for which we tested values of \(16\), \(32\) and \(64\). For C-LSTM, we tested layers with \(100\), \(150\) and \(200\) filters, and filters of size \(2\), \(3\) and \(4\), memory dimensions of size \(100\), \(150\) and \(200\), and batch size of \(16\), \(32\) and \(64\). Finally, for CNN-GRU, we tested layers with \(100\) and \(200\) filters, filters of size \(3\) and \(4\), GRU dimensionality of \(100\) and \(150\), pool sizes of \(2\) and \(3\), and batch sizes of \(16\) and \(32\). To run our experiments, we use a computer with the following configuration: 32 RAM, Intel Core i7 CPU 3.40 GHz and NVIDIA GeForce GTX GPU.

After executing cross-validation, we selected the best hyperparameters for each architecture and type of users comparing their F1-Score. We use \(256\) words for models trained with _expert_ data and \(512\) for those trained with _regular user_ data in all architectures. BERT achieved the best results using a batch size of \(16\) for both user types. For _experts_, C-LSTM uses a batch size of \(32\), \(100\) filters with size \(3\) in the convolutional layer, and \(200\) as memory dimension for LSTM. For _regular users_, the hyperparameters are the same, except in the LSTM layer, where a memory dimension of \(100\) was used. For _experts_, CNN-GRU uses \(100\) filters with size \(5\) as filter length and \(3\) as pool size for both CNNs. In the GRU, we used dimensionality of \(150\) and batch size of \(16\). For _regular users_, the differences are that we used a dimensionality of \(100\) in the GRU layer, size \(3\) as filter length and \(2\) as pool size for both CNNs. For both C-LSTM and CNN-GRU the differences in the hyperparameters are explained by the fact that our _expert_ reviews are significantly shorter than the ones wrote by _regular users_. After selecting the best hyperparameters, we trained two models for each architecture, one for _experts_ and other for _regular users_. Also, each result reported in the paper is the average of five runs, where for each run the model is trained from start using the whole training dataset. With that, we can measure their parameter sensitivity and calculate confidence intervals for the results. In addition, C-LSTM and CNN-GRU took approximately half a day to train and BERT one day. Finally, we noted that the performance of all three models were not significantly affected by the hyperparameter configurations we tested