SlowGuess/ACL_2020_OCR huggingface dataset:
Cansu Sen<sup>1</sup>, Thomas Hartvigsen<sup>2</sup>, Biao Yin<sup>2</sup>, Xiangnan Kong<sup>1,2</sup>, and Elke Rundensteiner<sup>1,2</sup>
<sup>1</sup>Computer Science Department, Worcester Polytechnic Institute <sup>2</sup>Data Science Program, Worcester Polytechnic Institute {csen,twhartvigsen,byin,xkong,rundenst}@wpi.edu
Motivated by human attention, computational attention mechanisms have been designed to help neural networks adjust their focus on specific parts of the input data. While attention mechanisms are claimed to achieve interpretability, little is known about the actual relationships between machine and human attention. In this work, we conduct the first quantitative assessment of human versus computational attention mechanisms for the text classification task. To achieve this, we design and conduct a large-scale crowd-sourcing study to collect human attention maps that encode the parts of a text that humans focus on when conducting text classification. Based on this new resource of human attention dataset for text classification, YELP-HAT, collected on the publicly available YELP dataset, we perform a quantitative comparative analysis of machine attention maps created by deep learning models and human attention maps. Our analysis offers insights into the relationships between human versus machine attention maps along three dimensions: overlap in word selections, distribution over lexical categories, and context-dependency of sentiment polarity. Our findings open promising future research opportunities ranging from supervised attention to the design of human-centric attention-based explanations.
Attention-based models have become the architectures of choice for a vast number of NLP tasks including, but not limited to, language modeling (Daniluk et al., 2017), machine translation (Bahdanau et al., 2015), document classification (Yang et al., 2016), and question answering (Kundu and Ng, 2018; Sukhbaatar et al., 2015). While attention mechanisms have been said to add interpretability since their introduction (Bahdanau et al., 2015), the investigation of whether this claim is correct has
I really really enjoy this place!! But, I'm going to agree with a few other folks on 1 issue... Why is the music so damn loud in the bar?? Anyway, drinks are tasty and I love their "Social Hour" from 2-6 pm. Will definitely be going back to this place!
i really really enjoy this place but im going to agree with a few other folks on 1 issue why is the music so damn loud in the bar anyway drinks are tasty and i love their social hour from  $26\mathrm{pm}$  will definitely be going back to this place
Figure 1: Examples of binary human attention (blue in top two texts) and continuous machine attention (red in bottom text).
only just recently become a topic of high-interest (Mullenbach et al., 2018; Thorne et al., 2019; Serrano and Smith, 2019). If attention mechanisms indeed offer a more in-depth understanding of a model's inner-workings, application areas from model debugging to architecture selection would benefit greatly from profound insights into the internals of attention-based neural models.
Recently, Jain and Wallace (2019), Wiegreffe and Pinter (2019), and Serrano and Smith (2019) proposed three distinct approaches for evaluating the explainability of attention. Jain and Wallace (2019) base their work on the premise that explainable attention scores should be unique for a given prediction as well as consistent with other feature-importance measures. This prompts their conclusion that attention is not explanation. Based on similar experiments on alternative attention scores, Serrano and Smith (2019) conclude that attention does not necessarily correspond to the importance of inputs. In contrast, Wiegreffe and Pinter (2019) find that attention learns a meaningful relationship between input tokens and model predictions, which cannot be easily hacked adversarially.
While these works ask valuable questions, they embrace model-driven approaches for manipulating the attention weights and thereafter evaluate the
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4596-4608 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
post-hoc explainability of the generated machine attention. In other words, they overlook the human factor in the evaluation process – which should be integral in assessing the plausibility of the generated explanations (Riedl, 2019).
In this work, we adopt a novel approach to attention explainability from a human-centered perspective and, in particular, investigate to what degree machine attention mimics human behavior. More precisely, we are interested in the following research question: Do neural networks with attention mechanisms attend to the same parts of the text as humans? To this end, we first collect a large dataset of human-attention maps and then compare the validated human attention with a variety of machine attention mechanisms for text classification.
Figure 1 displays examples of human and machine-generated attention for classifying a restaurant review's overall rating. Our goal is to quantify the similarity between human attention and machine-generated attention scores. Measuring this similarity is non-trivial and is not appropriately captured by an existing similarity metric (e.g., Euclidean) between two vectors for the following reasons. A binary human attention vector does not solely denote which tokens are given higher importance but also implies information about the underlying grammatical structure and linguistic construction. For example, whether or not adjectives tend to be high-importance is encoded in the attention weights as well. Further, it is well known that human attention is itself subjective: given the same text and task, human annotators may not always agree on which words are important. That is, one single human's attention should rarely be regarded as the ground-truth for attention.
Given this objective, we use crowd-sourcing to collect a large set of human attention maps. We provide a detailed account of the iterative design process for our data collection study in §3. We design new metrics that quantify the similarity between machine and human attention from three perspectives (§4): Behavioral similarity measures the number of common words selected by human and machine discerning if neural networks with attention mechanisms attend to the same parts of the text as humans. Humans associate certain lexical categories (e.g., adjectives) with a sentiment more heavily. Lexical (grammatical) similarity identifies if machine attention favors similar lexical categories with humans. A high lexical similarity shows that
the attention mechanism learns similar language patterns with humans. Context-dependency quantifies sentiment polarity of word selections.
We then employ these metrics to compare attention maps from a variety of attention-based Recurrent Neural Networks (RNN). We find that bidirectional RNNs with additive attention demonstrate strong similarities to human attention for all three metrics. In contrast, uni-directional RNNs with attention differ from human attention significantly. Finally, as the text length increases, and with it, the prediction task becomes more difficult, both the accuracy of the models and similarity between human and machine decrease.
- We conduct a large-scale collection of 15,000 human attention maps as a companion to the publicly-available Yelp Review dataset. Our collected Yelp-HAT (Human ATtention) dataset is publicly available as a valuable resource to the NLP community.
- We develop rich metrics for comparing human and machine attention maps for text. Our new metrics cover three complementary perspectives: behavioral similarity, lexical similarity, and context-dependency.
- We conduct the first in-depth assessment comparing human versus machine attention maps, with the latter generated by a variety of state-of-the-art soft and hard attention.
- We show that when used with bidirectional architectures, attention can be interpreted as human-like explanations for model predictions. However, as text length increases, machine attention resembles human attention less.
2 Preliminaries on Attention Maps
In this section, we define the concepts of Human Attention Map and Machine Attention Map.
Definition 2.1. Attention Map. An Attention Map (AM) is a vector where each entry in sequence is associated with a word in the corresponding position of the associated text. The value of the entry indicates the level of attention the corresponding word receives with respect to a classification task.
Definition 2.2. Human Attention Map. A Human Attention Map (HAM) is a binary attention
map produced by a human, where each entry with a set-bit indicates that the corresponding word receives high attention.
Definition 2.3. Machine Attention Map. A Machine Attention Map (MAM) is an attention map generated by a neural network model. If computed through soft-attention, a MAM corresponds to an AM of continuous values, that capture a probability distribution over the words. If computed through hard-attention, a MAM is a binary AM.
We now introduce the application of aggregation operators to coalesce HAMs by multiple annotators into aggregated HAMs.
Definition 2.4. Consensus Attention Map. If multiple HAMs exist for the same text, a Consensus Attention Map (CAM) is computed through a bitwise AND operation of the HAMs.
Definition 2.5. Super Attention Map. If multiple HAMs exist for the same text, a Super Attention Map (SAM) is computed by a bitwise OR operation of the HAMs.
3 Collection and Analysis of Human Attention Maps
3.1 HAM Collection by Crowd-sourcing
We collect human attention maps for the Yahoo dataset on the classification task of rating a review as positive or negative on Amazon Mechanical Turk. Participants are asked to complete two tasks: 1) Identify the sentiment of the review as positive, negative, or neither, and 2) Highlight the words that are indicative of the chosen sentiment. Our interface used for data collection is in Figure 2.
Preliminary investigation of the quality of human annotations. First, we conduct a series of data collection studies on two subsets of the Yelp dataset. Both subsets consist of 50 randomly-selected reviews from the Restaurant category. The first subset contains reviews with exactly 50 words, while the second contains reviews with exactly 100 words. For each review, human annotation is collected from two unique users.
We explore the quality of data we can collect on Mechanical Turk, as it encourages users to complete their tasks as quickly as possible since the number of completed tasks determines their income. This may lower the quality of collected
Figure 2: User interface we used for data collection on Amazon Mechanical Turk.
data since users may not select all relevant words, instead opting for the few most obvious ones, or they may choose words randomly.
Based on our preliminary investigations, we observe that both the average time users spend on the task (44 vs. 70 seconds) and the average number of words selected per review (9 vs. 13 words) increase as the number of words in the review increases from 50 to 100. This suggests that users do not choose words randomly; instead, they make an informed decision. We also visually examine the collected human attention maps and confirm that subjects make meaningful selections.
Pilot study assessing two design choices for data collection. Next, we design another pilot study to understand how humans perform the cognitive task of classifying a text and selecting the particular words that led to this decision. In this study, we ask eight participants to perform the same task while adhering to one of two strategies. The first strategy, the read-first design, involves reading the review first, deciding on the sentiment, then rereading the review, this time to highlight the relevant words. The second strategy, the free-style design, gives participants the freedom to choose the relevant words as they read the review to determine the sentiment. Each participant is asked to complete two tasks to experience both strategies. Half of the participants first work with the read-first design followed by the free-style design while the other half work in the reverse order. After completing the tasks, we ask the participants which strategy they find more natural in a post-task questionnaire.
Findings from the pilot study. Out of eight participants, half of them find it more useful reading the review first then deciding on the words whereas the other half indicated the opposite. We then evaluate
1https://www.yelp.com/dataset/challenge
4598
the collected data from three perspectives to decide which design is most suitable for our purposes.
We first examine the agreement between participants adhering to a particular strategy. This involves calculating the percentage of participants that mutually select the same phrase. We find that participant agreement is higher (73%) when the participants are forced to read the review before making any selections compared to using the freestyle design (69%). Next, we investigate how similar the results are to the ground truth we defined for each review. The read-first design achieves better performance (3.30) compared to the freestyle design (3.10). Our final criterion involves examining the amount of noise in the data (i.e., selections which deviate from the chosen sentiment). Only one review exhibits this situation where the review is clearly positive; however, it also contains a negative-opinion sentence. We observe that the read-first design reduces this cross-sentiment noise (1 vs. 0.5 scores).
Data collection protocol for the main study. Based on conclusions from the pilot studies, the read-first design is adopted to conduct the main data collection for 5,000 reviews on Amazon Mechanical Turk. For this study, three different subjects annotated each review, resulting in a total of 15,000 human attention maps. The resulting Yelp Human Attention Dataset (YELP-HAT) is publicly available<sup>2</sup>.
3.2 Analysis and Insights About HAMs
Factors that affect human accuracy. Some reviews contain a mixture of opinions, even though the reviewer felt strongly positive or negative about the restaurant. For example, consider the following review: "Nothing to write home about, the chicken seems microwaved and the appetizers are meh. ... If your [sic] looking for a quick oriental fix I'd say go for it... otherwise look elsewhere." This review is labeled as negative, positive, and neither. The annotator who assigned it to the positive class selected the words "go for it" while the annotator who assigned it to the negative class selected the words "otherwise look elsewhere". This type of "mixed review" is the principal reason for discrepancies in classifications by the human annotators. The nature of crowd-sourcing also causes such inconsistencies as not all annotators provide reviews
of equal quality.
Ambiguity in human attention. Intuitively, human attention is highly subjective. Some common patterns across annotators lead to differences in human annotations. A common behavior is to select keywords that indicate a sentiment. Another typical action is to select entire sentences if the sentence expresses an opinion.
Some reviews include subjective phrases that people interpret differently with regard to sentiment-polarity. For instance, "I come here often" can be construed as a favorable opinion. However, some people find it neutral. In some cases, an overwhelmingly-positive review incorporates a negative remark (or vice versa). In these cases, some people select all pieces of evidence of any sentiment, whereas others only choose words that indicate the prevailing sentiment.
4 Attention Map Similarity Framework
We quantify the similarity between HAMs and MAMs through our similarity framework that contains three new metrics as described in this section.
4.1 Overlap in Word Selections
For two attention mechanisms to be similar, they must put attention on the same parts of the text. Thus, we first define a metric for quantifying the overlap in the words selected by human annotators and by deep learning models.
Definition 4.1. Behavioral Similarity. Given a collection of attention maps  $\mathrm{HAM}_{\mathcal{D}}$  and  $\mathrm{MAM}_{\mathcal{D}}$  for a text dataset  $\mathcal{D}$ , behavioral similarity between human (H) and machine (M) corresponds to the average pair-wise similarity between each  $(\mathrm{HAM}_i, \mathrm{MAM}_i)$  vector pair  $\forall i \in \mathcal{D}$  as defined below:
$$
\operatorname {P a i r w i s e S i m} _ {i} = A U C \left(\mathrm {H A M} _ {i}, \mathrm {M A M} _ {i}\right)
$$
BehavioralSim(M,H) =  $\frac{1}{|\mathcal{D}|}\sum_{i}(\mathrm{PairwiseSim}_{i})$
where  $|\mathcal{D}|$  is the number of reviews in the dataset  $\mathcal{D}$ . Intuitively, this corresponds to adopting the human attention vector as binary ground truth. That is, it measures how similar the machine-generated continuous vector is to this ground truth. AUC is between 0 and 1 with .5 representing no similarity, and 1 the perfect similarity.
$^{2}$ http://davis.wpi.edu/dsrg/PROJECTS/YELPHAT/index.html
4.2 Distribution over Lexical Categories
Previous work has found that lexical indicators of sentiment are commonly associated with syntactic categories such as adjective, adverb, noun, and verb (Marimuthu and Devi, 2012). We define the following lexical similarity metric to test if human and machine adopt similar behaviors in terms favoring certain lexical categories.
Definition 4.2. Lexical Similarity. Given a collection of attention maps  $\mathrm{HAM}_{\mathcal{D}}$  and  $\mathrm{MAM}_{\mathcal{D}}$  for a text dataset  $\mathcal{D}$ , Lexical Similarity (LS) between human (H) and machine (M) over  $\mathcal{D}$  is computed:
$$
\operatorname {L S} (M, H) = \operatorname {c o r r} (\operatorname {d i s t} (\operatorname {w o r d s} _ {H}), \operatorname {d i s t} (\operatorname {w o r d s} _ {M}))
$$
where  $\mathrm{words}_H$  is a list of all selected words in all reviews of  $\mathcal{D}$  by human,  $\mathrm{words}_M$  is a list of all selected words in all reviews of  $\mathcal{D}$  by machine,  $\mathrm{dist}()$  is a function that computes the distribution of a word list over a tagset (e.g., nouns, verbs, etc.). After computing two distributions, the  $\mathrm{corr}()$  function computes the correlation between them. In our experiments, we adopt Pearson Correlation. If MAM is continuous, selected words by M corresponds to k words with the highest attention scores, where k is the number of words selected by human for that text.
Using a random attention  $R$  as a baseline where the most important  $k$  words are selected randomly, we then compute an Adjusted Lexical Similarity which is between 0 and 1 as follows.
$$
\text {A d j u s t e d L S} = \frac {\mathrm {L S} (M , H) - \mathrm {L S} (R , H)}{1 - \mathrm {L S} (R , H)}
$$
4.3 Context-dependency of Sentimental Polarity
When deciding the sentiment of a review, human subjects may consider positive-sentiment words in a negative review and vice versa. To assess how context-dependant human and machine attentions are, we compute cross-sentiment selections rates.
Definition 4.3. Cross-sentiment selection rate (CSSR). Assume we have a collection of attention maps  $\mathrm{AM}_{\mathcal{D}}$  for a dataset  $\mathcal{D}$ , ground truth for overall sentiment  $Y$  for each review in  $\mathcal{D}$  ( $y_{i} \in \{0,1\}$ ), and a list of positive words  $\mathcal{P}$  and negative words  $\mathcal{N}$  in the English language. CSSR denotes the ratio of selected words from the opposite sentiment.
$$
p _ {-} w o r d s = g e t _ {-} w o r d s \left(H A M _ {\mathcal {D}}, Y = 1\right)
$$
$$
n _ {-} w o r d s = g e t _ {-} w o r d s \left(H A M _ {\mathcal {D}}, Y = 0\right)
$$
$$
C S S R _ {p} = \frac {\left| p \_ w o r d s \cap \mathcal {N} \right|}{\left| p \_ w o r d s \cap \mathcal {P} \right|}
$$
$$
C S S R _ {n} = \frac {\left| n _ {-} w o r d s \cap \mathcal {P} \right|}{\left| n _ {-} w o r d s \cap \mathcal {N} \right|}
$$
get_words() function returns a list of attention-receiving words where  $\mathrm{HAM}_{ij} = 1, \forall i, j$  for the entire set of  $\mathrm{HAM}_D$ , for positive-sentiment reviews ( $Y = 1$ ) and negative-sentiment reviews ( $Y = 0$ ) separately. A list of words with positive and negative connotations,  $\mathcal{P}$  and  $\mathcal{N}$ , are obtained from Hu and Liu (2004).  $\mathrm{CSSR}_p$  (positive) and  $\mathrm{CSSR}_n$  (negative) is then computed as the ratio of the number of cross-sentiment words over the number of same-sentiment words. A high CSSR means many words from the opposite sentiment are selected. This metric provides insights about how similar human and machine attentions are with regard to their context-dependant behaviour.
5 Is Machine Attention Similar to Human Attention?
5.1 Generating Machine Attention Maps
The Yelp dataset contains reviews and their rating scores between 0 and 5 (stars). This rating score corresponds to the ground truth for the review's overall sentiment. We create a binary classification task by assigning 1 and 2-star reviews to the negative class and 4 and 5-star reviews to the positive class. We omit 3-star reviews as they may not exhibit a clear sentiment. For training neural network models, we extract balanced subsets and split them into  $80\%$  training set,  $10\%$  validation set and  $10\%$  test sets. We then generate MAMs using the following machine learning models.
RNN with soft attention. Recurrent Neural Networks (RNN) enhanced with attention mechanisms have emerged as the state-of-the-art for NLP tasks (Bahdanau et al., 2015; Yang et al., 2016; Daniluk et al., 2017; Kundu and Ng, 2018). We implement the additive attention for many-to-one classification task as it is commonly used in the literature (Yang et al., 2016; Bahdanau et al., 2015) and paired it with both uni- and bi-directional RNN. In our implementation, we use LSTM memory cells.
Accuracy
Yelp-50	Yelp-100	Yelp-200
Human	0.96	0.94	0.94
RNN	0.91 ± 0.006	0.90 ± 0.013	0.88 ± 0.01
biRNN	0.93 ± 0.008	0.91 ± 0.005	0.88 ± 0.02
Rationales	0.90 ± 0.004	0.85 ± 0.035	0.77 ± 0.015
Table 1: Test accuracy from three subsets of Yelp data.
Assuming that  $\Gamma$  is the recurrence function of LSTM and  $x_{i}$  is the embedded  $i$ -th word of  $T$  words in a review, we model our method as:
$$
h _ {i} = \Gamma \left(x _ {i}, h _ {i - 1}\right), i \in [ 1, T ] \tag {1}
$$
$$
u _ {i} = \tanh  \left(W h _ {i} + b\right) \tag {2}
$$
$$
\alpha_ {i} = \frac {\exp \left(u _ {i} ^ {\top} u\right)}{\sum_ {t} \exp \left(u _ {i} ^ {\top} u\right)} \tag {3}
$$
Here  $h_i, i \in [1,T]$  are hidden representations,  $W, b,$  and  $u$  are trainable parameters, and  $\alpha_{i}, i \in [1,T]$  are the attention scores for each word  $x_{i}$ . A context vector  $c_{i}$  corresponds to the weighted average of the hidden representations of words with attention weights, denoted by:
$$
c _ {i} = \sum_ {j} \alpha_ {j} h _ {j} \tag {4}
$$
Through a softmax layer, context vector  $c_{i}$  is then used for further classifying the input sequence.
Rationale mechanism. An alternative approach, referred to as "rationale mechanism", can be seen as a type of hard attention (Lei et al., 2016; Bao et al., 2018). This model consists of two main parts that are jointly learned: a generator and an encoder. The generator specifies a distribution over the input text to select candidate rationales. The encoder is used to make predictions based on the rationales. The two components are integrated and regularized in the cost function with two hyper-parameters, selection lambda, and continuity lambda, for optimizing the representative selections. The selection lambda penalizes the number of words selected, while the continuity lambda encourages the continuity via minimizing the distances of the words chosen.
5.2 Behavioral Similarity Analysis
We conduct a set of controlled experiments with the length of the review changing across experiments. First, we generate MAMs for three subsets of the Yelp dataset: reviews containing 50 words