SlowGuess/ACL_2020_OCR huggingface dataset:
(Yelp-50), 100 words (Yelp-100) and 200 words (Yelp-200). Neural network models with attention mechanisms are trained on each of these subsets. The corresponding test set accuracies for sentiment classification of human versus machine are shown in Table 1. Next, we acquire the HAMs collected for each test set. Since each review is annotated by three people, we have three sets of HAMs:  $\mathrm{HAM}_1$ ,  $\mathrm{HAM}_2$ , and  $\mathrm{HAM}_3$ . Consensus among the three, CAM and SAM, are computed as per Defs. 2.4 and 2.5. Then we measure the Behavioral Similarity between human and machine. The amount of overlap in the selected words are presented in Table 2.
We observe that accuracy and similarity both decrease as the review-length increases and the classification task becomes more difficult for both humans and machine learning models. We identify two reasons for this: First, when a review is long, the prevailing opinion is usually not obvious at first glance and may require more intensive reading and contemplating. Second, the reviewers are more likely to state conflicting facts and opinion in long reviews. This, in turn, creates distracting and hard-to-read text. Compared to unidirectional model, bidirectional RNN with attention consistently rates closer to human attention. This is most striking for the Yelp-50 subset. This can be explained with the fact that bidirectional RNNs possess information from both directions of the text similar to humans.
For all three subsets, Yelp-50, Yelp-100, and Yelp-200, behavioral similarity for Consensus Attention Map is higher than all three HAMs. This is an important result because it indicates that the words all annotators agreed to be important are selected by machine attention too, whereas more subjective selections do not always get high attention from machine, indicated by lower SAM similarity.
Finally, we compare similarity of these three sets of HAMs. Even though human-to-human similarity is usually higher than human-to-machine similarity (as expected), the numbers still far from being close to 1. This confirms the subjectivity of human attention. Also, note that human-to-human similarity decreases as the review length increases.
We observe that the performance of the rationale-based models degrades more sharply as the review-length increases. As our goal is to compare human attention with machine-generated attention for model interpretability, we optimize the model not only for accuracy but also for the number of selected rationales. We aim to generate roughly an
Yelp-50	HAM1,k=10	HAM2,k=12	HAM3,k=12	CAM,k=5	SAM,k=22
HAM2	0.73	-	-	-	-
HAM3	0.74	0.75	-	-	-
RNN Attention	0.59 ± 0.021	0.59 ± 0.002	0.57 ± 0.012	0.59 ± 0.024	0.58 ± 0.021
Bi-RNN Attention	0.69 ± 0.004	0.70 ± 0.008	0.69 ± 0.007	0.79 ± 0.003	0.64 ± 0.008
Rationales	0.62 ± 0.014	0.62 ± 0.012	0.63 ± 0.015	0.68 ± 0.020	0.58 ± 0.010
Yelp-100	HAM1,k=15	HAM2,k=16	HAM3,k=16	CAM,k=6	SAM,k=30
HAM2	0.71	-	-	-	-
HAM3	0.73	0.74	-	-	-
RNN Attention	0.57 ± 0.009	0.58 ± 0.011	0.59 ± 0.012	0.57 ± 0.010	0.58 ± 0.008
Bi-RNN Attention	0.65 ± 0.011	0.65 ± 0.021	0.66 ± 0.021	0.73 ± 0.031	0.62 ± 0.012
Rationales	0.55 ± 0.015	0.55 ± 0.005	0.55 ± 0.010	0.59 ± 0.015	0.54 ± 0.005
Yelp-200	HAM1,k=26	HAM2,k=27	HAM3,k=25	CAM,k=11	SAM,k=45
HAM2	0.70	-	-	-	-
HAM3	0.69	0.71	-	-	-
RNN Attention	0.60 ± 0.011	0.60 ± 0.013	0.60 ± 0.014	0.60 ± 0.017	0.60 ± 0.011
Bi-RNN Attention	0.61 ± 0.015	0.61 ± 0.008	0.61 ± 0.018	0.63 ± 0.009	0.60 ± 0.008
Rationales	0.51 ± 0.013	0.52 ± 0.021	0.51 ± 0.018	0.52 ± 0.025	0.49 ± 0.019
Table 2: Behavioral similarity of human attention to machine on varying review length. k indicates the average number of words selected. (0.5:no similarity, 1.0:perfect similarity)
equal number of words selected by both human annotators and machine-generated rationales. Hence, we force the rationale-models to pick fewer words by tuning the selection lambda accordingly. This gives a comparative advantage to attention-based models against rationale-based models, as the rationale model is a hard-attention mechanism. In addition, rationales are better suited for sentence-level tasks as they encourage consecutive selection as opposed to the behavior of attention.
5.3 Lexical Similarity Analysis
Next, we analyze if humans and neural networks pay more attention to words from particular lexical categories using Adjusted Lexical Similarity score.
Lexical Similarity results, presented in Table 3, are consistent with Behavioral Similarity in that bidirectional model with attention is most similar to human (0.91 for Yelp-50 and 0.84 for Yelp-100). Rationales model follows bidirectional RNN, and unidirectional RNN is the least similar model to human. Overall, lexical similarity to human decreases for all models, as the reviews become longer.
Next, we inspect which lexical categories are selected more heavily by human and machine. For this, we provide relative frequency of lexical categories for human-selected words, machine-selected words (bi-RNN), and overall relative frequency of this tag within the dataset. Adjectives (Human:0.24
bi-RNN:0.23 Overall:0.02), comparative adjectives (Human:0.002 bi-RNN:0.001 Overall:0.0001), and nouns (Human:0.38 bi-RNN:0.37 Overall:0.09) are among the lexical categories that humans and biRNN models favor heavily. Similarly, personal pronouns are rarely selected by neither humans nor bi-RNN models (Human:0.005 bi-RNN:0.005 Overall:0.01).
5.4 Cross-sentiment Selection Rate Analysis
Finally, we compute CSSR scores, presented in Table 4, to evaluate the context-dependency of sentimental polarity for human and machine attentions. Our observations for Yelp-50 dataset are as follows. By human annotators, almost exclusively positive words are selected if the overall review sentiment is positive. For negative reviews, higher number of positive words are selected than negative words  $\mathrm{(CSSR_p = 0.06, CSSR_n = 0.20)}$ . Among the neural network models, the bidirectional RNN once more behaves most similar to human annotators with  $\mathrm{CSSR_p = 0.04}$  and  $\mathrm{CSSR_n = 0.19}$ . RNN model's approach differs from that of human's and bi-RNN's. Even though the behaviour is similar for positive polarity  $\mathrm{(CSSR_p = 0.06)}$ , the opposite is true for negative polarity. In fact, positive words selected 2.28 times more than negative words in negative reviews, which is counter-intuitive. For the Rationales model,  $\mathrm{CSSR_p}$  is 0.08 and  $\mathrm{CSSR_n}$
is 0.44. This indicates that Rationales model is more similar to human attention than RNN model with attention. We observe similar trends for the Yelp-100 and Yelp-200 datasets.
A large body of work has been using attention mechanisms to attempt to bring 'interpretability' to model predictions (Choi et al., 2016; Sha and Wang, 2017; Yang et al., 2016). However, they only assess the produced attention maps qualitatively by visualizing a few hand-selected instances. Recently, researchers began to question the interpretability of attention. Jain and Wallace (2019) and Serrano and Smith (2019) argue that if alternative attention distributions exist that produce similar results to those obtained by the original model, then the original model's attention scores cannot be reliably used to explain the model's prediction. They empirically show that achieving such alternative distributions is possible. In contrast, Wiegrefe and Pinter (2019) find that attention learns a meaningful relationship between input tokens and model predictions which cannot be easily hacked adversarially.
Das et al. (2016) conducted the first quantitative assessment of computational attention mechanisms for the visual question answering (VQA) task. Similar to our work, they collect a human attention dataset, then measure the similarity of human and machine attention within the context of VQA. This VQA-HAT dataset now provides a fertile research vehicle for researchers in computer vision for studying the supervision of the attention mechanism (Liu et al., 2017a). The development of a similar dataset and an in-depth quantitative evaluation for text to advance NLP research is sorely lacking. In a concurrent and independent work, DeYoung et al. (2019) collects the ERASER dataset for human annotations of rationales. While ERASER includes multiple datasets for a number of NLP tasks with relatively small amounts of data for each, we focus on text classification and collect a large amount of data on a different corpus.
Recent papers, including our work, take strides at answering the question if attention is interpretable. This is complicated by the fact that "interpretability" remains a not well-defined concept.
Attention adds transparency. Lipton (2018) defines transparency as overall human-
understanding of a model, i.e., why a model makes its decisions. Under this definition, attention scores can be seen as partial transparency. That is, they provide a look into the inner workings of a model, in that they produce an easily-understandable weighting of hidden states (Wiegreffe and Pinter, 2019).
Attention is not faithful. Whether adversarial attention scores exist that result in the same predictions as the original attention scores helps us understand if attention is faithful. With their empirical analyses, Serrano and Smith (2019) and Jain and Wallace (2019) show that attention is not faithful.
Rationale models for human-like explanations. Riedl (2019) argues that explanations are post-hoc descriptions of how a system came to a given conclusion. This raises the question of what makes a good explanation of the behavior of a machine learning system. One line of research offers these explanations in the form of binary rationales, namely, explanations that plausibly justify a model's actions (Bao et al., 2018; Lei et al., 2016).
Our approach at attention as human-like explanations. In claiming attention is explanation, it is seen to mimic humans in rationalizing past actions. In our work, we approach interpretability from this human-centric perspective. We develop a systematic approach to either support or refute the hypothesis that attention corresponds to humanlike explanations for model behavior. Based on our comparative analyses, we provide initial answers to this important question by finding insights into the similarities and dissimilarities of attention-based architectures to human attention.
Towards additional tasks beyond text classification. Confidently concluding whether attention mimics human requires tremendous efforts from many researchers with human data to be collected via a well-designed data collection methodology, both labor-intensive and costly task. In this work, we thus focus on one task, namely, sentiment classification, and collect HAM for this task and on a single dataset. We invite other researchers to continue this line of research by exploring other tasks (e.g., question answering).
Next steps in attention research. Our work opens promising future research opportunities. One is to supervise attention models explicitly. Attention mechanisms themselves are typically learned in an unsupervised manner. However, initial re
Yelp-50	Yelp-100	Yelp-200
Lexical Sim.	Adjusted LS	Lexical Sim.	Adjusted LS	Lexical Sim.	Adjusted LS
Random Attention	0.85 ± 0.006	-	0.84 ± 0.013	-	0.90 ± 0.010	-
RNN Attention	0.93 ± 0.015	0.54	0.91 ± 0.007	0.44	0.93 ± 0.005	0.37
Bi-RNN Attention	0.99 ± 0.005	0.91	0.98 ± 0.013	0.84	0.93 ± 0.003	0.36
Rationales	0.95 ± 0.012	0.66	0.93 ± 0.027	0.53	0.90 ± 0.002	0.05
Table 3: Lexical Similarity and Adjusted Lexical Similarity of human attention to machine on varying review length. (Adjusted LS 0:no similarity, 1:perfect similarity)
CSSRp	CSSRn
Human	0.06	0.20
RNN Attention	0.06	2.28
Bi-RNN Attention	0.04	0.19
Rationales	0.08	0.44
Table 4: Cross-sentiment Selection Rates for positive and negative reviews for Yelp-50 dataset.
search offers compelling evidence for the success of supervised attention models (Chen et al., 2017; Liu et al., 2017b) in the computer vision area. Also, attention has the potential to be leveraged for both making predictions and concurrently producing human-centric explanations similar to rationale-based architectures.
To gain a deeper understanding of the relationships between human and attention-based neural network models, we conduct a large crowd-sourcing study to collect human attention maps for text classification. This human attention dataset represents a valuable community resource that we then leverage for quantifying similarities between human and attention-based neural network models using novel attention-map similarity metrics. Our research not only results in insights into significant similarities between bidirectional RNNs and human attention, but also opens the avenue for promising future research directions.
This research was supported by the U.S. Dept. of Education grant P200A150306, Worcester Polytechnic Institute through the Arvid Anderson Fellowship, and the National Science Foundation through grants IIS-1815866, IIS-1910880, IIS-1718310, and CNS -1852498. We thank Prof. Lane Harrison, WPI, for his advice and guidance on the
design study for the data collection, and Prof. Jeanine Skorinko, WPI, for helpful discussion about the cognitive aspects of human attention.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. Proceedings of International Conference on Learning Representations.
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1903-1913.
Lei Chen, Mengyao Zhai, and Greg Mori. 2017. Attending to distinctive moments: Weakly-supervised attention models for action localization in video. In Proceedings of the IEEE International Conference on Computer Vision, pages 328-336.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504-3512.
Michal Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. *ICLR*.
Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra. 2016. Human attention in visual question answering: Do humans and deep networks look at the same regions? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 932-937.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2019. Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177. ACM.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556.
Souvik Kundu and Hwee Tou Ng. 2018. A question-focused multi-factor attention network for question answering. In Thirty-Second AAAI Conference on Artificial Intelligence.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117.
Zachary C Lipton. 2018. The mythos of model interpretability. Communications of the ACM, 61(10):36-43.
Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. 2017a. Attention correctness in neural image captioning. In Thirty-First AAAI Conference on Artificial Intelligence.
Shulin Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2017b. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1789-1798.
K Marimuthu and Sobha Lalitha Devi. 2012. How human analyse lexical indicators of sentiments-a cognitive analysis using reaction-time. In Proceedings of the 2nd Workshop on Sentiment Analysis where AI meets Psychology, pages 81-90.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101-1111.
Mark O Riedl. 2019. Human-centered artificial intelligence and machine learning. arXiv preprint arXiv:1901.11184.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Ying Sha and May D Wang. 2017. Interpretable predictions of clinical outcomes with an attention-based recurrent neural network. In Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pages 233-240. ACM.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 963-969.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489.
A.1 Training Rationale-based models
For the Rationale Neural Prediction Framework, we use the Pytorch implementation<sup>3</sup> suggested by Lei et al. (2016). In this framework, the encoder is built as Convolutional Neural Network (CNN) and the generator is built as Gumbel Softmax with independent selectors. The following hyper-parameters of CNN are used as pointed out by (Lei et al., 2016): 200 hidden dimensions, 0.1 dropout rate, 2 hidden layers, 128 batch size, 64 epochs, 0.0003 initial learning rate.
We conducted an extensive parameter search to find the optimum values for the two key hyper-parameters of the rationale model, selection-lambda, and continuity-lambda, which regularize the number and the continuity of words selected during the optimization process. For the selection lambda, we experimented with values 1, 1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7, 1e-8, 1e-9, and 0. For the continuity lambda, we experimented with values 0 and two times of selection lambda. We observe that the performance of the rationale-based model is extremely sensitive to its hyperparameters.
One conflicting interest with the rationale-based models is that the more words the model selects, the accuracy becomes higher. As our goal is to compare human attention with machine-generated attention for model interpretability, we optimize the model not only for accuracy but also for the number of selected rationales. We aim to generate roughly an equal number of words selected by both human annotators and machine-generated rationales.
A.2 Training Attention-based models
We used the following hyper-parameters to RNN-based models. 100 hidden dimensions, 100 attention size, 0.2 dropout rate, 128 batch size, 64 epochs, 0.0001 initial learning rate.
A.3 Additional Analysis Results
An example visualization of the attention maps annotated by human annotators and machine learning models is provided in Figure 4. The agreement between human annotators and all machine learning models can be considered high in this example, as there are many mutual selections.
Great place to go to have some drinks and food with a group of friends before heading out for the night. Service is mediocre and most of the menu is on the wall which was frustrating but the menu is very diverse. Really enjoyed the wagu carpaccio and the butterfish. Toro and uni were very fresh although the portions were small. One of the coolest things about this place is the fact that the pitchers of beer have a ice chamber in the center to keep your beer colder for longer! Genius! The wait can be long so come early.
Figure 3: Human attention is highly subjective. Some annotators tend to select only a few words, whereas others choose entire sentences.
Another example is provided in Figure 3, demonstrating the attention maps provided by two different annotators for the same review. This is an extreme example of the subjectivity of human attention. The first annotator only highlights individual words with the strongest cues of sentiment, whereas the second annotator sometimes selects entire sentences when they indicate a sentiment.
Table 5 shows the distribution of selected words over lexical categories for Human (CAM), Machine (bi-RNN), and the entire corpus for the Yelp-50 subset. Any divergence in the Human and Machine columns from the Corpus column indicates a tendency of selection for a lexical category. For example, adjectives are selected very heavily by both Human and Machine, even though they only make 0.02 of all words in the dataset.
<sup>3</sup>https://github.com/yala/text_nn
Lexical Category	Human	Machine(bi-RNN)	Corpus
Coordinating conjunction	0.0000	0.0098	0.0147
Cardinal number	0.0098	0.0077	0.0043
Determiner	0.0112	0.0168	0.0312
Existentialthere	0.0000	0.0000	0.0000
Foreign word	0.0000	0.0000	0.0000
Preposition or subordinating conjunction	0.0266	0.0084	0.0298
Adjective	0.2374	0.2269	0.0201
Adjective, comparative	0.0021	0.0014	0.0002
Adjective, superlative	0.0252	0.0287	0.0016
List item marker	0.0000	0.0000	0.0000
Modal	0.0035	0.0000	0.0030
Noun, singular or mass	0.3838	0.3711	0.0950
Noun, plural	0.0000	0.0000	0.0000
Proper noun, singular	0.0000	0.0000	0.0000
Proper noun, plural	0.0413	0.0665	0.0154
Predeterminer	0.0000	0.0000	0.0000
Possessive ending	0.0000	0.0000	0.0000
Personal pronoun	0.0056	0.0049	0.0141
Possessive pronoun	0.0035	0.0028	0.0067
Adverb	0.1296	0.0931	0.0277
Adverb, comparative	0.0070	0.0000	0.0014
Adverb, superlative	0.0000	0.0000	0.0000
Particle	0.0000	0.0000	0.0000
Symbol	0.0000	0.0000	0.0000
to	0.0035	0.0007	0.0077
Interjection	0.0000	0.0000	0.0000
Verb, base form	0.0196	0.0028	0.0098
Verb, past tense	0.0070	0.0609	0.0148
Verb, gerund or present participle	0.0357	0.0462	0.0053
Verb, past participle	0.0455	0.0455	0.0083
Verb, non-3rd person singular present	0.0000	0.0028	0.0023
Verb, 3rd person singular present	0.0007	0.0021	0.0065
Wh-determiner	0.0000	0.0000	0.0005
Wh-pronoun	0.0007	0.0000	0.0005
Possessive wh-pronoun	0.0000	0.0000	0.0000
Wh-adverb	0.0007	0.0007	0.0012
Table 5: Distribution over lexical categories for human-selected words, machine-selected words, and the entire corpus.
Stopped by on a Sunday around 11am after a trip to Freedom Park and had a lovely experience here- such cool ambiance and the staff was friendly and helpful. Chicken salad was good. The homemade pita chips were ok...a little thick for me. Others in our group enjoyed their food.
stopped by on a sunday around 11am after a trip to freedom park and had a lovely experience here such cool ambiance and the staff was friendly and helpful chicken salad was good the homemade pita chips were oka little thick for me others in our group enjoyed their food
Figure 4: Visualizations of attention maps by human annotators and machine learning models. From top to bottom: first human annotator, second human annotator, RNN, bi-RNN, Rationales.