SlowGuess/ACL_2020_OCR huggingface dataset:
Data Balancing. One critical aspect of our experimental design is our data balancing. We aim to prevent users from succeeding on our tests simply by guessing the true label for every instance. To do so, we ensure that true positives, false positives, true negatives, and false negatives are equally represented in the inputs. Likewise, for the counterfactual test, we sample perturbations such that for any instance, there is a  $50\%$  chance that the pertur
Method	Text Ratings	Tabular Ratings
n	μ	CI	σ	n	μ	CI	σ
LIME	144	4.78	1.47	1.76	130	5.36	0.63	1.70
Anchor	133	3.86	0.59	1.79	175	4.99	0.71	1.38
Prototype	191	4.45	1.02	2.08	144	4.20	0.82	1.88
DB	224	3.85	0.60	1.81	144	4.61	1.14	1.86
Composite	240	4.47	0.58	1.70	192	5.10	1.04	1.42
Table 3: User simulatability ratings by data domain, on a scale of 1 to 7. The mean and standard deviation for ratings are given by  $\mu$  and  $\sigma$ . The  $95\%$  confidence interval for the mean is given by CI, as calculated by bootstrap.
bation receives the same prediction as the original input. We confirm user understanding of the data balancing in our screening test.
Data Matching. Within each data domain, all users receive the same data points throughout the experiment. This design controls for any differences in the data across conditions and users, though this does reduce the information added by each test, making our confidence intervals relatively wide given the same sample size. We also match data across prediction rounds in order to control for the influence of particular data points on user accuracy between the Pre and Post phases.
4.4 Subjective Simulatability Ratings
Users see explanations in two phases of the tests: the second learning phase in the forward test, and the Post phase of the counterfactual test. In these stages, we ask users to give subjective judgments of the explanations. They rate each method on a 7 point Likert scale, in response to the question, "Does this explanation show me why the system thought what it did?" We explain that users should give higher ratings when the explanation shows the reasons for a model prediction, regardless of whether or not the prediction is correct.
We report data from a total of 2166 responses from 39 user tests. Each test is for a method and data domain pair, and contains either 16 or 32 task items, with some missingness due to users exiting the study early. In the results to follow, we use the term Change to refer to our estimate of explanation effectiveness: the difference in user accuracy across prediction phases in simulation tests. We perform two-sided hypothesis tests for this quantity by a block bootstrap, resampling both users and unique task items within each condition (Efron and Tibshirani, 1994). In addition, since users complete the first prediction round in either simulation test without access to explanations, we
estimate the mean Pre accuracy for each method with a random effects model. This allows us to share information across methods to yield more precise estimates of test performance.
Below, we analyze our experimental results and answer three questions: 1) Do explanations help users? 2) How do users rate explanations? 3) Can users predict explanation effectiveness?
5.1 Do explanations help users?
We show simulation test results in Tables 1 and 2. In Table 1, we group results by data domain, and in Table 2, we group results by test type.
Our principal findings are as follows:
1. LIME with tabular data is the only setting where there is definitive improvement in forward and counterfactual simulatability. With no other method and data domain do we find a definitive improvement across tests.
2. Even with combined explanations in the Composite method, we do not observe definitive effects on model simulatability.
3. Interestingly, our prototype method does reliably well on counterfactual simulation tests in both data domains, though not forward tests. It may be that the explanations are helpful only when shown side by side with inputs.
These results suggest that: (1) many explanation methods may not noticeably help users understand how models will behave, (2) methods that are successful in one domain might not work equally well in another, (3) combining information from explanations does not result in overt improvements in simulatability. Yet, given our wide confidence intervals, these results should be considered cautiously. It may also be that other methods do in fact improve simulatability, but we have not precisely estimated this. For example, our Prototype and Composite methods do the best on average with text data, though we cannot be confident that they improve simulatability.
Note that estimates of explanation effectiveness
could be influenced by users simply regressing to the mean accuracy between prediction rounds. We find that our primary results are not skewed by this phenomenon: the highest estimates of Change in each data domain and test type come from conditions where mean Pre test performance was either above the overall mean or, in one case, within 1.15 percentage points. This potential problem is further mitigated by our random effects model of Pre test performance, which pulls low Pre test means toward the overall mean.
5.2 How do users rate explanations?
It seems that, as intended, users rated explanations based on quality rather than model correctness, as we observe no significant difference in ratings grouped by model correctness (table in Appendix). In Table 3, we show user ratings for each method and data domain.
We observe that: 1) ratings are generally higher for tabular data, relative to text data, 2) the Composite and LIME methods receive the highest ratings in both domains, and 3) variance in explanation ratings is quite high, relative to their scale.
5.3 Can users predict explanation effectiveness?
We answer this question by measuring how explanation ratings relate to user correctness in the Post phase of the counterfactual simulation test. In this phase, users rate explanations of model predictions for an original input and predict model behavior for a perturbation of that input. If ratings of explanation quality are a good indicator of their effectiveness, we would expect to see that higher ratings are associated with user correctness.
We do not find evidence that explanation ratings are predictive of user correctness. We estimate the relationship via logistic regression with user correctness and ratings. We test models with both absolute ratings and ratings normalized within users, since ratings lack an absolute scale between users. With 640 text data points, we estimate with  $95\%$  confidence that moving from a rating of 4 to 5 is associated with between a  $-2.9$  and 5.2 percentage point change in expected user correctness. Using normalized ratings, we find that moving from the mean explanation rating to the first standard deviation is associated with between a  $-3.9$  and 12.2 percentage point change. With 515 tabular data points, we estimate that a change in rating from 4 to 5 is associated with between a  $-2.6$  and 5.3 per
centage point change in expected user correctness. Of course, we have not shown that there is no association. Yet it's important to note that if there is no relationship between user ratings and simulatability, then simply querying humans about explanation quality will not provide a good indication of true explanation effectiveness.
When do explanations succeed at improving user accuracy, and when do they fail at doing so? Below, we present example counterfactual test items, and we analyze how the explanations may have pointed to the reasons for model behavior.
6.1 Explanation Success Example
For the example below, 5 of 6 Post test responses for Prototype and LIME were correct that the model output did not change for the counterfactual, up from 3 of 6 in the Pre test.
Original  $(\hat{y} = pos)$ : "Pretty much sucks, but has a funny moment or two."
Counterfactual  $(\hat{y}_c = pos)$ : "Mostly just bothers, but looks a funny moment or two."
LIME identifies "funny" and "moment" as positive words, with weights adding to 1.04 after including the baseline. The notable negative word is "sucks"  $(w = -.23)$ , which changes to a similar word ("bothers"). All together, LIME suggests the prediction would stay the same since the positive words are unaffected and the only important negative word has a similar substitute.
The Prototype model gives the most activated prototype: "Murders by Numbers isn't a great movie, but it's a perfectly acceptable widget." It identifies "but" and "funny" as important words for the prototype's activation. The counterfactual is still similar to the prototype in key ways, suggesting the prediction would not change.
6.2 Explanation Failure Example
For the item below, only 7 of 13 responses were correct after seeing explanations, with no method improving correctness relative to the Pre test accuracy. Users needed to predict that the model prediction changed to negative for the counterfactual.
Original  $(\hat{y} = pos)$ : "A bittersweet film, simple in form but rich with human events."
Counterfactual  $(\hat{y}_c = neg)$ : "A teary film, simple in form but vibrant with devoid events."
Anchor gives one word as a condition for the original positive prediction: "bittersweet." But what happens when "bittersweet" changes to "teary"? The Anchor explanation does not actually apply to this counterfactual scenario, as its probabilistic description of model behavior is conditioned on the word bittersweet being present.
LIME gives five words, each with small weights  $(|w| < .04)$ , while the baseline is .91. This suggests that LIME has failed to identify features of the input that are necessary to the model output. Among these five words are the three that changed between sentences, but we would not suspect from their weights that the changes made in the counterfactual would flip the model output.
Decision Boundary gives a counterfactual input with a negative prediction: "A sappy film, simple in link but unique with human events." However, it is difficult to tell whether this counterfactual sentence is similar in decision-relevant ways to the proposed counterfactual sentence.
The Prototype model gives the activated prototype for the original prediction: "Watstein handily directs and edits around his screenplay's sappier elements...and sustains Off the Hook's buildup with remarkable assuredness for a first-timer." No important words are selected. We are left without a clear sense of why this was the most similar prototype and what circumstances would lead to the model output changing.
These examples reveal areas for improvement in explanations. Better methods will need to distinguish between sufficient and necessary factors in model behavior and clearly point to the ways in which examples share decision-relevant characteristics with new inputs. Further, they must do so in the appropriate feature space for the problem at hand, especially for models of complex data.
Forward Tests Stretch User Memory. We show users 16 examples during learning phases but do not allow them to reference the learning data during prediction phases. Reasonably, some users reported that it was difficult to retain insights from the learning phase during later prediction rounds.
Generating Counterfactual Inputs. It may be difficult to algorithmically construct counterfactual inputs that match the true data distribution, especially when seeking to change the model prediction. Our text counterfactuals are regularly out
of the data distribution, in the sense that no real movie review would exhibit the word choice they do. We still consider these inputs to be of interest, for the reason that a model will handle such inputs in some manner, and we aim to assess all possible model behaviors in our analysis.
Fair Comparison of Explanation Methods. In our forward simulation treatment phases, we provide users with 16 explained instances and allow them to read at their own pace. We control for the number of data points between methods, but one could instead control for user exposure time or computation time of explanation generation. Further, for LIME and Anchor, there are approaches for efficiently covering the space of inputs with a limited budget of examples (Ribeiro et al., 2018). We opt not to use them since 1) they are not applicable to the Decision Boundary and Prototype methods, which lack a similar notion of coverage, and 2) it is not clear whether these approaches are useful for text data. It may be that when using such approaches, LIME and Anchor perform better on forward simulation tasks.
Simulatability metrics give a quantitative measure of interpretability, capturing the intuition that explanations should improve a person's understanding of why a model produces its outputs. In this paper, we evaluated five explanation methods through simulation tests with text and tabular data. These are the first experiments to fully isolate the effect of algorithmic explanations on simulatability. We find clear improvements in simulatability only with LIME for tabular data and our Prototype method in counterfactual tests. It also appears that subjective user ratings of explanation quality are not predictive of explanation effectiveness in simulation tests. These results suggest that we must be careful about the metrics we use to evaluate explanation methods, and that there is significant room for improvement in current methods.
We thank the reviewers for their helpful feedback and our study users. This work was supported by NSF-CAREER Award 1846185, DARPA MCS Grant N66001-19-2-4031, a Royster Society PhD Fellowship, and Google and AWS cloud compute awards. The views contained in this article are those of the authors and not of the funding agency.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. 2019. Explaining a black-box using Deep Variational Information Bottleneck Approach. arXiv:1902.06918 [cs, stat]. ArXiv:1902.06918.
Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1036-1042, Brussels, Belgium. Association for Computational Linguistics.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. 2019. This Looks Like That: Deep Learning for Interpretable Image Recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).
Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608 [cs, stat]. ArXiv: 1702.08608.
Dheeru Dua and Casey Graff. 2017. UCI machine learning repository.
Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press.
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Overview of Interpretability of Machine Learning. The 5th IEEE International Conference on Data Science and Advanced Analytics (DSAA 2018).
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. 2019. Interpretable Image Recognition with Hierarchical Prototypes. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP-19), pages 32-40.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems.
Amanda Hutton, Alexander Liu, and Cheryl Martin. 2012. Crowdsourcing Evaluations of Classifier Interpretability. In AAAI Spring Symposium: Wisdom of the Crowd, pages 21-26.
Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. 2018. xGEMs: Generating Exemplars to Explain Black-Box Models. arXiv:1806.08867 [cs, stat]. ArXiv: 1806.08867.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In Proceedings of the  $35^{\text{th}}$  International Conference on Machine Learning.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and Understanding Neural Models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691. Association for Computational Linguistics.
Zachary C. Lipton. 2016. The Myth of Model Interpretability. 2016 ICML Workshop on Human Interpretability in Machine Learning.
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, pages 4765-4774.
Dong Nguyen. 2018. Comparing Automatic and Human Evaluation of Local Explanations for Text Classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1069-1078.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing - EMNLP '02, volume 10, pages 79-86. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?: Explaining the Predictions of Any Classifier. Knowledge Discovery and Data Mining (KDD).
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In AAAI Conference on Artificial Intelligence.
Cynthia Rudin. 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence, 1:206-215.
Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. 2018. ExplainGAN: Model Explanation via Decision Boundary Crossing Transformations. In ECCV 2018. Springer International Publishing.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical Attention Networks for Document Classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489. Association for Computational Linguistics.
A.1 Method Implementations
Explanation methods. For our tabular data, we use the implementations of Anchor and LIME provided in the code for Ribeiro et al. (2018). We implement our prototype and decision boundary methods. With text data, we use the implementation of Anchor provided by Ribeiro et al. (2018), and for LIME we use the code provided with Ribeiro et al. (2016). As before, we implement our prototype and decision boundary methods.
Text and Tabular Models. We train neural networks for both tasks as follows: for our tabular task model, we use a neural network with two hidden layers, each of width 50, as Ribeiro et al. (2018) do. For our text task model, we use a BiLSTM of the kind introduced by Yang et al. (2016), who reported state of the art results on a number of sentiment analysis tasks. Since their network is designed for classification of documents, we limit our network components to those relevant to classification of single sentences. We build our prototype models on top of the feature extractor layers of each of these models, meaning that we only replace the final classifier layer of the neural task model with a prototype layer. Accuracies for each model are shown in Table 4. The task models are trained with stochastic gradient descent and a cross-entropy loss function, using early stopping on a validation dataset and  $l_{2}$  regularization with
Model Accuracies
Data & Model	Test Acc
Text
Task Model	80.93
Prototype	80.64
Tabular
Task Model	83.49
Prototype	81.90
Table 4: Model accuracies on each data domain. Text data is split into partitions of  $70\%$ ,  $10\%$ , and  $20\%$  for the train, validation, and test sets, respectively. We use the same data processing scheme as Ribeiro et al. (2018) for tabular data.
Model Correctness	User Ratings
n	μ	CI	σ
Text
Correct	464	4.44	.49	1.89
Incorrect	468	4.12	.67	1.81
Tabular
Correct	391	5.09	.27	1.64
Incorrect	394	4.64	.27	1.69
Table 5: User simulatability ratings grouped by model correctness and data domain. Users do not seem to be rating explanations simply based on model correctness, as the differences in group means based on model correctness are not significant at a level of  $p < .05$ .
a coefficient of 1e-4. See training details for the prototype models below.
Prototype Model Training. Here we describe our prototype training algorithm, beginning with weight initialization. We initialize 1) feature extraction layers using the pretrained weights of our neural task model, 2) prototype vectors via k-means clustering on the latent representations of the entire training set, and 3) final classifier weights as 1 where the corresponding prototype's class matches the weight vector's class, and -.5 elsewhere. The objective function for our prototype models contains three terms: 1) a cross entropy loss, 2)  $l_{1}$  regularization on off-class weights in the classifier, and 3) a separation cost term, which is the minimum distance between a latent representation and any prototype not belonging to the input's class.
Importance Scores in Prototype Model. For a given feature, we compute an importance score by taking the difference in function output with that feature present in the input, relative to when that feature is omitted. With text data, there are a num
Figure 3: Forward simulation test procedure. We measure human users' ability to predict model behavior. We isolate the effect of explanations by first measuring baseline performance after users are shown examples of model behavior (Rounds 1, 2), and then measuring performance after they are shown explained examples of model behavior (Rounds 3, 4).
ber of mechanisms by which one can omit a word from an input; we opt for setting that word's embedding to the zero vector. For tabular data, to estimate a variable value's importance we compute a measure of evidence gain from knowing the value, relative to not knowing it. Formally, our importance function is the difference between the function value at the original input and the expected function value for the input with variable  $j$  removed. The expectation is taken over a distribution generated by an imputation model conditioned on the remaining covariates.
$$
\begin{array}{l} I m p o r t a n c e \left(\mathbf {x} _ {i, j}\right) = \\ f (\mathbf {x} _ {i}) - \mathbb {E} _ {p (\mathbf {x} _ {i, j} | \mathbf {x} _ {i, - j})} f (\mathbf {x} _ {i, - j} \cup \mathbf {x} _ {i, j}) \\ \end{array}
$$
where  $p(\mathbf{x}_{i,j}|\mathbf{x}_{i, - j})$  is given by a multinomial logistic regression fit to the training data, and  $\mathbf{x}_{i, - j}$  is the data point without feature  $j$ , and  $f(\mathbf{x}_{i, - j}\cup \mathbf{x}_{i,j})$  is the data point  $\mathbf{x}_{i, - j}$  with feature value  $\mathbf{x}_{i,j}$  imputed at index  $j$ . We choose to use logistic regressions with no feature engineering in order to 1) generate calibrated probability distributions, and 2) scale straightforwardly with dataset size.