SlowGuess/ACL_2020_OCR huggingface dataset:
Datasets	Methods	LSTM	CNN	BERT
AOPC	Log-odds	AOPC	Log-odds	AOPC	Log-odds
SST	Leave-one-out	0.441	-0.443	0.434	-0.448	0.464	-0.723
CD	0.384	-0.382	-	-	-	-
LIME	0.444	-0.449	0.473	-0.542	0.134	-0.186
L-Shapley	0.431	-0.436	0.425	-0.459	0.435	-0.809
C-Shapley	0.423	-0.425	0.415	-0.446	0.410	-0.754
KernelSHAP	0.360	-0.361	0.387	-0.423	0.411	-0.765
SampleShapley	0.450	-0.454	0.487	-0.550	0.462	-0.836
HEDGE	0.458	-0.466	0.494	-0.567	0.479	-0.862
IMDB	Leave-one-out	0.630	-1.409	0.598	-0.806	0.335	-0.849
CD	0.495	-1.190	-	-	-	-
LIME	0.764	-1.810	0.691	-1.091	0.060	-0.133
L-Shapley	0.637	-1.463	0.623	-0.950	0.347	-1.024
C-Shapley	0.629	-1.427	0.613	-0.928	0.331	-0.973
KernelSHAP	0.542	-1.261	0.464	-0.727	0.223	-0.917
SampleShapley	0.757	-1.597	0.707	-1.108	0.355	-1.037
HEDGE	0.783	-1.873	0.719	-1.144	0.411	-1.126
Table 2: AOPCs and log-odds scores of different interpretation methods in explaining different models on the SST and IMDB datasets.
Log-odds. Log-odds score is calculated by averaging the difference of negative logarithmic probabilities on the predicted class over all of the test data before and after masking the top  $r\%$  features with zero paddings,
$$
\operatorname {L o g - o d d s} (r) = \frac {1}{N} \sum_ {i = 1} ^ {N} \log \frac {p (\hat {y} \mid \tilde {\boldsymbol {x}} _ {i} ^ {(r)})}{p (\hat {y} \mid \boldsymbol {x} _ {i})}. \tag {7}
$$
The notations are the same as in Equation 6 with the only difference that  $\tilde{\pmb{x}}_i^{(r)}$  is constructed by replacing the top  $r\%$  word features with the special token  $\langle \mathrm{pad}\rangle$  in  $\pmb {x}_i$ . Under this metric, lower log-odds scores are better.
Cohesion-score. We propose cohesion-score to justify an important text span identified by HEDGE. Given an important text span  $\pmb{x}_{(a,b]}$ , we randomly pick a position in the word sequence  $(x_{1},\ldots ,x_{a},x_{b + 1},\ldots ,x_{n})$  and insert a word back. The process is repeated until a shuffled version of the original sentence  $\bar{\pmb{x}}$  is constructed. The cohesion-score is the difference between  $p(\hat{y}\mid \pmb {x})$  and  $p(\hat{y}\mid \bar{\pmb{x}})$ . Intuitively, the words in an important text span have strong interactions. By perturbing such interactions, we expect to observe the output probability decreasing. To obtain a robust evaluation, for each sentence  $\pmb{x}_i$ , we construct  $Q$  different word sequences  $\{\bar{\pmb{x}}_i^{(q)}\}_{q = 1}^Q$  and compute the aver
age as
$$
\text {C o h e s i o n - s c o r e} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{Q} \sum_ {q = 1} ^ {Q} \left(p (\hat {y} \mid \boldsymbol {x} _ {i}) - p (\hat {y} \mid \bar {\boldsymbol {x}} _ {i} ^ {(q)})\right), \tag {8}
$$
where  $\bar{\pmb{x}}_i^{(q)}$  is the  $q^{\mathrm{th}}$  perturbed version of  $\pmb{x}_i$ ,  $Q$  is set as 100, and the most important text span in the contribution set  $\mathcal{C}$  is considered. Higher cohesion-scores are better.
4.2.1 Results
We compare HEDGE with several competitive baselines, namely Leave-one-out (Li et al., 2016), LIME (Ribeiro et al., 2016), CD (Murdoch et al., 2018), Shapley-based methods, (Chen et al., 2018, L/C-Shapley), (Lundberg and Lee, 2017, KernelSHAP), and (Kononenko et al., 2010, SampleShapley), using AOPC and log-odds metrics; and use cohesion-score to compare HEDGE with another hierarchical explanation generation method ACD (Singh et al., 2019).
The AOPCs and log-odds scores on different models and datasets are shown in Table 2, where  $k = r = 20$ . Additional results of AOPCs and log-odds changing with different  $k$  and  $r$  are shown in Appendix B. For the IMDB dataset, we tested on a subset with 2000 randomly selected samples due to computation costs. HEDGE achieves the best performance on both evaluation metrics. Sam
5583
Methods	Models	Cohesion-score
SST	IMDB
HEDGE	CNN	0.016	0.012
BERT	0.124	0.103
LSTM	0.020	0.050
ACD	LSTM	0.015	0.038
Table 3: Cohesion scores of HEDGE and ACD in interpreting different models on the SST and IMDB datasets. For ACD, we adopt the existing application from the original paper (Singh et al., 2019) to explain LSTM on text classification.
(a) HEDGE for LSTM on the SST.
(b) ACD for LSTM on the SST.
Figure 2: Compare HEDGE with ACD in interpreting the LSTM model on a negative movie review from the SST dataset, where LSTM makes a wrong prediction (POSITIVE). The importance scores of HEDGE and CD scores are normalized for comparison.
pleShapley also achieves a good performance with the number of samples set as 100, but the computational complexity is 200 times than HEDGE. Other variants, L/C-Shapley and KernelSHAP, applying approximations to Shapley values perform worse than SampleShapley and HEDGE. LIME performs comparatively to SampleShapley on the
LSTM and CNN models, but is not fully capable of interpreting the deep neural network BERT. The limitation of context decomposition mentioned by Jin et al. (2019) is validated by the worst performance of CD in identifying important words. We also observed an interesting phenomenon that the simplest baseline Leave-one-out can achieve relatively good performance, even better than HEDGE when  $k$  and  $r$  are small. And we suspect that is because the criteria of Leave-one-out for picking single keywords matches the evaluation metrics. Overall, experimental results demonstrate the effectiveness of Equation 5 in measuring feature importance. And the computational complexity is only  $\mathcal{O}(n)$ , which is much smaller than other baselines (e.g. SampleShapley, and L/C-Shapley with polynomial complexity).
Table 3 shows the cohesion-scores of HEDGE and ACD with different models on the SST and IMDB datasets. HEDGE outperforms ACD with LSTM, achieving higher cohesion-scores on both datasets, which indicates that HEDGE is good at capturing important phrases. Comparing the results of HEDGE on different models, the cohesion-scores of BERT are significantly higher than LSTM and CNN. It indicates that BERT is more sensitive to perturbations on important phrases and tends to utilize context information for predictions.
For qualitative analysis, we present two typical examples. In the first example, we compare HEDGE with ACD in interpreting the LSTM model. Figure 2 visualizes two hierarchical explanations, generated by HEDGE and ACD respectively, on a negative movie review from the SST dataset. In this case, LSTM makes a wrong prediction (POSITIVE). Figure 2(a) shows HEDGE correctly captures the sentiment polarities of bravura and emptiness, and the interaction between them as bravura exercise flips the polarity of emptiness to positive. It explains why the model makes the wrong prediction. On the other hand, ACD incorrectly marks the two words with opposite polarities, and misses the feature interaction, as Figure 2(b) shows.
In the second example, we compare HEDGE in interpreting two different models (LSTM and BERT). Figure 3 visualizes the explanations on a positive movie review. In this case, BERT gives the correct prediction (POSITIVE), while LSTM makes
5584
(a) HEDGE for LSTM on SST.
(b) HEDGE for BERT on SST.
Figure 3: Compare HEDGE in interpreting different models (LSTM and BERT) on a positive movie review from the SST dataset, where BERT makes the correct prediction (POSITIVE), while LSTM makes a wrong prediction (NEGATIVE). HEDGE explains that BERT captures the important phrase not a bad for making the correct prediction, while LSTM ignores it and is misled by the negative word bad.
a wrong prediction (NEGATIVE). The comparison between Figure 3(a) and 3(b) shows the difference of feature interactions within the two models and explains how a correct/wrong prediction was made. Specifically, Figure 3(b) illustrates that BERT captures the key phrase not a bad at step 1, and thus makes the positive prediction, while LSTM (as shown in Figure 3(a)) misses the interaction between not and bad, and the negative word bad pushes the model making the NEGATIVE prediction. Both cases show that HEDGE is capable of explaining model prediction behaviors, which helps humans understand the decision-making. More examples are presented in Appendix C due to the page limitation.
We had 9 human annotators from the Amazon Mechanical Turk (AMT) for human evaluation. The features (e.g., words or phrases) with the highest importance score given by HEDGE and other baselines are selected as the explanations. Note that HEDGE and ACD can potentially give very long top features which are not user-friendly in human evaluation, so we additionally limit the maximum length of selected features to five. We provided the input text with different explanations in the user interface (as shown in Appendix D) and asked human annotators to guess the model's prediction (Nguyen, 2018) from {"Negative", "Positive", "N/A"} based on each explanation, where "N/A" was selected when annotators cannot guess the model's prediction. We randomly picked 100 movie reviews from the IMDB dataset for human evaluation.
There are two dimensions of human evaluation. We first compare HEDGE with other baselines using the predictions made by the same LSTM model. Second, we compare the explanations generated by HEDGE on three different models: LSTM, CNN, and BERT. We measure the number of human annotations that are coherent with the actual model predictions, and define the coherence score as the ratio between the coherent annotations and the total number of examples.
4.4.1 Results
Table 4 shows the coherence scores of eight different interpretation methods for LSTM on the IMDB dataset. HEDGE outperforms other baselines with higher coherence score, which means that HEDGE can capture important features which are highly consistent with human interpretations. LIME is still a strong baseline in providing interpretable explanations, while ACD and Shapley-based methods perform worse. Table 5 shows both the accuracy and coherence scores of different models. HEDGE succeeds in interpreting black-box models with relatively high coherence scores. Moreover, although BERT can achieve higher prediction accuracy than the other two models, its coherence score is lower, manifesting a potential tradeoff between accuracy and interpretability of deep models.
In this paper, we proposed an effective method, HEDGE, building model-agnostic hierarchical interpretations via detecting feature interactions. In
Methods	Coherence Score
Leave-one-out	0.82
ACD	0.68
LIME	0.85
L-Shapley	0.75
C-Shapley	0.73
KernelSHAP	0.56
SampleShapley	0.78
HEDGE	0.89
Table 4: Human evaluation of different interpretation methods with LSTM model on the IMDB dataset.
Models	Accuracy	Coherence scores
LSTM	0.87	0.89
CNN	0.90	0.84
BERT	0.97	0.75
Table 5: Human evaluation of HEDGE with different models on the IMDB dataset.
this work, we mainly focus on sentiment classification task. We test HEDGE with three different neural network models on two benchmark datasets, and compare it with several competitive baseline methods. The superiority of HEDGE is approved by both automatic and human evaluations.
David Alvarez-Melis and Tommi S Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In NeurIPS.
Kaylee Burns, Aida Nematzadeh, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Exploiting attention to reveal shortcomings in memory models. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 378-380.
Jianbo Chen and Michael I Jordan. 2019. Ls-tree: Model interpretation when the data are linguistic. arXiv preprint arXiv:1902.04187.
Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. 2018. L-shapley and c-shapley: Efficient model interpretation for structured data. arXiv preprint arXiv:1808.02610.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced LSTM for natural language inference. arXiv preprint arXiv:1609.06038.
Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic transparency via quantitative input influence:
Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP), pages 598-617. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Katsushige Fujimoto, Ivan Kojadinovic, and Jean-Luc Marichal. 2006. Axiomatic characterizations of probabilistic and cardinal-probabilistic interaction indices. Games and Economic Behavior, 55(1):72-99.
Reza Ghaeini, Xiaoli Z Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. arXiv preprint arXiv:1808.03894.
Frédéric Godin, Kris Demuynck, Joni Dambre, Wesley De Neve, and Thomas Demeester. 2018. Explaining character-aware neural networks for word-level prediction: Do they discover linguistic rules? arXiv preprint arXiv:1808.09551.
Michel Grabisch. 1997. K-order additive discrete fuzzy measures and their representation. Fuzzy sets and systems, 92(2):167-189.
Yotam Hechtlinger. 2016. Interpretation of prediction models using the input gradient. arXiv preprint arXiv:1611.07634.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4555-4564.
Alon Jacovi, Oren Sar Shalom, and Yoav Goldberg. 2018. Understanding convolutional neural networks for text classification. arXiv preprint arXiv:1809.08037.
Xisen Jin, Junyi Du, Zhongyu Wei, Xiangyang Xue, and Xiang Ren. 2019. Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models.
Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? on the ability of lstms to understand negative polarity items. arXiv preprint arXiv:1808.10627.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
5586
Igor Kononenko et al. 2010. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11(Jan):1-18.
Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 121-126.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. 2018. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142-150. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
W James Murdoch, Peter J Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. arXiv preprint arXiv:1801.05453.
Dong Nguyen. 2018. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1069-1078.
Guillermo Owen. 1972. Multilinear extensions of games. Management Science, 18(5-part-2):64-79.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. 2018. Model agnostic supervised local explanations. In Advances in Neural Information Processing Systems, pages 2515-2524.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144. ACM.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660-2673.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? arXiv preprint arXiv:1906.03731.
Lloyd S Shapley. 1953. A value for n-person games. Contributions to the Theory of Games, 2(28).
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3145-3153. JMLR.org.
Chandan Singh, W. James Murdoch, and Bin Yu. 2019. Hierarchical interpretations for neural network predictions. In International Conference on Learning Representations.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Erik Strumbelj and Igor Kononenko. 2014. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3):647-665.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319-3328. JMLR.org.
Michael Tsang, Youbang Sun, Dongxu Ren, and Yan Liu. 2018. Can i trust you more? model-agnostic hierarchical explanations. arXiv preprint arXiv:1812.04801.
5587
A Comparison between Top-down and Bottom-up Approaches
Given the sentence a waste of good performance for example, Figure 4 shows the hierarchical interpretations for the LSTM model using the bottom-up and top-down approaches respectively. Figure 4(a) shows that the interaction between waste and good can not be captured until the last (top) layer, while the important phrase waste of good can be extracted in the intermediate layer by top-down algorithm. We can see that waste flips the polarity of of good to negative, causing the model predicting negative as well. Top-down segmentation performs better than bottom-up in capturing feature interactions. The reason is that the bottom layer contains more features than the top layer, which incurs larger errors in calculating interaction scores. Even worse, the calculation error will propagate and accumulate during clustering.
(a) Bottom-up clustering.
(b) Top-down segmentation.
B Results of AOPCs and log-odds changing with different  $k$  and  $r$
(a) AOPCs of LSTM on the SST dataset.
Figure 4: Hierarchical interpretations for the LSTM model using the bottom-up and top-down approaches respectively. Red and blue colors represent the negative and positive sentiments respectively.
(b) Log-odds of LSTM on the SST dataset.
Figure 5: The AOPC and log-odds for LSTM on the SST dataset.
5588
(a) AOPCs of LSTM on the IMDB dataset.
(a) AOPCs of CNN on the SST dataset.
(b) Log-odds of LSTM on the IMDB dataset.
(b) Log-odds of CNN on the SST dataset.
Figure 6: The AOPC and log-odds for LSTM on the IMDB dataset.
Figure 7: The AOPC and log-odds for CNN on the SST dataset.
(a) AOPCs of CNN on the IMDB dataset.
(a) AOPCs of BERT on the SST dataset.
(b) Log-odds of CNN on the IMDB dataset.
(b) Log-odds of BERT on the SST dataset.
Figure 8: The AOPC and log-odds for CNN on the IMDB dataset.
Figure 9: The AOPC and log-odds for BERT on the SST dataset.
(a) AOPCs of BERT on the IMDB dataset.
(b) Log-odds of BERT on the IMDB dataset.
C Visualization of Hierarchical Interpretations
Figure 11: HEDGE for BERT on a positive movie review from the SST dataset. BERT makes the correct prediction because it captures the interaction between never and fails.
Figure 12: HEDGE for LSTM on a positive movie review from the SST dataset. LSTM makes the wrong prediction because it misses the interaction between never and fails.
Figure 10: The AOPC and log-odds for BERT on the IMDB dataset.
Figure 13: ACD for LSTM on a positive movie review from the SST dataset, on which LSTM makes wrong prediction.
Figure 14: HEDGE for BERT on a positive movie review from the SST dataset, on which BERT makes correct prediction.
5591
Figure 15: HEDGE for LSTM on a positive movie review from the SST dataset, on which LSTM makes wrong prediction.
Figure 16: ACD for LSTM on a positive movie review from the SST dataset, on which LSTM makes wrong prediction.
D Human Evaluation Interface
5592
[Example] Not a bad movie at all!
Figure 17: Interfaces of Amazon Mechanical Turk where annotators are asked to guess the model's prediction based on different explanations.
5593