SlowGuess/NAACL_2021_OCR huggingface dataset:
Baselines We compare GMASK with four baseline methods: (1) LIME (Ribeiro et al., 2016)-fitting a local linear model with perturbations to approximate the neural network and produce word attributions; (2) L2X (Chen et al., 2018)-constructing a network to learn feature attributions by maximizing the mutual information between the selected features and model output; (3) IBA (Per-Sample framework) (Schulz et al., 2020) - learning feature attributions by optimizing the information bottleneck which restricts feature information flow by adding noise; (4) IMASK (subsection 3.3)-learning individual word masks. Note that here we use standalone IMASK as one of the baselines, as oppose to applying it for selecting preliminary important words for GMASK as in subsection 3.3.
More details about experimental setup are in Appendix B, including data pre-processing and model configurations.
We compare the faithfulness of generated post-hoc explanations via both quantitative and qualitative evaluations.
5.1 Quantitative Evaluation
We adopt three metrics from prior work to evaluate the faithfulness of learned feature attributions: AOPC score (Nguyen, 2018; Samek et al., 2016), post-hoc accuracy (Chen et al., 2018; Chen and Ji,
Models	Methods	e-SNLI	Quora	QQP	MRPC
DAttn	LIME	0.286	0.120	0.079	0.064
L2X	0.299	0.128	0.079	0.035
IBA	0.354	0.137	0.104	0.109
IMASK	0.324	0.140	0.087	0.064
GMASK	0.361	0.142	0.095	0.091
BERT	LIME	0.221	0.153	0.110	0.062
L2X	0.310	0.119	0.134	0.083
IBA	0.282	0.199	0.144	0.114
IMASK	0.292	0.232	0.139	0.130
GMASK	0.319	0.309	0.181	0.200
Table 2: AOPC scores of different explanation methods with the DAttn and BERT models on the four datasets.
2020), and degradation score (Ancona et al., 2017; Schulz et al., 2020). We evaluate explanations on all test data for the MRPC dataset, and on 2000 examples randomly selected from the test set for other three datasets due to computational complexity. The average runtime is in Appendix D.
5.1.1 AOPC score
We adopt the area over the perturbation curve (AOPC) (Nguyen, 2018; Samek et al., 2016) metric to evaluate the comprehensiveness of explanations to models. It calculates the average change of prediction probability on the predicted class over all examples by removing top  $1 \ldots u$  words in explanations.
$$
\operatorname {A O P C} = \frac {1}{U + 1} \left\langle \sum_ {u = 1} ^ {U} p (y | \boldsymbol {x}) - p \left(y \mid \boldsymbol {x} \backslash 1 \dots u\right) \right\rangle_ {\boldsymbol {x}}, \tag {9}
$$
where  $p(y|x_{\backslash 1\dots u})$  is the probability for the predicted class when words  $1\dots u$  are removed and  $\langle \cdot \rangle_{x}$  denotes the average over all test examples. Higher AOPC score indicates better explanations.
Table 2 shows the results of AOPC scores of different explanation methods when  $U = 10$ . GMASK outperforms other baseline methods on most of the datasets. Especially for the BERT model, GMASK achieves significantly higher AOPC scores than other methods, indicating that BERT tends to rely on word correlations to make predictions. IBA and IMASK, either learning continuous or binary individual word masks, perform better than learning word attributions via an additional network (L2X) or using linear approximation (LIME).
5.1.2 Post-hoc Accuracy
The post-hoc accuracy (Chen et al., 2018; Chen and Ji, 2020) evaluates the sufficiency of important
<sup>3</sup>https://gluebenchmark.com/
(a) DAttn on e-SNLI
(b) DAttn on Quora
(c) DAttn on QQP
(d) DAttn on MRPC
(e) BERT on e-SNLI
(f) BERT on Quora
(g) BERT on QQP
(h) BERT on MRPC
Figure 3: Post-hoc accuracy of different explanation methods with the DAatn and BERT models on the four datasets.
(a) LIME
Figure 4: Degradation test of different explanation methods with the DAattn model on the e-SNLI dataset.
(b) L2X
(c) IBA
(d) IMASK
(e) GMASK
words to the model prediction. For each test data, we select top  $v$  important words based on word attributions for the model to make a prediction, and compare it with the original prediction made on the whole input text. We compute the post-hoc accuracy on  $M$  examples,
$$
\operatorname {p o s t - h o c - a c c} (v) = \frac {1}{M} \sum_ {m = 1} ^ {M} \mathbb {1} \left[ y _ {v} ^ {(m)} = y ^ {(m)} \right],
$$
where  $y^{(m)}$  is the predicted label on the m-th test data, and  $y_{v}^{(m)}$  is the predicted label based on the top  $v$  important words. Higher post-hoc accuracy indicates better explanations.
Figure 3 shows the results of post-hoc accuracy of different explanation methods where we increase  $v$  from 1 to 10. Similar to the results of the AOPC scores, GMASK achieves higher post-hoc accuracy than other methods for both DAatn and BERT models.
The explanations of GMASK for the BERT model achieve about  $80\%$  post-hoc accuracy on all datasets except the MRPC dataset. This is only by
relying on top 4 important words, which means that GMASK captures informative words for model predictions. The post-hoc accuracies of the BERT model on the MRPC dataset are lower than those on other three datasets because the average sentence length of MRPC is twice as long as the others, indicating that BERT tends to use larger context for predictions. The post-hoc accuracies of the DAttn model on the MRPC dataset are extremely high for all the explanation methods. The reason is that the prediction accuracy of DAttn model on the MRPC dataset is relatively low (Table 1). Any random words picked up by explanations could make the model output wrong predictions since the original predictions on the whole texts are also wrong, hence causing high post-hoc accuracy.
5.1.3 Degradation Test
Degradation test (Ancona et al., 2017; Schulz et al., 2020) evaluates the ranking of importance by removing the most important words or least important words first, and observing model prediction probability drop on the predicted class. We draw
Models	Methods	Texts
DAttn	LIME	a man playing an electric guitar on stage . a man playing banjo on the floor .
L2X	a man playing an electric guitar on stage . a man playing banjo on the floor .
IBA	a man playing an electric guitar on stage . a man playing banjo on the floor .
IMASK	a man playing an electric guitar on stage . a man playing banjo on the floor .
GMASK	a man playing an electric guitar on stage . a man playing banjo on the floor .
BERT	LIME	why are vikings portrayed wearing horned helmets ? why did vikings have horns on their helmets ?
L2X	why are vikings portrayed wearing horned helmets ? why did vikings have horns on their helmets ?
IBA	why are vikings portrayed wearing horned helmets ? why did vikings have horns on their helmets ?
IMASK	why are vikings portrayed wearing horned helmets ? why did vikings have horns on their helmets ?
GMASK	why are vikings portrayed wearing horned helmets ? why did vikings have horns on their helmets ?
Table 3: Examples of different explanations, where the top four important words are highlighted. The important words in the first and second sentences are highlighted in pink and blue colors respectively. The color saturation indicates word attribution. The first example is from the e-SNLI dataset, and the DAatn model makes a correct prediction as CONTRADICTION. The second example is from the Quora dataset, and the BERT model makes a correct prediction as PARAPHRASES.
Models	Methods	e-SNLI	Quora	QQP	MRPC
DAttn	LIME	0.502	0.070	0.091	1.367
L2X	0.366	0.002	0.036	1.779
IBA	0.423	0.110	0.197	2.775
IMASK	0.436	0.152	0.214	2.037
GMASK	0.620	0.178	0.238	2.790
BERT	LIME	0.188	0.192	0.087	0.018
L2X	0.303	0.168	0.173	-0.003
IBA	0.166	0.038	0.176	0.050
IMASK	0.369	0.303	0.172	0.251
GMASK	0.576	0.726	0.707	0.533
Table 4: Degradation scores of different explanation methods with the DAtn and BERT models on the four datasets.
two curves as shown in Figure 4, one with the most relevant words removed first (MoRF) and another one with the least relevant words removed first (LeRF). x-axis is the percentage of words removed (degradation proportion), and y-axis is the normalized model output probability as
$$
S \left(\boldsymbol {x} _ {\rho}\right) = \frac {p (y \mid \boldsymbol {x} _ {\rho}) - p (y \mid \boldsymbol {x} _ {o})}{p (y \mid \boldsymbol {x}) - p (y \mid \boldsymbol {x} _ {o})}, \tag {10}
$$
where  $\pmb{x}$  is the original input,  $y$  is the predicted label,  $\pmb{x}_{\rho}$  means  $\rho \% (\rho \in [0,100])$  degradation of  $\pmb{x}$ , and  $\pmb{x}_o$  is full degradation. We compute the averages of  $p(y|\pmb{x}_{\rho})$ ,  $p(y|\pmb{x})$ , and  $p(y|\pmb{x}_o)$  over all test examples. The degradation score is calculated as the integral between the MoRF and LeRF curves,
$$
\text {d e g r a - s c o r e} = \int_ {\rho = 0} ^ {1 0 0} \frac {S ^ {L} \left(\boldsymbol {x} _ {\rho}\right) - S ^ {M} \left(\boldsymbol {x} _ {\rho}\right)}{1 0 0} d \rho , \tag {11}
$$
where  $S^L(\boldsymbol{x}_{\rho})$  and  $S^M(\boldsymbol{x}_{\rho})$  are normalized model outputs by removing the least or most important
words respectively. Higher degradation score is better.
Table 4 shows the results of degradation scores of different explanation methods. GMASK shows superiority to other baseline methods under this metric. Figure 4 shows the degradation test results of DAatn model on the e-SNLI dataset. GMASK can distinguish both important and unimportant words, while IBA does not learn the correct order of unimportant words. LIME does not perform well in identifying important words, but captures the correct order of unimportant words. The MoRF and LeRF curves of L2X and IMASK are relatively symmetric, but not as expanded as GMASK.
5.2 Qualitative Evaluation
Table 3 shows different explanations on two examples from e-SNLI and Quora respectively. See Appendix E for more examples. For the first example, the DAttn model makes a correct prediction as CONTRADICTION. For the second example, the BERT model also makes a correct prediction as PARAPHRASES. We highlight the top four important words, where the words in the first and second sentences are in pink and blue colors respectively. The color saturation indicates word attribution.
For the first example, LIME and IBA mainly capture the important words from the first sentence, while ignoring the ones in the second sentence (e.g. banjo, floor). On the contrary, L2X focuses on the words in the second sentence, while ignoring the important word guitar in the first sentence. IMASK picks up two irrelevant words man and a as important words, which can not explain the model prediction. GMASK correctly identifies top
four important words and captures two correlated words guitar and banjo from the two input sentences respectively.
For the second example, only GMASK captures the two important correlated words horned and horns, which explains why the BERT model predicts the two input questions as paraphrases. LIME captures the overlapped word helmets in the two sentences, while L2X only selects some irrelevant words. Both IBA and IMASK identify a question mark as the important word, which is untrustworthy to the model prediction.
In this paper, we focused on sentence pair modeling and proposed an effective method, GMASK, learning group masks for correlated words and calculating weighted word attributions. We tested GMASK with two different neural network models on four datasets, and assessed its effectiveness via both quantitative and qualitative evaluations.
The motivation of this work is aligned with the merits of explainable AI, in which the goal is to increase the trustworthiness of neural network models in decision making. One potential ethical concern of this work is that explanations can be used to design adversarial examples for attacking. Although the main focus of this work is about generating faithful explanations, we do realize the importance of whether human users can actually understand explanations. To address this concern, a better strategy is to collaborate with HCI experts in our future work. In addition, we provide necessary implementation details to make sure the results in this paper are reproducible.
Marco Ancona, Enea Ceolini, Cengiz Özireli, and Markus Gross. 2017. A unified view of gradient-based attribution methods for deep neural networks. In NIPS Workshop on Interpreting, Explaining and Visualizing Deep Learning.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Oana-Maria Camburu, Tim Roktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, pages 9539-9549.
Hanjie Chen and Yangfeng Ji. 2020. Learning variational word masks to improve the interpretability of neural text classifiers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4236-4251, Online. Association for Computational Linguistics.
Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. 2020. Generating hierarchical explanations on text classification via feature interaction detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5578-5593, Online. Association for Computational Linguistics.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018. Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning, pages 883-892. PMLR.
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243-3255, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Mengnan Du, Ninghao Liu, Fan Yang, Shuiwang Ji, and Xia Hu. 2019. On attribution of recurrent neural network predictions via additive decomposition. In The World Wide Web Conference, pages 383-393.
Reza Ghaeini, Xiaoli Z Fern, and Prasad Tadepalli. 2018. Interpreting recurrent and attention-based neural models: a case study on natural language inference. arXiv preprint arXiv:1808.03894.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1-42.
Yotam Hechtlinger. 2016. Interpretation of prediction models using the input gradient. arXiv preprint arXiv:1611.07634.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. arXiv preprint arXiv:1902.10186.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Xisen Jin, Junyi Du, Zhongyu Wei, Xiangyang Xue, and Xiang Ren. 2019. Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models. arXiv preprint arXiv:1911.06194.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. 2018. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pages 4765-4774.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712.
W James Murdoch, Peter J Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. arXiv preprint arXiv:1801.05453.
Dong Nguyen. 2018. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1069-1078.
Ankur P Parikh, Oscar Tackström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kocisky, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660-2673.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. 2020. Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? arXiv preprint arXiv:1906.03731.
Lloyd S Shapley. 1953. A value for n-person games. Contributions to the Theory of Games, 2(28).
Chandan Singh, W James Murdoch, and Bin Yu. 2018. Hierarchical interpretations for neural network predictions. arXiv preprint arXiv:1806.05337.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319-3328. JMLR.org.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. arXiv preprint arXiv:1904.10717.
Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057.
Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. 2020. Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. arXiv preprint arXiv:2006.10966.
Michael Tsang, Youbang Sun, Dongxu Ren, and Yan Liu. 2018. Can i trust you more? model-agnostic hierarchical explanations. arXiv preprint arXiv:1812.04801.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. arXiv preprint arXiv:1908.04626.
Wenpeng Yin and Hinrich Schütze. 2015. Convolutional neural network for paraphrase identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 901-911.
A Sampling with Gumbel-softmax Trick
For a random variable  $A \in \{1, \dots, t\}$  which follows categorical distribution with probabilities  $[\lambda_1, \dots, \lambda_t]$ . We draw samples from a Gumbel(0, 1) distribution for each category  $\iota \in \{1, \dots, t\}$ :
$$
s _ {l} = - \log (- \log u), u \sim \operatorname {U n i f o r m} (0, 1), \tag {12}
$$
and then apply a temperature-dependent softmax over the  $t$  categories with each dimension calculated as
$$
a _ {s _ {\iota}} = \frac {\exp \left(\left(\log \left(\lambda_ {\iota}\right) + s _ {\iota}\right) / \tau\right)}{\sum_ {\iota} \exp \left(\left(\log \left(\lambda_ {\iota}\right) + s _ {\iota}\right) / \tau\right)}, \tag {13}
$$
where  $\tau$  is a hyperparameter called the softmax temperature.
B Supplement of Experiment Setup
Datasets The datasets are all in English. Table 5 shows the statistics of the four datasets. We adopt the data splits of e-SNLI (Camburu et al., 2018) from the ERASER benchmark  ${}^{4}$  . We adopt the data splits of Quora released by Wang et al. (2017). The data splits of QQP (Wang et al., 2018) and MRPC (Dolan and Brockett, 2005) are from the GLUE benchmark. We clean up the text by converting all characters to lowercase, removing extra whites-paces and special characters, and build vocabulary.
Models We set the hidden size of feed forward networks in the DAattn model (Parikh et al., 2016) as 300, and initialize word embeddings with pretrained fastText (Bojanowski et al., 2017). For BERT model (Devlin et al., 2018), we use the pretrained BERT-base model<sup>5</sup> with 12 transformer layers, 12 self-attention heads, and the hidden size of 768.
We implement the models in PyTorch 3.6. The number of parameters in the DAttn and BERT models are 11046303 and 109484547 respectively. We fine-tune hyperparameters manually for each model to achieve the best prediction accuracy, such as learning rate  $lr \in \{1e - 4, 1e - 3, \dots, 1\}$ , clipping norm  $clip \in \{1e - 3, 1e - 2, \dots, 1, 5, 10\}$ .
The corresponding validation accuracy for each reported test accuracy is in Table 6.
D Average Runtime
The average runtime of each approach for each model on each dataset is recorded in Table 7. All experiments were performed on a single NVidia GTX 1080 GPU. Note that L2X is efficient in generating explanations for test data, but it costs more time on training the interpretation model on the whole training set.
E Examples of Different Explanations
Table 8 shows more examples of different explanations for the DAatn and BERT model on different datasets.
4https://www.eraserbenchmark.com/
<sup>5</sup>https://github.com/huggingface/
pytorch-transformers
Datasets	C	L	V	#train	#dev	#test
e-SNLI	3	10.2	64291	549K	9K	9K
Quora	2	11.5	85249	384K	10K	10K
QQP	2	11.1	126266	364K	40K	391K
MRPC	2	22	15547	3668	408	1725
Table 5: Summary statistics for the datasets, where  $C$  is the number of classes,  $L$  is average sentence length,  $V$  is vocab size, and # counts the number of examples in the train/dev/test sets.
Models	e-SNLI	Quora	QQP	MRPC
DAtn	87.75	87.36	87.19	73.77
BERT	90.43	91.21	91.31	86.52
Table 6: The validation accuracy (%) of different models on the four datasets.
Models	Methods	e-SNLI	Quora	QQP	MRPC
DAttn	LIME	20.17	20.45	20.43	19.12
L2X	0.01	0.02	0.02	0.01
IBA	9.87	9.93	9.96	9.24
IMASK	0.17	0.19	0.18	0.12
GMASK	11.10	11.23	11.25	10.96
BERT	LIME	13.41	14.01	14.08	13.21
L2X	0.02	0.02	0.02	0.01
IBA	3.29	3.38	3.34	3.20
IMASK	0.62	0.70	0.71	0.59
GMASK	43.24	43.56	43.77	42.84
Table 7: The average runtime (s/example) of each approach for each model on each dataset.
Table 8: Examples of different explanations for the DAttn and BERT model on different datasets.