SlowGuess/ACL_2020_OCR huggingface dataset:
Comparison of metric instantiations We calculate PPL on the IWSLT  $\mathrm{De} \Rightarrow \mathrm{En}$  dataset for four metric instantiations (FN, RN, SA, Comb) and Baseline (Base) with  $k = 1$  to extract the most relevant words. Table 1 summarizes the results for two translation systems (TRANSFORMER annotated as Trans and RNN-SEARCH annotated as RNN), respectively. Note that since there is no target-side attention in RNN-SEARCH, we can not extract the best relevant target word, so Table 1 does not include the results of ATTN method for RNN-SEARCH.
The baseline (Base) achieves undesirable PPL which indicates the relevant words identified by PD failed to make the same decision as the NMT system. The main reason is that the mismatch between training and testing leads to the issue as presented in section §3.3. On the contrary, the other four metric instantiations attain much lower PPL than the Baseline. In addition, the PPLs on PD, NGRAD, and ATTN are much better than those on WGRAD. This finding shows that all PD, NGRAD, and ATTN are good explanation methods except WGRAD in terms of fidelity.
Density of generalizable rules To understand possible reasons for why one explanation method is better under our metric, we make a naive conjecture: when it tries to reveal the patterns that the
Table 1: The PPL comparison for the five metric instantiations on the IWSLT  $\mathrm{De}\Rightarrow \mathrm{En}$  dataset.
Method	Total	B1	B2	B3	B4	B5
ATTN	1.97M	1.65M	298K	23.7K	1.54K	104
PD	1.62M	1.25M	328K	31.2K	2.11K	108
NGRAD	1.89M	1.54M	326K	27.6K	1.64K	83
WGRAD	2.62M	2.37M	278K	17.5K	0.86K	34
Table 2: Density of the extracted rules from TRANSFORMER on the IWSLT  $\mathrm{De}\Rightarrow \mathrm{En}$ . The density is measured by the total number of unique rules and the number of rules with certain frequency in each interval  $B_{i}$ :  $B_{1} = (0,1]$ ,  $B_{2} = (1,10]$ ,  $B_{3} = (10,100]$ ,  $B_{4} = (100,1000]$ , and  $B_{4} = (1000,\infty)$ .
well-trained NMT has captured, it extracted more concentrated patterns. In other words, a generalized rule  $\mathcal{W}_{\phi}^{k}(c_{t})\to f(c_{t})$  from one sentence pair can often be observed among other examples.
To measure the density of the extracted rules, we first divide all extracted rules into five bins according to their frequencies. Then we collect the number of rules in each bin as well as the total number of rules. Table 2 shows the statistics to measure the density of rules obtained from different evaluation methods. From this table, we can see that the density for PD is the highest among those for all explanation methods, because it contains fewer infrequent rules in  $B_{1}$ , whereas there are more frequent rules in other bins. This might be one possible reason that PD is better under our fidelity-based evaluation metric.
Stability of ranking order In Table 1 the ranking order is PD > NGRAD > ATTN > WGRAD regarding all five metric instantiations. Generally, a good metric should preserve the ranking order of explanation methods independent of the test dataset. Regarding this criterion of order-preserving property, we analyze the stability of different fidelity-based metric instantiations. To this end, we randomly sample one thousand test data with replacement whose sizes are variant from  $1\%$  to  $100\%$  and then calculate the rate whether the ranking order is preserved on these test datasets. The results in Table 3 indicate that FN, RN, SA, Comb are more stable than Base to the change of distribution of test sets.
According to Table 1 and Table 3, SA performs similar to the best metric Comb and it is faster than Comb or RN for training and testing, thereby, in the rest of experiments, we mainly employ SA to measure evaluation methods.
Base	FN	SA	RN	Comb
1%	53.0%	97.1%	99.9%	99.8%	99.8%
5%	56.1%	100%	100%	100%	100%
20%	60.8%	100%	100%	100%	100%
50%	66.8%	100%	100%	100%	100%
100%	75.4%	100%	100%	100%	100%
Figure 1: PPL for each explanation method on TRANSFORMER over the IWSLT  $\mathrm{De} \Rightarrow \mathrm{En}$  dataset with different  $k$  value.
Effects on different  $k$  In this experiment, we examine the effects of explanation methods on larger  $k$  with respect to SA. Figure 1 depicts the effects of  $k$  for TRANSFORMER on  $\mathrm{De} \Rightarrow \mathrm{En}$  task. One can clearly observe two findings: 1) the ranking order of explanation methods is invariant for different  $k$ . 2) as  $k$  is larger, the PPL is much better for each explanation method. 3) the PPL improvement for PD, ATTN, and NGRAD is less after  $k > 2$ , which further validates that they are powerful in explaining NMT using only a few words.
Testing on other scenarios In the previous experiments, our metric instantiations are trained and evaluated under the same scenario, where  $c_t$  used to extract relevant words is obtained from gold data and its label  $f(c_t)$  is the prediction from NMT  $f$ , namely Teacher Forcing Decode. To examine the robustness of our metric, we apply the trained metric to two different scenarios: real decoding scenario (Real-Decode) where both  $c_t$  and its label  $f(c_t)$  are from the NMT output; and golden data scenario (Golden-Data) where both  $c_t$  and its label are from golden test data. The results for both scenarios are shown in Table 5.
From Table 5, we see that the ranking order for
Table 3: The rate (percentage) of sampled test dataset that have the same rankings as the test set on the IWSLT  $\mathrm{Zh}\Rightarrow \mathrm{En}$  dataset.
NMT	Methods	Zh→En	Fr→En
Base	SA	Base	SA
Trans	ATTN	897.1	30.8	359.6	12.1
PD	215.1	10.8	55.3	4.6
NGRAD	583.7	19	271.0	8.7
WGRAD	24126	180.9	44287	155.4
RNN	ATTN	-	-	-	-
PD	139.9	11.3	49.0	5.5
NGRAD	263.0	13.2	85.8	6.7
WGRAD	23068	243.1	50657	194.9
Table 4: The PPL comparison for two fidelity-based metric instantiations on two IWSLT datasets.
Methods	R-Dec	Golden	T-Dec
ATTN	11.5	57.1	13.8
PD	4.7	23.3	5.5
NGRAD	8.2	42.0	10.7
WGRAD	115.0	223.4	119.5
Table 5: Evaluating four explanation methods on 3 different scenarios Real-Decode (R-Dec), Golden-Data (Golden) and Teacher-Forcing Decode (T-Dec)) for TRANSFORMER over IWSLT  $\mathrm{De}\Rightarrow \mathrm{En}$  task.
both scenarios is the same as before. To our surprise, the results in Real-Decode are even better than those in the matched Teacher Forcing Decode scenario. One possible reason is that the labels generated by a NMT system in the Real-Decode tend to be high-frequency words, which leads to better PPL. In contrast, our metric instantiation in the Golden-Data results in much higher PPL due to the mismatch between training and testing. The performance of experimenting training and testing in the same scenario like Golden-Data can be experimented in future works, however, it's not the focus of this paper.
4.3 Scalability on WMT tasks
Since our metric such as SA requires to extract generalized rules for each explanation method from the entire training dataset, it is computationally expensive for some explanation methods such as gradient methods to directly run on WMT tasks with large scale training data.
Effects on sample size We randomly sample some subsets over WMT  $\mathrm{Zh}\Rightarrow \mathrm{En}$  training data that includes 22 million sentence pairs to form several new training sets. The sample sizes of the new training sets are set up to 2 million and the results are illustrated in Figure 2. The following facts are revealed. Firstly, the ranking order of
Figure 2: PPL for each explanation method on TRANSFORMER over WMT  $\mathrm{Zh} \Rightarrow \mathrm{En}$  task with different sample sizes.
Datasets	Methods	Base	SA
PPL	Rank	PPL	Rank
Zh⇒En	ATTN	336.4	2	27.3	3
PD	165.3	1	7.7	1
NGRAD	435.2	3	16.5	2
WGRAD	1615.5	4	263.5	4
De⇒En	ATTN	1862.3	2	17.0	3
PD	1118.2	1	5.4	1
NGRAD	2827.7	3	15.1	2
WGRAD	6678.1	4	197.4	4
Fr⇒En	ATTN	4271.0	3	41.1	3
PD	1646.6	1	4.1	1
NGRAD	2810.2	2	11.8	2
WGRAD	6703.8	4	163.7	4
four explanation methods remains unchanged with respect to different sample sizes. Secondly, with the increase of the sample size, the metric score decreases slower and slower and there is no significant drop from sampling 2 million sentence pairs to sampling 1 million.
Results on WMT With the analysis of effects on various sample sizes, we choose a sample size of 1 million for the following scaling experiments. The PPL results for WMT  $\mathrm{De}\Rightarrow \mathrm{En}$  ,  $\mathrm{Zh}\Rightarrow \mathrm{En}$  ,and  $\mathrm{Fr}\Rightarrow \mathrm{En}$  are listed in Table 6. We can see that the order  $\mathrm{PD} > \mathrm{NGRAD} > \mathrm{ATTN} > \mathrm{WGRAD}$  evaluated by SA still remains unchanged on these three datasets as before. One can observe that the ranking order under the baseline doesn't agree with SA on WMT  $\mathrm{De}\Rightarrow \mathrm{En}$  and  $\mathrm{Zh}\Rightarrow \mathrm{En}$  . Since the baseline yields in high PPL due to the mismatch we mentioned in section §3.3 ,in this case, we tend to trust
Table 6: The PPL and Ranking Order comparison between two fidelity-based metric instantiations (Base and SA) on three WMT datasets. “_” denotes the mismatch of ranking order.
Datasets	Methods	SA	Alignment
PPL	Rank	AER	Rank
IWSLT Zh⇒En	ATTN	30.8	3	55.0	3
PD	10.8	1	50.6	1
NGRAD	19	2	52.9	2
WGRAD	180.9	4	79.2	4
WMT Zh⇒En	ATTN	27.3	3	42.1	2
PD	7.7	1	32.7	1
NGRAD	16.5	2	49.3	3
WGRAD	263.5	4	79.2	4
WMT De⇒En	ATTN	17.0	3	48.7	3
PD	5.4	1	34.1	1
NGRAD	15.1	2	48.1	2
WGRAD	194.7	4	73.5	4
Table 7: Relation with word alignment. “_” denotes the mismatch of ranking order.
Figure 3: AER can not evaluate explanation methods on those target words "as a result of", which are not aligned to any word in the source sentence according to human annotation.
the evaluation results from SA that achieves lower PPL leading to better fidelity.
4.4 Relation to Alignment Error Rate
Since the calculation of the Alignment Error Rate (AER) requires manually annotated test datasets with ground-truth word alignments, we select three different test datasets contained such alignments for experiments, namely, IWSLT  $\mathrm{Zh}\Rightarrow \mathrm{En}$ , NIST05  $\mathrm{Zh}\Rightarrow \mathrm{En}^5$  and Zenkel  $\mathrm{De}\Rightarrow \mathrm{En}$  (Zenkel et al., 2019). Note that unaligned target words account for  $7.8\%$ ,  $4.7\%$ , and  $9.2\%$  on these three test sets respectively, which are skipped by AER for evaluating explanation methods. For example, in Figure 3, those target words 'as a result' cannot be covered by AER due to the impossibility of human annotation, but for a fidelity-based metric, they can be analyzed as well.
Table 7 demonstrates that our fidelity-based metric does not agree very well with AER on the WMT  $\mathrm{Zh}\Rightarrow \mathrm{En}$  task: NGRAD is better than ATTN in terms of SA but the result is opposite in terms of AER. Since the evaluation criteria of SA and AER are different, it is reasonable that their evaluation results are different. This finding is in line with
5https://www.ldc.upenn.edu/ collaborations/evaluations/nist
the standpoint by Jacovi and Goldberg (2020): SA is an objective metric that reflects fidelity of models while AER is a subject metric based on human evaluation. However, it is observed that the ranking by SA is consistent on all three tasks but that by AER is highly dependent on different tasks.
In recent years, explaining deep neural models has been a growing interest in the deep learning community, aiming at more comprehensible and trustworthy neural models. In this section, we mainly discuss two dominating ways towards it. One way is to develop explanation methods to interpret a target black-box neural network (Bach et al., 2015; Zintgraf et al., 2017). For example, on classification tasks, Bach et al. (2015) propose layer-wise relevance propagation to visualize the relationship between a pair of neurons within networks, and Li et al. (2016) introduce a gradient-based approach to understanding the compositionality in neural networks for NLP. In particular, on structured prediction tasks, many research works design similar methods to understand NMT models (Ding et al., 2017; Alvarez-Melis and Jaakkola, 2017; Ding et al., 2019; He et al., 2019).
The other way is to construct an interpretable model for the target network and then indirectly interpret its behavior to understand the target network on classification tasks (Lei et al., 2016; Murdoch and Szlam, 2017; Arras et al., 2017; Wang et al., 2019). The interpretable model is defined on top of extracted rational evidence and learned by model distillation from the target network. To extract rational evidence from the entire inputs, one either leverages a particular explanation method (Lei et al., 2016; Wang et al., 2019) or an auxiliary evidence extraction model (Murdoch and Szlam, 2017; Arras et al., 2017). Although our work focuses on evaluating explanation methods and does not aim to construct an interpretable model, we draw inspiration from their ideas to design  $Q \in \mathcal{Q}$  in Eq. (6) for our evaluation metric.
With the increasing efforts on designing new explanation methods, yet there are only a few works proposed to evaluate them. Mohseni and Ragan (2018) propose a paradigm to evaluate explanation methods for document classification that involves human judgment for evaluation. Poerner et al. (2018) conduct the first human-independent comprehensive evaluation of explanation meth
ods for NLP tasks. However, their metrics are task-specific because they make some assumptions for a specific task. Our work proposes a principled metric to evaluate explanation methods for NMT and our evaluation paradigm is independent of any assumptions as well as humans. It is worth noting that Arras et al. (2016); Denil et al. (2014) directly measure the performance of the target model  $P$  on the extracted words without constructing  $Q$  to evaluate explanation methods for classification tasks. However, since translation is more complex than classification tasks,  $P$  trained on the entire context  $c_{t}$  typically makes a terrible prediction when testing on the compressed context  $\mathcal{W}_{\phi}^{k}(c_{t})$ . As a result, the poor prediction performance makes it difficult to discriminate one explanation method from others, as observed in our internal experiments. Concurrently, Jacovi and Goldberg (2020) make a proposition to evaluate faithfulness of an explanation method separately from readability and plausibility (i.e., human-interpretability), which is similar to our definition of fidelity, but they do not formalize a metric or propose algorithms to measure it.
This paper has made an initial attempt to evaluate explanation methods from a new viewpoint. It has presented a principled metric based on fidelity in regard to the predictive behavior of the NMT model. Since it is intractable to exactly calculate the principled metric for a given explanation method, it thereby proposes an approximate approach to address the minimization problem. The proposed approach does not rely on human annotation and can be used to evaluate explanation methods on all target words. On six standard translation tasks, the metric quantitatively evaluates and compares four different explanation methods for two popular translation models. Experiments reveal that PD, NGRAD, and ATTN are all good explanation methods that are able to construct the NMT model's predictions with relatively low perplexity and PD shows the best fidelity among them.
We would like to thank all anonymous reviews for their valuable suggestions. This research was supported by Tencent AI Lab.
David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412-421, Copenhagen, Denmark. Association for Computational Linguistics.
Leila Arras, Franziska Horn, Gregoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. Explaining predictions of non-linear classifiers in nlp. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 1-7.
Leila Arras, Franziska Horn, Gregoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2017. "what is relevant in a text document?": An interpretable machine learning approach. PloS one, 12(8):e0181142.
Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. *PloS one*, 10(7):e0130140.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Osbert Bastani, Carolyn Kim, and Hamsa Bastani. 2017. Interpreting blackbox models via model extraction. arXiv preprint arXiv:1705.08504.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541.
Misha Denil, Alban Demiraj, and Nando De Freitas. 2014. Extraction of salient sentences from labelled documents. arXiv preprint arXiv:1412.6815.
Shuoyang Ding, Hainan Xu, and Philipp Koehn. 2019. Saliency-driven word alignment interpretation for neural machine translation. In Proceedings of WMT, page 1.
Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1150-1159.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243-1252. JMLR.org.
Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019. Towards understanding neural machine translation with word importance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 952-961, Hong Kong, China. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? ArXiv, abs/2004.03685.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675-1684. ACM.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas. Association for Computational Linguistics.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in NLP. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 681-691, San Diego, California. Association for Computational Linguistics.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293-1303, Florence, Italy. Association for Computational Linguistics.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eichiro Sumita. 2016. Neural machine translation with supervised attention. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3093-3102, Osaka, Japan. The COLING 2016 Organizing Committee.
Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, and Ting Liu. 2018. Distilling knowledge for search-based structured prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1393-1402, Melbourne, Australia. Association for Computational Linguistics.
Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2283-2288, Austin, Texas. Association for Computational Linguistics.
Sina Mohseni and Eric D Ragan. 2018. A human-grounded evaluation benchmark for local explanations of machine learning. arXiv preprint arXiv:1801.05075.
W James Murdoch and Arthur Szlam. 2017. Automatic rule extraction from long short term memory networks. In International Conference on Learning Representations.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Nina Poerner, Hinrich Schütze, and Benjamin Roth. 2018. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 340-350, Melbourne, Australia. Association for Computational Linguistics.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. CoRR, abs/1802.05668.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85, Berlin, Germany. Association for Computational Linguistics.
Vladimir N Vapnik. 1999. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988-999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Zhiguo Wang, Yue Zhang, Mo Yu, Wei Zhang, Lin Pan, Linfeng Song, Kun Xu, and Yousef El-Kurdi. 2019. Multi-granular text encoding for self-explaining categorization. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 41-45, Florence, Italy. Association for Computational Linguistics.
Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding interpretable attention to neural translation models improves word alignment. arXiv preprint arXiv:1901.11359.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595.