SlowGuess/NAACL_2022_OCR huggingface dataset:
Language Model We fine-tuned BERT-base models to classify the relations for NYT10 and Webnlg datasets, respectively. We use the plain text as input  $X$ , and relations as multi-class label  $Y$  in the model fine-tuning. Since multiple relations may exist in single text, we use the Sigmoid activation in the output layer. Mean Square Error (MSE) is used as loss objective and Adam (Kingma and Ba, 2014) is adopted as the optimizer. The micro Precision, Recall and F1 results are reported in Table 1 with 0.5 threshold of output score. From the result, the F1 scores are high for both NYT10 and Webnlg dataset, hence we can apply feature attribution methods to the fine-tuned models and identify the important features in the text which help to classify the relations.
Table 1: Fine-tuned result on multi-label relation classification task.
Method	AUC
NYT	Webnlg
Rand	0.498 (0.143)	0.501 (0.121)
SimpleGrad	0.949 (0.071)	0.670 (0.135)
InputGrad	0.953 (0.061)	0.713 (0.120)
InteGrad	0.948 (0.077)	0.663 (0.126)
SmoothGrad	0.960 (0.064)	0.664 (0.142)
SHAP + Zero	0.805 (0.213)	0.670 (0.133)
SHAP + Ref.	0.872 (0.169)	0.675 (0.133)
LAFA	0.958 (0.060)	0.724 (0.115)
Table 2: Feature attribution result on Relation Classification model (Case I). Top two results are highlighted in bold.
Evaluation Metric In datasets, NYT10 and Webnlg, the positions of entities in triples are provided. Therefore, we can constructed the golden feature attribution label as follow. For text  $X = (w_{1}, w_{2}, \dots, w_{T})^{T}$  and triple  $(s, r, o)$ , where subject  $s = (w_{i}, \dots, w_{j})$  and object  $o = (w_{k}, \dots, w_{s})$  are words shown in the text from positions  $i$  to  $j$  and  $k$  to  $s$ , respectively. The gold labels of feature attribution for relation  $r$  is constructed as
$$
\mathcal {M} _ {g o l d} (X) = (0,.., 0, 1, .., 1, 0, .., 0, 1, .., 1,.. 0) ^ {T}
$$
where we set 1 from positions  $i$  to  $j$  as well as  $k$  to  $s$  and set 0 on other positions.
We use the evaluation metric Area under Curve (AUC) to compare the feature attribution  $\mathcal{M}(X)$
$^{2}$ https://github.com/slundberg/shap (MIT License)
Method	Pearson Correlation
SST-2	SST
Rand	0.039(0.074)	0.040(0.072)
SimpleGrad	0.441(0.083)	0.430(0.081)
InputGrad	0.456(0.080)	0.448(0.078)
InteGrad	0.468(0.071)	0.454(0.072)
SmoothGrad	0.484(0.073)	0.471(0.073)
SHAP + Zero	0.400(0.087)	0.392(0.085)
SHAP + Ref.	0.279(0.093)	0.278(0.091)
LAFA	0.494(0.070)	0.481(0.070)
Table 3: Feature attribution result on Sentiment Analysis Model (Case II).
and  $\mathcal{M}_{\mathrm{gold}}(X)$  for the test dataset. AUC ranges from 0 to 1, higher AUC represents the feature attribution result is closer to the gold feature attribution.
Main Results The results of AUC under different methods are summarized in Table 2. The popular feature attribution methods are listed and compared. More introduction about the competitors can be found in Section 4. "Rand", as a baseline method, denotes that the feature score is randomly assigned, therefore, the AUC score is about 0.5. InputGrad method performs better than the SimpleGrad, showing the effectiveness of Taylor approximation of layer-wise relevance propagation. "SHAP + Zero" means zero references are used in SHAP and "SHAP + Ref." means  $\mathcal{X}_{sim}$  is used as references. SHAP-based methods show low AUC values, because such methods aggregate the gradients of input and reference by simply taking average aggregation (see details in Section 4), which is not meaningful in NLP tasks. From the result, our method LAFA achieves a superior performance in Webnlg dataset and comparable performance in NYT dataset, which indicates that our feature attribution method can identify entities well.
5.2 Case II: Feature Attribution on Sentiment Analysis
Motivation The goal of the sentiment classification task is to classify a text into a sentiment categories such as positive or negative sentiment (Aghajanyan et al., 2021; Raffel et al., 2019; Jiang et al., 2020). In this use case, we hope to explain the deep sentiment classification model and obtain sentiment factors that drive the model to identify the sentiment.
Data The Stanford Sentiment Treebank (SST) (Socher et al., 2013) is a sentiment analysis dataset collected from English movie reviews (Pang and Lee, 2005). For all 9,645 sentences in SST, Amazon Mechanical Turk labeled the sentiment for words/phrases/sentences yielded from the Stanford Parser (Manning et al., 2014) on a scale between 1 and 25. SST-2 is first introduced by GLUE (Wang et al., 2018), a famous multi-task benchmark and analysis platform for natual language understanding, which took a subset from the SST and applied a two-way split (positive or negative) on sentence-level labels. Owing to the fact that the train/Validation/test split are aligned between SST and SST-2, we can run gradient-based methods on the either one of them. Note that we are only working with the test split for both data sets, which contains 2210 and 1821 sentences respectively.
Language Model We use a popular and publicly available Distill-BERT (Sanh et al., 2019) model which is fine-tuned on SST-2<sup>3</sup>. The accuracy of the Distill-BERT model on SST and SST-2 is  $86.6\%$  and  $92.4\%$  respectively.
Evaluation Metric We extract word-level sentiments from the phrase structure tree (PTB) in SST dataset. We take an absolute value after centralization to yield the golden label  $\mathcal{M}_{\mathrm{gold}}(X)$ . Pearson correlation coefficient,  $\rho$ , is the evaluation metric for feature attribution  $\mathcal{M}_{\mathrm{gold}}(X)$  and  $\mathcal{M}(X)$ . The correlation  $\rho$  takes value from the range from  $-1$  to  $1$ , and a higher  $\rho$  means better feature attribution result.
Main Results The main results of the correlation are summarized in Table 3. Popular feature attribution methods are listed and compared. To leverage the problem that some words can have opposite meaning when their sentiment are different, we only limited the sentences neighbor for same category. Based on the preliminary experiment, we choose the second layer with 10 neighbors and 0.39 cut-off rate, more details about preliminary experiment can be found in Appendix D that using all layers of DistilBERT as the encoder will improve the performance.
From the result Table 3, it is interesting to point out that the DistilBERT model fine-tuned on SST-2 does not perform equally well on the remaining sentences in SST, so the explanation we yield also
<sup>3</sup>https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
has lower correlation for all methods compared with the SST-2. Some example and analysis when LAFA works and fails in this dataset by showing neighbor sentences can be found in Appendix B.
We can find out that the InputGrad method outperforms SimpleGrad on SST/SST2 as well. SmoothGrad method achieves a good result by introducing random noise. From our observation, Sharpley-Value based methods, "SHAP + Zero" and "SHAP + Ref." can identify important features with a good chance but may include several irrelevant tokens leading to higher variance. Our method LAFA achieves a superior performance in larger average correlation and smaller variance on both SST-2 and SST data sets.
5.3 Case III: Feature Attribution on Regression Model
Motivation Amazon's online stores contain rich information about millions of products in product title, brand and description. We hope to better understand the trendy features that affecting price directly from such unstructured raw data, without the need for human labelers / data cleaning. In this application, we fine-tuned a deep language model with price as labels and aim to understand important factors from product descriptions with the given language model.
Data We collected the product catalog data of about one million products in personal computer category on Amazon's online store. We concatenate product's title, brand, bullet points and description as the input  $X$ , and use product price as the label  $Y$ .
Language Model We use BERT-base model and fine-tuned on collected catalog data for price regression.
Evaluation Metric To evaluate the performance of feature attribution methods without golden labels, we follow a similar idea as in work (Shrikumar et al., 2017; Lundberg and Lee, 2017) where the difference of prediction log-odds are measured by deleting pixels with highest importance scores. In our application, we first randomly select 200 input texts within a threshold of  $1\%$  prediction error as evaluation set. For each input text, we then mask  $\mathsf{p}\%$  of the tokens with highest feature attribution scores according to different feature attribution methods. Then we obtain new prediction result from the masked text denoted as  $\hat{y}_{\text {masked }}$  and
calculate the new mean absolute percentage error (MAPE). Higher value of MAPE means that the corresponding method excels in picking important features.
Figure 2: Feature attribution result on Case III. Comparisons of MAPE under different mask proportion.
Main Results The results are shown in Figure 2 where x-axis is the mask proportion  $p$ , and y-axis is MAPE. We observe that the random method has very low MAPE, because randomly masking the input texts will not affect the predicted result as much as the other feature attribution methods. ShapDeep and ShapGrad also have low MAPE values since simply taking average as aggregation is meaningless in NLP tasks. Other competing methods have similar performances on this case study and non of these performs better than others in a wide range of mask ratio. The proposed LAFA method outperforms other methods by significant margin with masking proportion from  $5\%$  to  $50\%$ , which demonstrates that smoothing over context-level neighbors helps to highlight the important features in similar type of products.
This paper presents a novel locally aggregated feature attribution method in NLP, which efficiently captures the important features by leveraging similar input texts in the embedding space. We focused on feature attribution of single input based on a fine-tuned model instead of training a language model, henceforth the computation time is of less concern.
One limitation of the LAFA model is that it requires informative neighbor sentences that carry similar information. Otherwise, aggregating information from other sentences could be misleading. Experiments in our datasets show that our method is effective, but the improvements gained from the
LAFA varies among different datasets based on the information that neighbor carries.
There are several future directions worthy of study. Firstly, labeling feature attribution result in the NLP requires massive human labor, and few datasets are available with golden feature attribution label. Developing new evaluation techniques to further measure model performance is interesting to investigate. Also, readable feature attribution results could help human beings to develop more business applications. For example, developing a key-value pair like processor-i5 as important feature can provide a more plausible feature attribution result to customers.
Armen Aghajanyan, Anchit Gupta, Akshit Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint arXiv:2101.11038.
Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. *PloS one*, 10(7):e0130140.
Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2016. Towards end-to-end reinforcement learning of dialogue agents for information access. arXiv preprint arXiv:1609.00777.
Xin Luna Dong, Xiang He, Andrey Kan, Xian Li, Yan Liang, Jun Ma, Yifan Ethan Xu, Chenwei Zhang, Tong Zhao, Gabriel Blanco Saldana, et al. 2020. Autoknow: Self-driving knowledge collection for products of thousands of types. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2724-2734.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planning. In 55th annual meeting of the Association for Computational Linguistics (ACL).
Thomas Hofmann, Bernhard Scholkopf, and Alexander J Smola. 2008. Kernel methods in machine learning. The annals of statistics, pages 1171-1220.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. arXiv preprint arXiv:1902.10186.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, Online. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In *Joint European Conference on Machine Learning and Knowledge Discovery in Databases*, pages 148-163. Springer.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626.
Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? arXiv preprint arXiv:1906.03731.
Lloyd S Shapley. 1953. A value for n-person games. Contributions to the Theory of Games, 2(28):307-317.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153. PMLR.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. 2017. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. arXiv preprint arXiv:1906.03158.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2019. A novel cascade binary tagging framework for relational triple extraction. arXiv preprint arXiv:1909.03227.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not explanation. arXiv preprint arXiv:1908.04626.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 506-514.
A Model Implementation Detail
All experiments are conducted with eight NVIDIA Tesla V100 GPUs with 2.5 GHz (base) and 3.1 GHz (sustained all-core turbo) Intel Xeon 8175M processors.
Case I For LAFA, we adopt the cosine function as the kernel function and hyper-parameter with  $\lambda = 1$  and SimpleGrad is implemented and aggregated by  $\mathcal{M}(x)$  in Equation (5).
Case II For LAFA, we adopt the Polynomial function as the kernel function  $k(\cdot ,\cdot) = \mathbb{I}(\cdot ,\cdot)$  and hyper-parameter with  $\lambda = 0.44$  and SmoothGrad is chosen and aggregated by  $\mathcal{M}(x)$  in Equation (5).
For gradient-based model with hyperparameters, we tuned them on the first 100 sentences in the test set. From the grid [10, 25, 50], we choose 25 as the integral iteration and the smooth candidates.
Case III The indicator function as the kernel function  $k(\cdot, \cdot) = \mathbb{I}(\cdot, \cdot)$  with  $\lambda = 1$  as hyperparameter is adopted for LAFA. Neighbor information is aggregated by  $\mathcal{M}(x)$  in Equation (5) from the SimpleGrad.
B Example of Neighbor Sentences found by Case Studies
B.1 Relation Extraction
In Figure 3, we show two examples in NYT and Webnlg with their neighbors. We can observe that detected neighbor sentences have a similar meaning, which can be utilized as a reference to help extract the key features from the original sentence.
Example of Neighbors for NYT Dataset :
Figure 3: Example of neighbors for NYT and Webnlg. The head and tail entities are highlighted with red color.
B.2 Sentiment Analysis
In SST-2, finding informative neighbors for every sentence is difficult because top sentences may not contain similar tokens, thus does not help. For this reason, we used a cut-off value for this data set to filter out non-informative sentences. In figure 4 we can find two examples from SST, one with "informative" good neighbors but another without them. Here for the word "informative" we use a quote because we are judging them based on our human understanding.
Figure 4: Example of neighbors for the SST data set. Sentiment factors found by SimpleGrad are highlighted with red color.
C Examples of Different Feature Attribution Methods under Multiple Cases
Here we provide an example in Cases I and II. In Figure 5, LAFA identified locations and "lived" as the important factors for relation extraction, and the importance of the "Atlantic City" and "Bader Field" is stronger than the backbone SimGrad because of aggregation.
D Experiment on Different Layer as Neighbor Encoder
Denote the size of  $\mathcal{X}_{sim}$  as  $M$ , the choice of which can be a critical and challenging task. Intuitively, an overly small  $M$  would lead to under-smoothing because the target text cannot incorporate enough information from the neighbors. On the contrary, an overly large  $M$  would cause over-smoothing by introducing too much noise.
Figure 5: Examples of Case Study I. Important factors are highlighted with red color.
Figure 6: Examples of Case Study III. Important features are highlighted with red color.
To clarify the neighbor searching process and the difference in the result using different layers, we show some experiments below.
Admittedly, we can directly use the WordPiece embedding as the encoder, which is the input of BERT-based models and enable us to find neighbors in the sense of "Word Similarity". However, since the same word can have different meanings in different sentences, and thus different importance in yielded gradients, we might need to use another layer in the BERT model as the encoder to incorporate contextual information.
We separate the layer search process into two cases depending on the availability of a set of labels that categorizes similar contents into the same group. Generally speaking, both cases recommend the middle layer as the encoder based on our experience.
D.1 When extra labels are not available
In the case of SST data, we do not have anything to group similar sentences, so we need to try for differ
Figure 7: Precision Result in Case Study III
ent possible layers and find the one that performs the best.
Here we fixed the max number of neighbors as 10 and uses 0.05 quantile of sampled similarities as the cut-off rate to filter those neighbors that are not "actually close". We use the SimpleGrad and the SmoothGrad as the baseline for comparison on the first 100 sentences in the test set.
From table A1 we can find out that LAFA is a generally good method that always beats the baseline when we use the smooth gradient as the basement method. Layer 2 performs the best among candidates. The combination of SmoothGrad and Layer  $w$  is the final choice and we showed the results on entire SST in the main result part. From here we can find out that for all seven layers, information from faithful neighbors can bring some useful information to an existing sentence.
Method	SST_first100	Method	SST_first100
SimpleGrad	0.457(0.074)	SmoothGrad	0.481(0.064)
SpG + LAFA + L1	0.457(0.073)	SmG + LAFA + L1	0.488(0.063)
SpG + LAFA + L2	0.458(0.072)	SmG + LAFA + L2	0.490(0.063)
SpG + LAFA + L3	0.456(0.072)	SmG + LAFA + L3	0.489(0.065)
SpG + LAFA + L4	0.456(0.072)	SmG + LAFA + L4	0.478(0.064)
SpG + LAFA + L5	0.454(0.072)	SmG + LAFA + L5	0.479(0.063)
SpG + LAFA + L6	0.454(0.073)	SmG + LAFA + L6	0.483(0.065)
SpG + LAFA + L7	0.456(0.074)	SmG + LAFA + L7	0.481(0.059)
Table A1: Feature Attribution Result on first 100 test cases in SST, using simple and smooth gradient as baselines
D.2 When we have extra label
The performance of encoders can be evaluated by the similarity between text  $X_0$  and similar texts  $\mathcal{X}_{sim}$  obtained from Equation (4). In this application, we use the product category or subcategory which is an additional source of labels produced by Amazon to construct a proxy metric to evaluate the
Figure 8: Comparisons for different kernel functions in Case III.
similarity. Define the metric of precision as:
$$
P r e c i s i o n = \frac {1}{M} \sum_ {j = 1} ^ {M} \mathbb {I} (c (X _ {j}) = c (X _ {0})), \tag {8}
$$
where  $c(\cdot)$  denotes the category or subcategory of the corresponding product,  $\mathbb{I}(\cdot)$  is the indicator function. A high precision represents that the text found  $\mathcal{X}_{sim}$  are similar to the text of interest  $X_0$ .