SlowGuess/NAACL_2022_OCR huggingface dataset:
Sheng Zhang<sup>1</sup> Jin Wang<sup>2</sup> Haitao Jiang<sup>3</sup> Rui Song<sup>2,3</sup>
$^{1}$ AWS AI Labs,  $^{2}$ Amazon Core AI
$^{3}$ Department of Statistics, North Carolina State University
{zshe, jiwngn}@amazon.com, {hjiang24, rsong}@ncsu.edu
With the growing popularity of deep-learning models, model understanding becomes more important. Much effort has been devoted to demystify deep neural networks for better interpretability. Some feature attribution methods have shown promising results in computer vision, especially the gradient-based methods where effectively smoothing the gradients with reference data is key to a robust and faithful result. However, direct application of these gradient-based methods to NLP tasks is not trivial due to the fact that the input consists of discrete tokens and the "reference" tokens are not explicitly defined. In this work, we propose Locally Aggregated Feature Attribution (LAFA), a novel gradient-based feature attribution method for NLP models. Instead of relying on obscure reference tokens, it smoothes gradients by aggregating similar reference texts derived from language model embeddings. For evaluation purpose, we also design experiments on different NLP tasks including Entity Recognition and Sentiment Analysis on public datasets as well as key feature detection on a constructed Amazon catalogue dataset. The superior performance of the proposed method is demonstrated through experiments.
With the growing popularity of deep-learning models, model understanding becomes more and more critical in many folds. In one aspect, model understanding helps us understand what the model is doing by identifying crucial features among unstructured raw data. For example, Shrikumar et al., 2017 utilized the model explainability technique to discover motifs in regulatory DNA elements from distinct molecular signatures in the field of Genomics. In another aspect, model understanding helps people audit or debug the deep models. An interesting example is that Ribeiro et al. (Ribeiro et al., 2016) found that their image classification model sometimes misclassifies a husky as a wolf.
The model explainability tool reveals that their model relies on the snow in the background rather than the appearance when distinguishing the two animals. More importantly, model understanding helps gain trust when making important decisions based on the model. In the NLP domain, deep language models are quickly evolving and show superior performance in various benchmark tasks. However, even experts struggle to understand the mechanism of complex language models.
Much effort has been devoted to demystifying the "black box" of deep models. A natural idea is through feature attribution, explaining the model by attributing the prediction to each input feature according to how much it affects the model output, of which two main directions emerge. One is model agnostic approaches including Shapley regression values (Shapley, 1953) and LIME (Ribeiro et al., 2016). We can apply these methods regardless of the model structure, however, they could suffer from computational inefficiency in the scenario of high dimensional input space and complex deep models when making inferences across all possible permutations or with small perturbations in the local neighborhood.
Another direction is model-specific approaches which look into the internal model mechanism to understand specific models. Gradient-based feature attribution models are often adopted to explain neural networks since gradients can be easily accessed through back-propagation, which gives a great computational advantage over model-agnostic methods. Since the gradient map itself is often noisy and challenging to interpret, most gradient-based methods aim to stabilize the feature attribution score by smoothing the gradients or learning from the reference data (Sundararajan et al., 2017; Smilkov et al., 2017; Lundberg and Lee, 2017). However, direct application of these gradient-based methods to NLP problems is not trivial, due to the fact that the input consists of discrete tokens and the "reference"
Human Language Technologies, pages 2189 - 2201
tokens are not explicitly defined.
In this paper, we propose Locally Aggregated Feature Attribution (LAFA), a novel gradient-based approach that leverages sentence-level embedding as a smoothing space for the gradients, motivated by the observation that the feature attribution is often shared by similar text inputs. For example, key features in product descriptions on an online marketplace are often shared by similar products. We implement a neighbor-searching method to ensure the quality of neighboring sentences.
Furthermore, to evaluate feature attribution methods in NLP, we consider two situations. For datasets with golden labels of feature score, we use the Area Under Curve (AUC) or Pearson correlation as the performance metric. As for datasets without golden labels, we conduct a similar evaluation task following prior works (Shrikumar et al., 2017; Lundberg and Lee, 2017) by masking tokens with high importance scores and find the change in the predicted log-odds.
In summary, our contributions are threefold: First, we build a novel context-level smooth gradient approach for feature attribution in NLP. The key ingredients of our method are constructing an appropriate aggregation function over the smoothing space. Second, to the best of our knowledge, this is the first proposal to conduct numerical studies on multiple NLP tasks, including Entity Recognition and Sentiment Analysis, for feature attribution. Third, our method achieves superior performance compared with the state-of-the-art feature attribution methods.
The paper is organized as follows. Section 2 elaborates the current challenges of feature attribution in NLP and recaps the preliminaries about gradient-based feature attribution approaches. The proposed feature attribution method is described in section 3, followed by a review of other existing approaches in Section 4. The evaluation tasks and the application results on NLP are presented in Section 5.
2 Feature Attribution in NLP
Challenge Direct application of gradient-based methods to NLP problems is not trivial. There are three main challenges. First, NLP models consist of non-differentiable discrete input tokens, hence the gradient hook can only reach out to the embedding space and gradient-based feature attribution methods are not directly applicable to word tokens.
Second, the reference data in NLP are difficult to define. It is studied by Sundararajan. et al (Sundararajan et al., 2017) that using the gradient as the feature attribution may suffer from the problems of model saturation or thresholding. Model saturation means the perturbation of some elements in the input cannot change the output, and the thresholding problem indicates discontinuous gradients can produce misleading importance scores. Such problems can be addressed by comparing the difference between the gradient of input and reference data. The guiding principle to select reference data is to ask ourselves that "what am I interested in measuring differences against?" For example, in the tasks of binary classification on DNA sequence inputs, the reference data are chosen as the expected frequencies of DNA sequence or randomly shuffling the original sequence. However, in NLP tasks, randomly shuffling texts as reference may not be grammatically sensible.
Lastly, we note that the evaluation of the language model is much more challenging than the explanations of the images. In the image application, the important features of an image obtained from feature attribution methods can be visually validated by checking the composition of objects. However, the detected important features in language may require more domain knowledge to validate.
Problem Definition Feature attribution task can be formally formulated as follows. A deep model  $\mathcal{F}$  is provided to be explained, which is fine-tuned on dataset  $\mathcal{X}$ . The input sentence is denoted as  $X_0 = (w_1, w_2,.., w_T)^T$  where  $w_i$  represents  $i$ -th word. The goal for feature attribution is to determine function  $\mathcal{M}(\cdot)$  by quantifying the importance score of each word  $\mathcal{M}(x) = (m_1, m_2,.., m_T)^T$ , where  $m_i$  denotes the importance score for  $w_i$ .
Simple Gradient as Feature Attribution As illustrated in the first challenge above, in NLP models, directly taking derivative on each word is infeasible due to the non-differentiable embedding layer. We can resolve the challenge as follows.
The fist layer of the NLP model usually maps input discrete tokens to embedding from a predefined dictionary.
$$
h _ {0, i} = \operatorname {e m b} \left(w _ {i}\right), i = 1, 2,.., T, \tag {1}
$$
where  $h_{0,i}\in \mathbb{R}^d$  represents the word embedding for  $w_{i}$ . This step is non-derivative. But we can
Figure 1: Upper panel shows the overview of LAFA methods; Lower panel provides a motivating example of LAFA method. In this motivating example, the input text is a description of computer. The key features of the computer should include "Brand", "CPU type" and "RAM size". The simple gradient method may miss certain feature, such as "RAM size" in the example, while the gradients on similar texts can provide more contexts. The proposed method is constructed to aggregate the information from similar texts summarized in Algorithm 1.
obtain the derivative of output with respect to the word embedding:
$$
\mathcal {S} \left(H _ {0}\right) = \partial \mathcal {F} / \partial H _ {0} \in \mathbb {R} ^ {T \times d}, \tag {2}
$$
where  $H_0 = (h_{0,1}, h_{0,2}, \dots, h_{0,T})^T \in \mathbb{R}^{T \times d}$ . Then, we consider the feature attribution score of a token  $\mathcal{M}(X) \in \mathbb{R}^T$  as the sum of squares of the gradients with regard to each word embedding dimension:
$$
\mathcal {M} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} \left(H _ {0}\right) _ {i, j} ^ {2}, i = 1, 2,.., T. \tag {3}
$$
However, simply using the gradients of one token as feature attribution would lead to noisy results (Sundararajan et al., 2017). The next section describes a novel feature attribution approach that smoothes the gradients by leveraging similar input texts.
3 The Proposed Framework: LAFA
The proposed method contains three steps: (1) find the appropriate neighbors of the input text for gradient smoothing; (2) calculate the gradients of texts as well as neighbors; (3) aggregation of the gradients. The proposed framework is summarized in the upper panel of Figure 1. One motivating example is shown in the lower panel of Figure
1. In this motivating example, the input text is a description of computer. The key features of the computer should include "Brand", "CPU type" and "RAM size". The simple gradient method may miss certain feature, such as "RAM size" in the example, while the gradients on similar texts can provide more contexts. The proposed method is constructed to aggregate the information from similar texts.
Step I: Context-level Localization Given the input text  $X_0 \in \mathcal{X}$ , where  $\mathcal{X}$  denotes the input datasets, the goal is to find similar texts  $\mathcal{X}_{sim} = \{X_1, X_2,.., X_M\} \subset \mathcal{X}$  such that the feature attributions of  $X_0$  and  $X_j \in \mathcal{X}_{sim}$  are similar under a defined similarity metric.
To obtain similar texts  $\mathcal{X}_{sim}$ , we first define an encoder that maps the text with discrete word tokens to a continuous embedding vector; then, in the embedding space, similar texts are found in the neighbor of  $X_0$ . To be specific, let  $\mathcal{H}_{encoder}$  denote the mapping from input to one of the hidden layers in deep model  $\mathcal{F}$ .  $\mathcal{X}_{sim}$  can be obtained by choosing closest texts in the dataset as follows:
$$
X \in \mathcal {X}
$$
$$
s. t. \left\| \mathcal {H} _ {\text {e n c o d e r}} (X) - \mathcal {H} _ {\text {e n c o d e r}} \left(X _ {0}\right) \right\| _ {2} <   \epsilon \tag {4}
$$
where  $||\cdot ||_2$  represents  $L^2$  norm.  $\epsilon$  is a threshold score to guarantee that founded neighbors are simi
lar to the center text  $X_0$  to improve the faithfulness of aggregation. In our application, a fixed quantile served as the cut-off rate of L2 distance for all possible pairs is chosen as the threshold score to filter the nearest-neighbor result. During inference time, we apply the hidden layer encoder  $\mathcal{H}_{\text {encoder }}$  to all the input datasets and index, then using FAISS 1 (Johnson et al., 2017) offline. FAISS is an efficient, open-source library for similarity search and clustering on dense vectors, which can be applied to large-scale vectors.
The output of this step,  $\mathcal{X}_{sim}$  can be viewed as the reference data to smooth the feature attribution of  $X_0$ , which addresses the second challenge listed in Section 2.
Step II: Taking Gradients According to Equation (3), the gradient of  $X_{i}$  can be denoted as  $\mathcal{M}(X_i)\coloneqq (m_{i,0},m_{i,1},\dots,m_{i,T_i})^T$  for  $i = 0,1,\ldots ,M$  where  $T_{i}$  represent the token length of  $X_{i}$ . To be noticed that our proposed method can be easily extended to variants of simple gradient including smooth gradient or integrated gradient methods (Smilkov et al., 2017; Sundararajan et al., 2017) in Step II.
Step III: Aggregation over Multiple Feature Attribution Our goal is to smooth the gradient  $\mathcal{M}(X_0)$  by aggregating the gradients of similar text inputs:
$$
\mathcal {M} _ {L A F A} \left(X _ {0}\right) = A G G R E G A T E (\mathcal {M} \left(X _ {0}\right);
$$
$$
\mathcal {M} (X _ {1}),.., \mathcal {M} (X _ {M}))
$$
Since the lengths of  $X_0, X_1, \ldots, X_M$  may vary, the lengths of gradients  $\mathcal{M}(X_0), \mathcal{M}(X_1), \ldots, \mathcal{M}(X_M)$  are different as well. Consequently, aggregation by simply taking the average is infeasible. Following the intuition that, the tokens with high gradients in  $\mathcal{X}_{sim}$  should be important in  $X_0$ , we propose the following aggregation function:
$$
\begin{array}{l} \mathcal {M} _ {L A F A} (X _ {0}) = \mathcal {M} (X _ {0}) + \lambda (\mathcal {E} (w _ {0, 1}; \mathcal {X} _ {s i m}),.., \\ \mathcal {E} \left(w _ {0, T}; \mathcal {X} _ {s i m}\right) ^ {T}, \tag {6} \\ \end{array}
$$
where  $\lambda$  is a hyper-parameter for leveraging the feature attribution from similar inputs.  $\mathcal{E}(w;\mathcal{X}_{sim})$  is a scalar representing the importance of token  $w$  obtained from the neighbor inputs  $\mathcal{X}_{sim}$ . Formally,
it can be defined as
$$
\mathcal {E} (w; \mathcal {X} _ {s i m}) = \frac {1}{| \mathcal {X} _ {s i m} |} \sum_ {i = 1} ^ {| \mathcal {X} _ {s i m} |} \sum_ {k = 1} ^ {T _ {i}} \frac {m _ {i , k} \times k (h , h _ {i , k})}{T _ {i}}, \tag {7}
$$
where  $h$ ,  $h_{i,k}$  are the word embedding of  $w$  and  $w_{i,k}$  as in Equation (1) respectively, and  $k(\cdot ,\cdot)$  is a kernel function (Hofmann et al., 2008) (examples of kernel function are listed in the Appendix E.). According to Equation (7), if word  $w$  and  $w_{i,k}$  have a high similarity, then inner product between the embeddings  $h$  and  $h_{i,k}$  in the kernel space would be large, which would assign a large weight to the corresponding importance score  $m_{i,k}$ . On the contrary, dissimilar word  $w_{i,k}$  in  $\mathcal{X}_{sim}$  has little effect to the word  $w$  in  $\mathcal{E}(w;\mathcal{X}_{sim})$ . The whole process is summarized in Algorithm 1.
Algorithm 1 Feature attribution method with smoothing over similar inputs.
1: Input: Text of interest  $X_0$ , input datasets  $\mathcal{X}$  and fine-tuned deep model  $\mathcal{F}$ .
2: Output: Feature attribution of  $X_0$
3: Step I: Localization
4: Construct encoder  $\mathcal{H}$  which maps from input space to the space of hidden layer in  $\mathcal{F}$ .
5: Obtain the similar texts set  $\mathcal{X}_{sim} = \{X_1, X_2,.., X_M\}$  of  $X_0$  according to Equation (4).
6: Step II: Taking Gradient
7: Calculate the gradient of texts  $X_0, X_1, \ldots, X_M$  according to Equation (3).
8: Step III: Aggregation
9: Smooth the gradient of text of interest over the gradient of similar texts according to Equation (6).
10: Output the aggregated gradient  $\mathcal{M}_{LAFA}(X_0)$  as the feature attribution.
Discussion of Faithfulness One important criteria for model explainability method is "faithfulness", which refers to how accurately it reflects the true reasoning process of the model (Jacovi and Goldberg, 2020). In our proposed method, the original input  $X_0$  is infused with similar texts in the input dataset  $\mathcal{X}$  for better interpretation. Since the deep model  $\mathcal{F}$  is also trained on  $\mathcal{X}$ , using similar texts  $\mathcal{X}_{sim} \subset \mathcal{X}$  to facilitate explanation will not violate the faithfulness.
In the localization step (Step I), out of the consideration about faithfulness, we do not use popular
'https://github.com/facebookresearch/ faiss (MIT license)
bi-encoder frameworks, such as S-BERT (Reimers and Gurevych, 2019) or DenseRetrival (Karpukhin et al., 2020), to obtain similar neighbors. Because it will involve an extra black box model when explaining deep model  $\mathcal{F}$ .
In NLP, transformer-based models yield great successfulness and some works focus on explaining the attention mechanism. For example, Serrano and Smith, 2019 and Jain and Wallace, 2019 inspected a single attention layer and found out that attention weights only weakly and inconsistently correspond to feature importance; Wegreffe and Pinter, 2019 argued that we cannot separate the attention layer and should view the entire model as a whole. In this section, We mainly review the gradient-based methods for feature attribution.
Feature Attribution on Single Input Simonyan et al (Simonyan et al., 2013) computed the "saliency map" denoted as Simple Gradient from the derivative of the output with respect to the input in an image classification task. In the NLP application, "saliency map" is obtained as the derivative of the output with respect to the word embedding as in Equation (2). However, "saliency map" can be visually noisy. Several methods are proposed to improve the gradient method from different perspectives. Gradient*Input method (Shrikumar et al., 2017) improves the visual sharpness of the "saliency map" by multiplying gradient with the input itself. In NLP, we can write it as:
$$
\mathcal {S} _ {G r a d * I n p u t} (H _ {0}) = H _ {0} \times \mathcal {S} (H _ {0})
$$
$$
\mathcal {M} _ {G r a d * I n p u t} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} _ {G r a d * I n p u t} (H _ {0}) _ {i, j} ^ {2}.
$$
Layerwise Relevance Propagation method (Bach et al., 2015) is shown to be equivalent to the Gradient*Input method up to a scaling factor. Smooth Gradient method (Smilkov et al., 2017) smoothes the feature attribution score by adding random noises to the input and taking average of the gradients from noisy inputs, formally:
$$
\begin{array}{l} \mathcal {S} _ {\text {S m o o t h G r a d}} (H _ {0}) \approx \frac {1}{N} \sum_ {k = 1} ^ {N} \mathcal {S} (H _ {0} + \epsilon_ {k}), \\ \epsilon_ {k} \sim N (0, \sigma^ {2}), \\ \mathcal {M} _ {S m o o t h G r a d} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} _ {S m o o t h G r a d} (H _ {0}) _ {i, j} ^ {2}. \\ \end{array}
$$
Guided Backpropagation method (Springenberg et al., 2014) modifies the back-propagation to preserve negative gradients in the ReLU activation layer which also sharpens the "saliency map" visually. Other methods, such as Grad-CAM or Guided-CAM (Selvaraju et al., 2017), are applicable to specific architecture of neural networks in the field of computer vision.
Since language models like BERT do not contain specific architecture utilized in Guided Backpropagation or Grad-CAM method, we ignore the mathematical formulation here.
Feature Attribution on Input with Reference Data Integrated Gradient method computes the feature score by integrating the gradients from single pre-determined reference input to the target input (Sundararajan et al., 2017). In computer vision problems, black image is usually considered as the reference data, and integrating gradients from the black image to the input image represents the feature attribution of the input image. In NLP problems, we can define the  $i$ -th element of feature attribution as:
$$
\begin{array}{l} \mathcal {S} _ {\text {I n t e G r a d}} (H _ {0}) _ {i j} \\ \approx \frac {H _ {0 , i j} - H _ {i j} ^ {\prime}}{N} \sum_ {k = 1} ^ {N} \mathcal {S} (H ^ {\prime} + k \frac {H _ {0} - H ^ {\prime}}{N}) _ {i j}, \\ \end{array}
$$
$$
\mathcal {M} _ {I n t e G r a d} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} _ {I n t e G r a d} (H _ {0}) _ {i, j} ^ {2}.
$$
where  $H^{\prime}$  denotes the embedding of reference text.
SHAP-Gradient method which combines ideas from Integrated Gradient and Smooth Gradient into a single expected value equation (Lundberg and Lee, 2017). To be specific, the feature attribution is defined from:
$$
\begin{array}{l} \mathcal {S} _ {S h a p G r a d} (H _ {0}) \\ \approx \frac {1}{N} \sum_ {k = 1} ^ {N} \mathcal {S} (\alpha_ {k} H _ {0} + (1 - \alpha_ {k}) H _ {k}), \\ \end{array}
$$
$$
\mathcal {M} _ {S h a p G r a d} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} _ {S h a p G r a d} (H _ {0}) _ {i, j} ^ {2}.
$$
where  $\alpha_{k}\sim U(0,1)$  denotes uniform distribution from zero to one,  $H_{k}\in \mathcal{H}_{ref}$  denotes the embedding of reference text.
DeepLIFT (Shrikumar et al., 2017) assigns the feature score by comparing the difference of contribution between input and some reference inputs
via gradient. As discussed in (Lundberg and Lee, 2017), DeepLIFT can be considered as an approximation of Shapley Value estimation. Specifically, as in the application of SHAP  $^{2}$ , the feature attribution of SHAP-Deep as a variant of DeepLIFT is defined as:
$$
\mathcal {S} _ {S h a p D e e p} (H _ {0}) \approx \frac {1}{N} \sum_ {k = 1} ^ {N} \mathcal {S} (H _ {k}) \times (H _ {0} - H _ {k}).
$$
$$
\mathcal {M} _ {S h a p D e e p} (X) _ {i} = \sum_ {j = 1} ^ {d} \mathcal {S} _ {S h a p D e e p} (H _ {0}) _ {i, j} ^ {2}.
$$
In this section, we compare the proposed method to the state-of-the-art feature attribution methods under different use cases.
5.1 Case I: Feature Attribution on Relation Classification Model
Dataset	Precision	Recall	F1
NYT10	94.8	93.3	94.1
Webnlg	93.6	82.5	87.7
Motivation Relation Classification is beneficial to downstream problems, including question answering and knowledge graph (KG) construction tasks (Wen et al., 2016; Dhingra et al., 2016; Dong et al., 2020). With the development of deep language model, existing relation extraction methods have achieved significant performance in relation classification task (Soares et al., 2019; Wei et al., 2019). We hope to better understand the features in the text that help deep language model to classify the relations. In this use case, we fine-tune a deep language model with relation as labels. With the fine-tuned model, the feature attribution technique is applied to identify the entities in the text as important features.
Data We use the public available datasets NYT10 (Riedel et al., 2010) and Webnlg (Gardent et al., 2017) for numerical study. Zeng et al., 2018 adapted the original dataset for relation extraction task. We follow the same setting as in Zeng et al., 2018, i.e. NYT10 dataset contains
56,196/5,000/5,000 plain texts in train/val/test set, 24 relation type, averaged 2.01 relational triples in each text. Webnlg dataset contains 5,019/500/703 plain texts in train/val/test set, 211 relation type, averaged 2.78 relation triples in each text.