SlowGuess/ACL_2020_OCR huggingface dataset:
Hanjie Chen, Guangtao Zheng, Yangfeng Ji
Charlottesville, VA, USA
{hc9mx, gz5hp, yangfeng}@virginia.edu
Generating explanations for neural networks has become crucial for their applications in real-world with respect to reliability and trustworthiness. In natural language processing, existing methods usually provide important features which are words or phrases selected from an input text as an explanation, but ignore the interactions between them. It poses challenges for humans to interpret an explanation and connect it to model prediction. In this work, we build hierarchical explanations by detecting feature interactions. Such explanations visualize how words and phrases are combined at different levels of the hierarchy, which can help users understand the decision-making of black-box models. The proposed method is evaluated with three neural text classifiers (LSTM, CNN, and BERT) on two benchmark datasets, via both automatic and human evaluations. Experiments show the effectiveness of the proposed method in providing explanations that are both faithful to models and interpretable to humans.
Deep neural networks have achieved remarkable performance in natural language processing (NLP) (Devlin et al., 2018; Howard and Ruder, 2018; Peters et al., 2018), but the lack of understanding on their decision making leads them to be characterized as blackbox models and increases the risk of applying them in real-world applications (Lipton, 2016; Burns et al., 2018; Jumelet and Hupkes, 2018; Jacovi et al., 2018).
Understanding model prediction behaviors has been a critical factor in whether people will trust and use these blackbox models (Ribeiro et al., 2016). A typical work on understanding decision-making is to generate prediction explanations for each input example, called local explanation generation. In NLP, most of existing work on local
explanation generation focuses on producing word-level or phrase-level explanations by quantifying contributions of individual words or phrases to a model prediction (Ribeiro et al., 2016; Lundberg and Lee, 2017; Lei et al., 2016; Plumb et al., 2018).
Figure 1: Different explanations for a NEGATIVE movie review a waste of good performance, where the color of each block represents the contribution of the corresponding word/phrase/clause (feature) to the model prediction. From the hierarchical explanation, we obtain a set of features in each timestep  $(t)$ , where the most important one is waste of good.
Figure 1 (a) and (b) present a word-level and a phrase-level explanation generated by the LIME (Ribeiro et al., 2016) and the Contextual Decomposition (CD) (Murdoch et al., 2018) respectively for explaining sentiment classification. Both explanations provide scores to quantify how a word or a phrase contributes to the final prediction. For example, the explanation generated by LIME captures a keyword waste and the explanation from CD identifies an important phrase waste of.
5578
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5578-5593 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
However, neither of them is able to explain the model decision-making in terms of how words and phrases are interacted with each other and composed together for the final prediction. In this example, since the final prediction is NEGATIVE, one question that we could ask is that how the word good or a phrase related to the word good contributes to the model prediction. An explanation being able to answer this question will give users a better understanding on the model decision-making and also more confidence to trust the prediction.
The goal of this work is to reveal prediction behaviors of a text classifier by detecting feature (e.g., words or phrases) interactions with respect to model predictions. For a given text, we propose a model-agnostic approach, called HEDGE (for Hierarchical Explanation via Divisive Generation), to build hierarchical explanations by recursively detecting the weakest interactions and then dividing large text spans into smaller ones based on the interactions. As shown in Figure 1 (c), the hierarchical structure produced by HEDGE provides a comprehensive picture of how different granularity of features interacting with each other within the model. For example, it shows how the word good is dominated by others in the model prediction, which eventually leads to the correct prediction. Furthermore, the scores of text spans across the whole hierarchy also help identify the most important feature waste of good, which can be served as a phrase-level explanation for the model prediction.
The contribution of this work is three-fold: (1) we design a top-down model-agnostic method of constructing hierarchical explanations via feature interaction detection; (2) we propose a simple and effective scoring function to quantify feature contributions with respect to model predictions; and (3) we compare the proposed algorithm with several competitive methods on explanation generation via both automatic and human evaluations. The experiments were conducted on sentiment classification tasks with three neural network models, LSTM (Hochreiter and Schmidhuber, 1997), CNN (Kim, 2014), and BERT (Devlin et al., 2018), on the SST (Socher et al., 2013) and IMDB (Maas et al., 2011) datasets. The comparison with other competitive methods illustrates that HEDGE provides more faithful and human-understandable explanations.
Our implementation is available at https://
github.com/UVa-NLP/HEDGE.
Over the past years, many approaches have been explored to interpret neural networks, such as contextual decomposition (CD) for LSTM (Murdoch et al., 2018) or CNN model (Godin et al., 2018), gradient-based interpretation methods (Hechtlinger, 2016; Sundararajan et al., 2017), and attention-based methods (Ghaeini et al., 2018; Lee et al., 2017; Serrano and Smith, 2019). However, these methods have limited capacity in real-world applications, as they require deep understanding of neural network architectures (Murdoch et al., 2018) or only work with specific models (Alvarez-Melis and Jaakkola, 2018). On the other hand, model-agnostic methods (Ribeiro et al., 2016; Lundberg and Lee, 2017) generate explanations solely based on model predictions and are applicable for any black-box models. In this work, we mainly focus on model-agnostic explanations.
2.1 Model-Agnostic Explanations
The core of generating model-agnostic explanations is how to efficiently evaluate the importance of features with respect to the prediction. So far, most of existing work on model-agnostic explanations focus on the word level. For example, Li et al. (2016) proposed Leave-one-out to probe the black-box model by observing the probability change on the predicted class when erasing a certain word. LIME proposed by Ribeiro et al. (2016) estimates individual word contribution locally by linear approximation from perturbed examples. A line of relevant works to ours is Shapley-based methods, where the variants of Shapley values (Shapley, 1953) are used to evaluate feature importance, such as SampleShapley (Kononenko et al., 2010), KernelSHAP (Lundberg and Lee, 2017), and L/C-Shapley (Chen et al., 2018). They are still in the category of generating word-level explanations, while mainly focus on addressing the challenge of computational complexity of Shapley values (Datta et al., 2016). In this work, inspired by an extension of Shapley values (Owen, 1972; Grabisch, 1997; Fujimoto et al., 2006), we design a function to detect feature interactions for building hierarchical model-agnostic explanations in subsection 3.1. While, different from prior work of using Shapley values for feature importance evaluation, we propose an effective and simpler way to
5579
evaluate feature importance as described in subsection 3.3, which outperforms Shapley-based methods in selecting important words as explanations in subsection 4.2.
2.2 Hierarchical Explanations
Addressing the limitation of word-level explanations (as discussed in section 1) has motivated the work on generating phrase-level or hierarchical explanations. For example, Tsang et al. (2018) generated hierarchical explanations by considering the interactions between any features with exhaustive search, which is computationally expensive.
Singh et al. (2019) proposed agglomerative contextual decomposition (ACD) which utilizes CD scores (Murdoch et al., 2018; Godin et al., 2018) for feature importance evaluation and employ a hierarchical clustering algorithm to aggregate features together for hierarchical explanation. Furthermore, Jin et al. (2019) indicated the limitations of CD and ACD in calculating phrase interactions in a formal context, and proposed two explanation algorithms by quantifying context independent importance of words and phrases.
A major component of the proposed method on feature interaction detection is based on the Shapley interaction index (Owen, 1972; Grabisch, 1997; Fujimoto et al., 2006), which is extended in this work to capture the interactions in a hierarchical structure. Lundberg et al. (2018) calculated features interactions via SHAP interaction values along a given tree structure. Chen and Jordan (2019) suggested to utilize a linguistic tree structure to capture the contributions beyond individual features for text classification. The difference with our work is that both methods (Lundberg et al., 2018; Chen and Jordan, 2019) require hierarchical structures given, while our method constructs structures solely based on feature interaction detection without resorting external structural information. In addition, different from Singh et al. (2019), our algorithm uses a top-down fashion to divide long texts into short phrases and words based on the weakest interactions, which is shown to be more effective and efficient in the experiments in section 4.
This section explains the proposed algorithm on building hierarchical explanations (subsection 3.1) and two critical components of this algorithm: detecting feature interaction (subsection 3.2) and
quantifying feature importance (subsection 3.3).
Algorithm 1 Hierarchical Explanation via Divisive Generation
1: Input: text  $x$  with length  $n$ , and predicted label  $\hat{y}$
2: Initialize the original partition  $\mathcal{P}_0\gets \{\pmb {x}_{(0,n]}\}$
3: Initialize the contribution set  $\mathcal{C}_0 = \emptyset$
4: Initialize the hierarchy  $\mathcal{H} = [\mathcal{P}_0]$
5: for  $t = 1, \dots, n - 1$  do
6: Find  $\pmb{x}_{(s_i, s_{i+1}]}$  and  $j$  by solving Equation 1
7: Update the partition
$$
\mathcal {P} _ {t} ^ {\prime} \leftarrow \mathcal {P} _ {t - 1} \backslash \{\boldsymbol {x} _ {(s _ {i}, s _ {i + 1} ]} \}
$$
$$
\mathcal {P} _ {t} \leftarrow \mathcal {P} _ {t} ^ {\prime} \cup \{\boldsymbol {x} _ {(s _ {i}, j ]}, \boldsymbol {x} _ {(j, s _ {i + 1} ]} \}
$$
8:  $\mathcal{H}.add(\mathcal{P}_t)$
9: Update the contribution set  $\mathcal{C}$  with
$$
\mathcal {C} _ {t} ^ {\prime} \leftarrow \mathcal {C} _ {t - 1} \cup \left\{\left(\boldsymbol {x} _ {\left(s _ {i}, j \right]}, \psi \left(\boldsymbol {x} _ {\left(s _ {i}, j \right]}\right)\right) \right\}
$$
$$
\mathcal {C} _ {t} \leftarrow \mathcal {C} _ {t} ^ {\prime} \cup \left\{\left(\boldsymbol {x} _ {\left(j, s _ {i + 1} \right]}, \psi \left(\boldsymbol {x} _ {\left(j, s _ {i + 1}\right)}\right)\right) \right\}
$$
10: end for
11: Output:  $C_{n-1}$ ,  $\mathcal{H}$
3.1 Generating Hierarchical Explanations
For a classification task, let  $\pmb{x} = (x_{1},\dots ,x_{n})$  denote a text with  $n$  words and  $\hat{y}$  be the prediction label from a well-trained model. Furthermore, we define  $\mathcal{P} = \{\pmb{x}_{(0,s_1]},\pmb{x}_{(s_1,s_2]},\dots ,\pmb{x}_{(s_{P - 1},n]}\}$  be a partition of the word sequence with  $P$  text spans, where  $\pmb{x}_{(s_i,s_{i + 1}]} = (x_{s_i + 1},\dots ,x_{s_{i + 1}})$ . For a given text span  $\pmb{x}_{(s_i,s_{i + 1}]}$ , the basic procedure of HEDGE is to divide it into two smaller text spans  $\pmb{x}_{(s_i,j]}$  and  $\pmb{x}_{(j,s_{i + 1}]}$ , where  $j$  is the dividing point  $(s_i < j < s_{i + 1})$ , and then evaluate their contributions to the model prediction  $\hat{y}$ .
Algorithm 1 describes the whole procedure of dividing  $x$  into different levels of text spans and evaluating the contribution of each of them. Starting from the whole text  $x$ , the algorithm first divides  $x$  into two segments. In the next iteration, it will pick one of the two segments and further split it into even smaller spans. As shown in algorithm 1, to perform the top-down procedure, we need to answer the questions: for the next timestep, which text span the algorithm should pick to split and where is the dividing point?
Both questions can be addressed via the following optimization problem:
$$
\min  _ {\boldsymbol {x} _ {(s _ {i}, s _ {i + 1} ]} \in \mathcal {P}} \min  _ {j \in (s _ {i}, s _ {i + 1})} \phi (\boldsymbol {x} _ {(s _ {i}, j ]}, \boldsymbol {x} _ {(j, s _ {i + 1} ]} \mid \mathcal {P}), \tag {1}
$$
where  $\phi (\pmb{x}_{(s_i,j]},\pmb{x}_{(j,s_{i + 1}]}\mid \mathcal{P})$  defines the interaction score between  $\pmb{x}_{(s_i,j]}$  and  $\pmb{x}_{(j,s_{i + 1}]}$  given the
current partition  $\mathcal{P}$ . The detail of this score function will be explained in subsection 3.2.
For a given  $\pmb{x}_{(s_i,s_{i + 1}]} \in \mathcal{P}$ , the inner optimization problem will find the weakest interaction point to split the text span  $\pmb{x}_{(s_i,s_{i + 1}]}$  into two smaller ones. It answers the question about where the dividing point should be for a given text span. A trivial case of the inner optimization problem is on a text span with length 2, since there is only one possible way to divide it. The outer optimization answers the question about which text span should be picked. This optimization problem can be solved by simply enumerating all the elements in a partition  $\mathcal{P}$ . A special case of the outer optimization problem is at the first iteration  $t = 1$ , where  $\mathcal{P}_0 = \{\pmb{x}_{(0,n)}\}$  only has one element, which is the whole input text. Once the partition is updated, it is then added to the hierarchy  $\mathcal{H}$ .
The last step in each iteration is to evaluate the contributions of the new spans and update the contribution set  $\mathcal{C}$  as in line 9 of the algorithm 1. For each, the algorithm evaluates its contribution to the model prediction with the feature importance function  $\psi (\cdot)$  defined in Equation 5. The final output of algorithm 1 includes the contribution set  $\mathcal{C}_{n - 1}$  which contains all the produced text spans in each timestep together with their importance scores, and the hierarchy  $\mathcal{H}$  which contains all the partitions of  $x$  along timesteps. A hierarchical explanation can be built based on  $\mathcal{C}_{n - 1}$  and  $\mathcal{H}$  by visualizing the partitions with all text spans and their importance scores along timesteps, as Figure 1 (c) shows.
Note that with the feature interaction function  $\phi (\cdot ,\cdot)$ , we could also design a bottom-up approach to merge two short text spans if they have the strongest interaction. Empirically, we found that this bottom-up approach performs worse than the algorithm 1, as shown in Appendix A.
3.2 Detecting Feature Interaction
For a given text span  $\pmb{x}_{(s_i,s_{i + 1}]}\in \mathcal{P}$  and the dividing point  $j$ , the new partition will be  $\mathcal{N} = \mathcal{P}\backslash \{\pmb{x}_{(s_i,s_{i + 1}]}\} \cup \{\pmb{x}_{(s_i,j]},\pmb{x}_{(j,s_{i + 1}]}\} = \{\pmb{x}_{(0,s_1]},\dots ,\pmb{x}_{(s_i,j]},\pmb{x}_{(j,s_{i + 1}]},\dots ,\pmb{x}_{(s_{P - 1},n]}\}$ . We consider the effects of other text spans in  $\mathcal{N}$  when calculate the interaction between  $\pmb{x}_{(s_i,j]}$  and  $\pmb{x}_{(j,s_{i + 1}]}$ , since the interaction between two words/phrases is closely dependent on the context (Hu et al., 2016; Chen et al., 2016). We adopt the Shapley interaction index from coalition game theory (Owen, 1972; Grabisch, 1997; Fujimoto et al.,
2006) to calculate the interaction. For simplicity, we denote  $\pmb{x}_{(s_i,j]}$  and  $\pmb{x}_{(j,s_{i+1}]}$  as  $j_1$  and  $j_2$  respectively. The interaction score is defined as (Lundberg et al., 2018),
$$
\phi \left(j _ {1}, j _ {2} \mid \mathcal {P}\right) = \sum_ {S \subseteq \mathcal {N} \backslash \left\{j _ {1}, j _ {2} \right\}} \frac {| S | ! (P - | S | - 1) !}{P !} \gamma \left(j _ {1}, j _ {2}, S\right), \tag {2}
$$
where  $S$  represents a subset of text spans in  $\mathcal{N}\backslash \{j_1,j_2\}$ $|S|$  is the size of  $S$  , and  $\gamma (j_{1},j_{2},S)$  is defined as follows,
$$
\begin{array}{l} \gamma \left(j _ {1}, j _ {2}, S\right) = \mathbb {E} \left[ f \left(\boldsymbol {x} ^ {\prime}\right) \mid S \cup \left\{j _ {1}, j _ {2} \right\} \right] - \mathbb {E} \left[ f \left(\boldsymbol {x} ^ {\prime}\right) \mid S \cup \left\{j _ {1} \right\} \right] \\ - \mathbb {E} [ f (\boldsymbol {x} ^ {\prime}) \mid S \cup \{j _ {2} \} ] + \mathbb {E} [ f (\boldsymbol {x} ^ {\prime}) \mid S ], \tag {3} \\ \end{array}
$$
where  $\pmb{x}^{\prime}$  is the same as  $\pmb{x}$  except some missing words that are not covered by the given subset (e.g.  $S$ ),  $f(\cdot)$  denotes the model output probability on the predicted label  $\hat{y}$ , and  $\mathbb{E}[f(\pmb{x}^{\prime})\mid S]$  is the expectation of  $f(\pmb{x}^{\prime})$  over all possible  $\pmb{x}^{\prime}$  given  $S$ . In practice, the missing words are usually replaced with a special token  $<\mathrm{pad}>$ , and  $f(\pmb{x}^{\prime})$  is calculated to estimate  $\mathbb{E}[f(\pmb{x}^{\prime})|S]$  (Chen et al., 2018; Datta et al., 2016; Lundberg and Lee, 2017). We also adopt this method in our experiments. Another way to estimate the expectation is to replace the missing words with substitute words randomly drawn from the full dataset, and calculate the empirical mean of all the sampling data (Kononenko et al., 2010; Štrumbelj and Kononenko, 2014), which has a relatively high computational complexity.
With the number of text spans (features) increasing, the exponential number of model evaluations in Equation 2 becomes intractable. We calculate an approximation of the interaction score based on the assumption (Chen et al., 2018; Singh et al., 2019; Jin et al., 2019): a word or phrase usually has strong interactions with its neighbours in a sentence. The computational complexity can be reduced to polynomial by only considering  $m$  neighbour text spans of  $j_{1}$  and  $j_{2}$  in  $\mathcal{N}$ . The interaction score is rewritten as
$$
\phi \left(j _ {1}, j _ {2} \mid \mathcal {P}\right) = \sum_ {S \subseteq \mathcal {N} _ {m} \backslash \{j _ {1}, j _ {2} \}} \frac {| S | ! (M - | S | - 2) !}{(M - 1) !} \gamma \left(j _ {1}, j _ {2}, S\right), \tag {4}
$$
where  $\mathcal{N}_m$  is the set containing  $j_{1}, j_{2}$  and their neighbours, and  $M = |\mathcal{N}_m|$ . In section 4, we set  $m = 2$ , which performs well. The performance can be further improved by increasing  $m$ , but at the cost of increased computational complexity.
3.3 Quantifying Feature Importance
To measure the contribution of a feature  $\pmb{x}_{(s_i,s_{i + 1}]}$  to the model prediction, we define the importance score as
$$
\begin{array}{l} \psi \left(\boldsymbol {x} _ {\left(s _ {i}, s _ {i + 1} \right]}\right) = f _ {\hat {y}} \left(\boldsymbol {x} _ {\left(s _ {i}, s _ {i + 1} \right]}\right) \\ - \max  _ {y ^ {\prime} \neq \hat {y}, y ^ {\prime} \in \mathcal {Y}} f _ {y ^ {\prime}} \left(\boldsymbol {x} _ {\left(s _ {i}, s _ {i + 1} \right]}\right), \tag {5} \\ \end{array}
$$
where  $f_{\hat{y}}(\pmb{x}_{(s_i,s_{i+1}]})$  is the model output on the predicted label  $\hat{y}$ ;  $\max_{y' \neq \hat{y}, y' \in \mathcal{Y}} f_{y'}(\pmb{x}_{(s_i,s_{i+1}]})$  is the highest model output among all classes excluding  $\hat{y}$ . This importance score measures how far the prediction on a given feature is to the prediction boundary, hence the confidence of classifying  $\pmb{x}_{(s_i,s_{i+1}]}$  into the predicted label  $\hat{y}$ . Particularly in text classification, it can be interpreted as the contribution to a specific class  $\hat{y}$ . The effectiveness of Equation 5 as feature importance score is verified in subsection 4.2, where HEDGE outperforms several competitive baseline methods (e.g. LIME (Ribeiro et al., 2016), SampleShapley (Kononenko et al., 2010)) in identifying important features.
The proposed method is evaluated on text classification tasks with three typical neural network models, a long short-term memories (Hochreiter and Schmidhuber, 1997, LSTM), a convolutional neural network (Kim, 2014, CNN), and BERT (Devlin et al., 2018), on the SST (Socher et al., 2013) and IMDB (Maas et al., 2011) datasets, via both automatic and human evaluations.
Datasets. We adopt the SST-2 (Socher et al., 2013) which has 6920/872/1821 examples in the train/dev/test sets with binary labels. The IMDB (Maas et al., 2011) also has binary labels with 25000/25000 examples in the train/test sets. We hold out  $10\%$  of the training examples as the development set.
Models. The CNN model (Kim, 2014) includes a single convolutional layer with filter sizes ranging from 3 to 5. The LSTM (Hochreiter and Schmidhuber, 1997) has a single layer with 300 hidden states. Both models are initialized with 300-dimensional pretrained word embeddings (Mikolov et al., 2013). We use the pretrained BERT model<sup>1</sup> with 12 trans
former layers, 12 self-attention heads, and the hidden size of 768, which was then fine-tuned with different downstream tasks to achieve the best performance. Table 1 shows the best performance of the models on both datasets in our experiments, where BERT outperforms CNN and LSTM with higher classification accuracy.
Models	Dataset
SST	IMDB
LSTM	0.842	0.870
CNN	0.850	0.901
BERT	0.924	0.930
Table 1: The classification accuracy of different models on the SST and IMDB datasets.
We adopt two metrics from prior work on evaluating word-level explanations: the area over the perturbation curve (AOPC) (Nguyen, 2018; Samek et al., 2016) and the log-odds scores (Shrikumar et al., 2017; Chen et al., 2018), and define a new evaluation metric called cohesion-score to evaluate the interactions between words within a given text span. The first two metrics measure local fidelity by deleting or masking top-scored words and comparing the probability change on the predicted label. They are used to evaluate Equation 5 in quantifying feature contributions to the model prediction. The cohesion-score measures the synergy of words within a text span to the model prediction by shuffling the words to see the probability change on the predicted label.
AOPC. By deleting top  $k\%$  words, AOPC calculates the average change in the prediction probability on the predicted class over all test data as follows,
$$
\operatorname {A O P C} (k) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left\{p \left(\hat {y} \mid \boldsymbol {x} _ {i}\right) - p \left(\hat {y} \mid \tilde {\boldsymbol {x}} _ {i} ^ {(k)}\right) \right\}, \tag {6}
$$
where  $\hat{y}$  is the predicted label,  $N$  is the number of examples,  $p(\hat{y}|\cdot)$  is the probability on the predicted class, and  $\tilde{\boldsymbol{x}}_i^{(k)}$  is constructed by dropping the  $k\%$  top-scored words from  $\boldsymbol{x}_i$ . Higher AOPCs are better, which means that the deleted words are important for model prediction. To compare with other word-level explanation generation methods under this metric, we select word-level features from the bottom level of a hierarchical explanation and sort them in the order of their estimated importance to the prediction.
'https://github.com/huggingface/pytorch-transformers