Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
Authors: Xisen Jin
Zhongyu Wei
Junyi Du
Xiangyang Xue
Xiang Ren
Venue:  (2020)
Keywords: natural language processing
interpretability

Abstract: The impressive performance of neural networks on natural language processing tasks attributes to their ability to model complicated word and phrase compositions. To explain how the model handles semantic compositions, we study hierarchical explanation of neural network predictions. We identify non-additivity and context independent importance attributions within hierarchies as two desirable properties for highlighting word and phrase compositions. We show some prior efforts on hierarchical explanations, e.g. contextual decomposition, do not satisfy the desired properties mathematically, leading to inconsistent explanation quality in different models. In this paper, we start by proposing a formal and general way to quantify the importance of each word and phrase. Following the formulation, we propose Sampling and Contextual Decomposition (SCD) algorithm and Sampling and Occlusion (SOC) algorithm. Human and metrics evaluation on both LSTM models and BERT Transformer models on multiple datasets show that our algorithms outperform prior hierarchical explanation algorithms. Our algorithms help to visualize semantic composition captured by models, extract classification rules and improve human trust of models.

Published as a conference paper at ICLR 2020
TOWARDS HIERARCHICAL IMPORTANCE ATTRIBU- TION: EXPLAINING COMPOSITIONAL SEMANTICS FOR NEURAL SEQUENCE MODELS
Xisen Jin Zhongyu Weil, Junyi Junyi Dus, Xiangyang Xuel, Xiang Ren
SUniversity of Southern California
iFudan University
jinsenjin, junyidu, xiangren)@uscoo (zywei, xyxue)0fudan edu.cn
ABSTRACT
The impressive performance of neural networks on natural language processing tasks attributes to their ability to model complicated word and phrase composi- tions. To explain how the model handles semantic compositions, we study hierar- chical explanation of neural network predictions. We identify non-additivity and context independent importance attributions within hicrarchies as two desirable properties for highlighting word and phrase compositions. We show some prior efforts on hierarchical explanations, e.g. contextual decomposition, do not satisfy the desired properties mathematically, leading inconsistent explanation quality in different models. In this paper, we start by proposing a formal and general way to quantify the importance of each word and phrase. Following the formulation, we propose Sampling and Contextual Decomposition (SCD) algorithm and Sam- pling and Occlusion (SOC) algorithm. Human and metrics evaluation on both LSTM models and BERT Transformer models on multiple datasets show that our algorithms outperform prior hierarchical explanation algorithms. Our algorithms help to visualize semantic composition captured by models, extract classification rules and improve human trust models"
INTRODUCTION
Recent advances in deep neural networks have led to impressive results on a range of natural language processing (NLP) tasks, by learning latent, compositional vector representations of text data (Peters et al. 2018 Devlin et al. 2018 Liu et al. 2019b). However, interpretability of the pre- dictions given by these complex, "black box" models has always been limiting factor for use cases that require explanations of the features involved in modeling (e.g., words and phrases) (Guidotti) (ct al. 2018 Ribeiro et al., 2016). Prior efforts on enhancing model interpretability have focused on either constructing models with intrinsically interpretable structures (Bahdanau et al., 2015 Luu jet al., 2019a), or developing post-hoc explanation algorithms which can explain model predictions without elucidating the mechanisms by which model works (Mohseni et al.) 2018 Guidotti et al. 2018). Among these work, post-hoc explanation has come to the fore as they can operate over a variety of trained models while not affecting predictive performance of models. Towards post-hoc explanation, a major line of work, additive feature attribution methods (Lundberg & Lee| [017]] Ribeiro et al., 2016 Binder et al. 2016: Shrikumar et al., 2017). explain a model prediction by as- signing importance scores to individual input variables. However, these methods are not ideal for explaining phrase-level importance, as phrase importance often a non non-linear combination of the importance of the words in the phrase. Contextual decomposition (CD) (Murdoch et al.) (018)) and its hierarchical extension (Singh al. (211)) go beyond the additive assumption and compute the contribution solely made by word/phrase to the model prediction (i.e., individual contribution). by decomposing the output variables the neural network at each layer. Using the individual contribu-
Project page: htttt:///init www..uc....dd/eeeelllll.

---

Published as a conference paper at ICLR 2020
not interesting
not interesting not interesting
Poethe
not
interesting
not
interesting
not
interesting
(a) Input occlusion
(b) Additive attribution
(c) Hierarchical
Negattve
Figure Different score attribution algorithms. (a) Input occlusion assigns a negative score for the word "interesting". as the sentiment of the phrase becomes less negative after removing "in- teresting" from the original sentence. (b) Additive attributions assign importance scores for words "not" and "interesting" by linearly distributing contribution score of "not interesting". exemplified with Shapley Values (Shapley) 1953). Intuitively, only (c) Hierarchical explanations highlight the negative compositional effect between the words "not" and "interesting".
tion scores so derived, these algorithms generate hierarchical explanation on how the model captures compositional semantics (e.g., stress negation) in making predictions (see Figure T for example).
However, despite contextual decomposition methods have achieved good results in practice, what reveals extra importance that emerge from combining two phrases has not been well studied. As result, prior lines of work on contextual decomposition have focused on exploring model-specific decompositions based on their performance on visualizations. We identify the extra importance from combining two phrases can be quantified by studying how the importance of the combined phrase differs from the sum of the importance of the two component phrases on its own. Similar strategies have been studied in game theory for quantifying the surplus from combining two groups of players (Fujimoto et al., 2006). Following the definition above, the key challenge is to formulate the importance of phrase on it own, i.e., context independent importance of a phrase. However, we show contextual decomposition do not satisfy this context independence property mathematically.
To this end, we propose a formal way to quantify the importance of each individual word/phrase, and develop effective algorithms for generating hierarchical explanations based on the new formu- lation. To mathematically formalize and efficiently approximate context independent importance, we formulate N-context independent importance of phrase, defined as the difference of model output after masking out the phrase, marginalized over all possible N words surrounding the phrase in the sentence. We propose two explanation algorithms according to our formulation, namely the Sampling and Contextual Decomposition algorithm (SCD), which overcomes the weakness of con- textual decomposition algorithms, and the Sampling and OCclusion algorithm (SOC), which is sim- ple, model-agnostic, and performs competitively against prior lines of algorithms. We experiment with both LSTM and fine-tuned Transformer models to evaluate the proposed methods. Quantita- tive studies involving automatic metries and human evaluation sentiment analysis and relation extraction tasks show that our algorithms consistently outperform competitors in the quality of ex- planations. Our algorithms manage provide hierarchical visualization of compositional semantics captured by models, extract classification rules from models, and help users to trust neural networks predictions.
In summary, our work makes the following contributions: (1) we identify the key challenges in generating post-hoc hierarchical explanations and propose mathematically sound way quantify context independent importance of words and phrases for generating hierarchical explanations; (2) we extend previous post-hoc explanation algorithm based on the new formulation of N --context dependent importance and develop two effective hierarchical explanation algorithms; and (3) exper- iments demonstrate that the proposed explanation algorithms consistently outperform the compared methods (with both LSTM and Transformer base models) over several datasets and models.
PRELIMINARIES
2.1 POST-HOC EXPLANATIONS OF NEURAL SEQUENCE MODELS
We consider a sequence low-dimensional word embeddings X1.7 == (x), S2,....... Xr),, or denoted x for brevity, the input to neural sequence model, such as standard RNNs, LSTT (Hochreiter
2

---

Published as a conference paper at ICLR 2020
& Schmidhuber 1997) and Transformers (Vaswani et al. 2017). These neural models extract hidden representations h1rr from the input sequence x, and feed these hidden representations to a prediction layer to generate prediction scores in the label space (e.g., sentiment polarity of sentence). For LSTM, we use the last hidden state hy to give unnormalized prediction scores s(x) e TR& over de label classes follows.
sxx) = Wihr;
(1)
where Wi Rddxdd is a trainable weight matrix. For Transformers, the representation correspond- ing to the [CLS] J token at the final layer is fed to the prediction layer to generate scores sxx).
Towards post-hoc explanation of s(x), a notable line of work, additive feature attribution meth- ods (Ribeiro et al., 2016 Shrikumar et al. 2017 Sundararajan et al.) 2 Lundberg & Lee 2017).
measure word-level importance to the model prediction s(x) by attributing an importance score (xx, to each word in the input sequence E x. However, the additive assumption hinders these methods from explaining the complex interactions between words and compositional semanties in a sentence (e.g., modeling negation, transition, and emphasis in sentiment classification), as shown in Figure
2.2 HIERARCHICAL EXPLANATIONS VIA CONTEXTUAL DECOMPOSITION
To caputure non-linear compositional semantics, the line of work on contextual decomposi- tion (CD) (Murdoch et al., (2118) designs non-additive measures of importance from individual words/phrases to the model predictions, and further extend to agglomerative contextual decomposi- tion (ACD) algorithm (Singh et al. 2019) for generating hierarchical explanations.
Given a phrase p = X.jj in the input sequence x, contextual decomposition (CD) attributes a score Φ(p, x) as the contribution solely from p to the model's prediction sxx). Note that o(p, x) does not equal to the sum of the scores of each word in the phrase, i.e., Φ(p, x) * Ex,ep ((x,, x). Starting from the input layer, CD iteratively decomposes each hidden state h of the model into the contribution solely made by p, denoted as B, and the contributions involving the words outside the phrase p. denoted as y, with the relation = + holds. The algorithm also keeps the contribution from the bias term, denoted as Ç. temporally before element-wise multiplication.
For a linear layer h W,x, + b, with input x,, when x, lies in the given phrase p. the contribution solely from x, to h is defined as B W,x, when is part of the phrase (i.e., contribution involving other words in the sentences (denoted as x x\p)) is defined as y = O. The contribution of the bias term Ç is thus b;- When x, lies outside of the phrase, 7 is quantified as Wix, and G3 is O. In the cases when CD encounters element-wise multiplication operations ha he (e.g., in LSTMs), it eliminates the multiplicative interaction terms which involve the information outside of the phrase Specifically, suppose that h, and h, have been decomposed and h, g + 70 + co CD computes the 3 term for he O h, as
p). and the
as h
3
+ 7a + aa
B = 3 c 3
+ 3
@ - 8*.
When dealing with non-linear activation h' o (h), CD computes the contribution solely from the phrase as the average activation differences caused /3 supposing 7 is present or absent,
1 B' = 2 1o [o 7 +) -
1 1o +) (C))
+)] +
Following the strategies introduced above, CD decomposes all the intermediate outputs starting from the input layer, until reaching the final output of the model hy 3 7y The logit score Wi is treated as the contribution of the given phrase p to the final prediction sxx).
As follow-up study, (Singh et al. (2019) extends CD algorithm to other families of neural network architectures, and proposes agglomerative contextual decomposition algorithm (ACD). The decom- position of activation functions is modified as f (B3) For a linear layer h' Wh + b with its decomposition h = B + y, the bias term is decomposed proportionally and merged into the 3 term of h', based on B W + WW Ø + [W>))- b.
3

---

Published as a conference paper at ICLR 2020
B
B
Original
The
very <pad>
interesting
The
film
interesting
The
very
G
The
film
<pad>
well
The
The
film
film
<pad>
very
good
pood
4%
Sampled
Sampled
The
Film
Tunny
B
B
h
5 (x
<pods
Funny
~ p(
The
film
<pod>
dars
(a) CD
(b) SCD (ours)
Figure 2: (Left) Illustration of the CD and SCD at calculating the decomposition for h = o + 7), following Eq. 2 Red lines indicate computation that make CD explanations dependent on the words outside the give phrase. (Right) Illustration of the sampling step X8 p(&A xx)) for calculating the The padding operation
importance of the word very in SOC and SCD, with size of context for SOC.
METHODOLOGY
In this section, start by identifying desired properties of phrase-level importance score attribution for hierarchical explanations. We propose a measure of context-independent importance of phrase and introduce explanation algorithms instantiated from the proposed formulation.
3.1 PROPERTIES IMPORTANCE ATTRIBUTION FOR HIERARCHICAL EXPLANATION
Despite the empirical success of CD and ACD, prior works analyze what common properties score attribution mechanism should satisfy to generate hicrarchical explanations that reveal com- positional semantics formed between phrases. Here identify two properties that attribution method should satisfy to generate informative hierarchical explanations.
Non-additivity. Importance of phrase o(p,x) x) should not be over the importance scores of all the component words x; E p. i.e., Φ(p, x) * Ex.ep ((x,, x), in contrast to the family of additive feature attribution methods. The property is also suggested by Murdoch et al. (2018).
Context Independence. For deep neural networks, when two phrases combine, their importance to predicting a class may greatly change. The surplus by combining two phrases can be quantified by the difference between the importance of the combined phrase and the of the importance of two phrases on its own. It follows the definition of marginal interactions (Fujimoto et al.) (006) in game theory. According to the definition, the importance of component phrases should be at least evaluated independently to each other. Formally, if we are only interested in how combining two phrases Pı and Р2 contribute
specific prediction for input x, we expect for input sentences where only P2 is replaced to another phrase, the importance attribution for remains the same, i.c., o ((P. x) Φ(P1; x). In our bottom-up hierarchical explanation setting, we are interested in how combining phrase and any other contextual words or phrases in the input x contribute. Therefore, we expect (pp, pp, x) = o(p, x) given the phrase p in two different contexts x and
Limitations of CD and ACD. Unfortunately, while CD and ACD try to construct decomposition so that (3 terms represent the contributions solely from given phrase, the assigned importance scores by these algorithms do not satisfy the context independence property mathematically. For CD, see the computation of [3 involves the 7 term of a specific input sentence in Eq. 2 (see Figure ((a) for visualization). Similarly, for ACD, the decomposition of the bias term involves the terms of specific input sentence. As a result, the B terms computed by both algorithms depend on the context of the phrase p.
Given the limitation of prior works, we start by formulating importance measure of phrases that satisfies both nonaadditvvy and context independence property.
4

---

Published as a conference paper at ICLR 2020
3.2 CONTEXT-INDEPENDEN IMPORTANCE
Given a phrase p :- Xitj appearing in a specific input XI.T, we first relax our setting and define the importance of a phrase independent of contexts of length N adjacent it. The N-context inde- pendent importance is defined as the output difference after masking out the phrase p. marginalized over all the possible contexts of length N, denoted X5. around p in the input x. For an intuitive example, to evaluate the context independent importance up to one word of very in the sentence The film is very interesting in a sentiment analysis model, we sample some possible adjacent words before and after the word very, and average the prediction difference after some practice of masking the word very. Figure 2 illustrates an example for the sampling and masking steps. The N-context independent importance is formally written as,
o x) = Ex, -xx pp:p:
-
(3)
where x-5 denotes the resulting sequence after masking out context of length N surrounding the phrase p from the input x. Here, X5 is a sequence of length N sampled from a distribution p({s|x=s), which is conditioned on the phrase p as well as other words in the sentence x. Accord- ingly, we use s 6x_ &s) ) to denote the model prediction score after replacing the masked-out context x-5 with a sampled context Xg. We use x x to denote the operation of masking out the phrase p from the input sentence x. The specific implementation of this masking out operation varies across different explanation algorithms.
Following the notion of N --context independent importance, we define context- inteext-ndeepeneent impor- tance of phrase p by increasing the size of the context N sufficiently large (e.g., length of the sentence). The context independent importance can be equivalently written as follows.
(p) = Ex(s(x) s(x C x].
While it is possible to compute the expectations in Eqs. 3 and by sampling from the training set, it is common that a phrase occurs sparsely in the corpus. Therefore, we approximate the N-context independent importance by sampling from a language model pretrained on the training corpus. The language model helps model a smoothed distribution of p(x, (x In practice, all our explanation algorithms make of N-context independent importance following Eq. 3 where the size of the neighborhood N is parameter to be specified to approximate the context independent importance.
3.3 SAMPLING AND CONTEXTUAL DECOMPOSITION ALGORITIM
In CD, the desirable context independence property is compromised when computing decomposi- tion of activation functions, as discussed in Section 3.1 Following the new formulation of context- independent importance introduced in Section 3.2 we present a simple modification of the contex- tual decomposition algorithm, and develop new sampling and contextual decomposition (SCD) algorithm for effective generation of hierarchical explanations.
SCD is a layer-wise application of our formulation. The algorithm only modifies the way to decom- pose activation functions in CD. Specifically, given the output h = s(4) (x) at an intermediate layer / with the decomposition following definition:
B + 7, we decompose the activation value (h)) into B + n', with the
3' = 1o
- a (<)) = En-pp(ix la((h) (h -3]],
(5)
ie.. B is defined as the expected difference between the activation values when the 3 term is present or absent. h is computed for different input sequences x with the contexts of the phrase p sampled from the distribution p(}s(x_s). Eq. 5 is a layer wise application of Eq. 4 where the masking operation is implemented with calculating (h - B). Figure 2(b) provides a visualization for the decomposition.
Algorithmic Details. To perform sampling, we first pretrain a LSTM language model from two directions on the training data. During sampling, mask the words that p({s|x=a) a). Some other sampling options include performing Gibbs sampling from a masked lan- guage model (Wang et al., (2019). The algorithm then obtain a set of samples S by sampled from the language model. For each sample in S, the algorithm records the input of the i-th non-linear activa-
conditioned in
tion function to obtain sample set c(()
During the explanation, the & term of the i-th non-linear
5

---

Published as a conference paper at ICLR 2020
activation function is calculated as,
3
Σ hes
(b( - o(h (3)..
Some neural models such as Transformers involve operations that normalize over different dimen- sions vectors, e.g. layer normalization operations. We observe improved performance by not decomposing the normalizer of these terms when the phrase p is shorter than threshold, assuming that the impact of to the normalizer can be ignored. Besides, for element-wise multiplication in LSTM models, treat them the same way as other non-linear operations and decompose them as S where the decomposition of h Ghe is written as 3 = ( We avoid decomposing softmax functions.
3.4 SAMPLING AND OCCLUSION ALGORITHM
We show it is possible fit input occlusion (Li 2016) occlusion algorithm calculate the importance of specific to an input sentence x by observing the prediction difference caused by replacing the phrase p with padding tokens, noted as Op.
algorithm into our formulation. Input
ppp x) = s(x) - s(x-p:0p)
obvious that the importance score obtained by the input occlusion algorithm is dependent on the all the context words of p in x. To eliminate the dependence, sample the context around the phrase p. This leads to the Sampling and Occlusion (SOC) algorithm, where the importance of phrases is defined as the expected prediction difference after masking the phrase for each replace- ment of contexts.
Algorithmic Details. Similar to SCD, SOC samples neighboring words X, from trained lan- guage model p(xs|x__>/ x-8) and obtain a set neighboring word replacement S. For each replacement &5 E S, the algorithm computes the model prediction differences after replacing the phrase p with padding tokens. The importance ((p,x) x) is then calculated as the average prediction differences. Formally, the algorithm calculates,
@ x)
S
Σ [s(x xxix=8;8s:= - -x0--ssx_____ (6.p) Ra: Op)]- RAES
(8)
Sampling and Occlusion advantageous in that is model-agnostic and easy to implement. The input occlusion algorithm inside Eq, ca also be replaced with other measure of phrase importance, such as Shapley values (Shapley 1953), with the phrase p and other input words considered as players. We expect it is helpful for longer sequences when there multiple evidences outside the context region saturating the prediction.
4 EXPERIMENTS
We evaluate explanation algorithms on both shallow LSTM models and deep fine-tuned BERT Transformer (Devlin et al. 2018) models. We use two sentiment analysis datasets, namely the Stanford Sentiment Treebank (SST-2) dataset (Socher et al.) 2013) and the Yelp Sentiment Polar- ity dataset (Zhang et al. 2015), as well as TACRED relation extraction dataset (Zhang et al.) 2017) for experiments. The two tasks are modeled as binary and multi-class classification tasks respec- tively. For the SST-2 dataset, while provides sentiment polarity scores for all the phrases on the nodes of the constituency parsing trees, do not train model these phrases, and use these scores as the evaluation for the phrase level explanations. Our Transformer model fine-tuned from pretrained BERT (Devlin et al. 2018 model. See Appendíx for other implementation details.
Compared Methods. We compare our explanation algorithm with following baselines: Input oc-
clusion (Li et al., (2166) and Integrated Gradient+SHAP (GradSHAP) (Lundberg & Lee 2017): two
algorithms applied for hierarchical explanations, namely Contextual Decomposition (C) Mur-
doch et al. 2018), and Agglomerative Contextual Decomposition (ACD) (Singh al., 2019 We

---

Published as a conference paper at ICLR 2020
Dataset
SST-2
Yelp Polarity
TACRED
Model
BERT
LSTM
BERT
LSTM
BERT
LSTM
Metric
word /
phrase p
word p
phrase p
word P
word p
word p
word p
Input Occlusion
0.2229
0.4081
0.6489
0.4899
0.3781
0.6935
0.7646
0.5756
Direct Feed
0,2005
0.4889
0.6798
0.5588
0.3875
0.7905
0.1986
0.5771
GradSHAP
0,5073
0.5991
0,7024
0,5402
0,5791
0.7388
0.2965
0.6651
CD
0.2334
0.3068
0.6231
0.4727
0.2645
0.7451
0.0052
0.6508
ACD
0.3053
0.36998
0.2495
0.1856
0.3010
0.5024
0.2027
0.0291
Statistic
0,5223
0.4741
0.7271
0.4959
0.7294
0,9094
0.5324
0.7662
SCD
0.5481
0.6015
0.7151
0.5664
0,7180
0.7793
0.7980
0.6823
SOC
0.6265
0.6628
0.7226
0.5649
0.6971
0.7683
0.7982
0.7354
Table I: Correlation between word & phrase importance attribution and linear model coefficients & SST-2 human annotations, achieved by baselines and our explanation algorithms.
way, the film feels breath of fresh air, but only to those that allow it in
film foels like breath of fresh air but only those that allow in
feels like breath of fresh air, but only to those that allow
like breath of fresh air
a way
breath of fresh sir of fresh air
only allow
the film
breath
only those that allow it in
In
way
the
film
fools
like
breath
of
fresh
air
but
only
thoce that allow it
Figure 3: Hierarchical Explanation of a prediction by the BERT Transformer model on SST-2. We generate explanations for all the phrases on the truncated constitueccy parsing tree, with positive sentiments shown in red and negative sentiments shown in blue. We see our method identify positive segments in the overall negative sentence, such as "a breath of fresh air"
also compare with naive however neglected baseline prior literature, which directly feed the given phrase to the model and take the prediction score the importance of the phrase, noted as Di- rect Feed. BERT models, Direct Feed implemented by turning off the attention mask words except the [CLS] token and the phrase to be explained. For our algorithms, we list the performance of corpus statistic based approach (Statistic) for approximating context independent importance in Eq. 3 Sampling and Contextual Decomposition (SCD), and Sampling and Occlusion (SOC) algo- rithm. In section 4.4 we also consider padding instead of sampling the context words in SCD and SOC.
4.1 HIERARCHICAL VISUALIZATION OF IMPORTANT WORDS AND PHRASES
We verify the performance of our algorithms in identifying important words and phrases captured by models. We follow the quantitative evaluation protocol proposed in CD algorithm (Murdoch) et al. (2188) for evaluating word-level explanations, which computes Pearson correlation between the coefficients learned by a linear bag-of-words model and the importance scores attributed by ex- planation methods, also noted as the word p. When the linear model is accurate, its coefficients could stand for general importance of words. For evaluating phrase level explanations, we notice the SST-2 dataset provides human annotated real-valued sentiment polarity for each phrase on con- stituency parsing trees. We generate explanan explanations for each phrase on the parsing tree and evaluate the Pearson correlation between the ground truth scores and the importance scores assigned for phrases, also noted as the phrase p. We draw K = 20 samples for = 10 words adjacent the phrase to be explained at the sampling step in our SOC and SCD algorithms. The parameter setting is trade-off between the efficiency and performance. See Section 4.4 for detailed parameter analysis.
Table shows word and phrase p achieved by our algorithms and competitors. Generally, explana- tion algorithms that follow our formulations achieve highest word and phrase for all the datasets and models. SOC and SCD perform robustly on the deep Transformer model, achieving higher word
7

---

Published as a conference paper at ICLR 2020
per:siblings
He
18
by
SPERSCM
parentt
and
sister
of
and
parents
sister
by
SPCCSOM
parents
and
sister
(a) SCD
per:siblings
by
parents
and
wiso by
also
www.
by SPRRSM parents
of
Wereepe
(b) CD
Figure 4: Extracting phrase-level classification patterns from LSTM relation extraction model with SCD. Red indicate evidence for predicting the class, and blue indicate distractor for predicting the class, By applying the agglomerative clustering algorithm and defining threshold score, we effe- cively extract "a sister, O-Person" classification rule for the relation per:siblings. However, we see CD fails in this example.
and phrase o than input occlusion and contextual decomposition algorithms by a large margin. We see the simple Direct Feed method provide promising results shallow LSTM networks, but fail in deeper Transformer models. The statistic based approximation of the context independent impor- tance, which do not employ trained sampler, yields competitive words p but low phrase p. Our analysis show that it is common that a long phrase does not exist in previously seen examples. In this case, phrase p achieved by statistic based approximation is pushed towards that of the input occlusion algorithm.
Qualitative study also shows that explanation visualize complicated compositional semantics captured by models, such as positive segments in the negative example, and ad adversative conjunctions connected with "but". We present an example explanation provided by SOC algorithm in Figure and Appendix.
4.2 EXPLANATION AS CLASSIHCCATION PATTERN EXTRACTION FROM MODELS
We show our explanation algorithm is a nature fit for extracting phrase level classification rules from neural classifiers. With the agglomerative clustering algorithm in [Singh al. (2019), our explana- tion effectively identify phrase-level classification patterns without evaluating all possible phrases in the sentence even when predefined hierarchy does exist. Figure show example of au- tomatically constructed hierarchy and extracted classification rules in an example in the TACRED dataset.
4.3 ENHANCING HUMAN TRUST OF MODELS
We follow the human evaluation protocol in Singh al. (2019) and study whether explanations help subjects to better trust model predictions. We ask subjects to rank the provided visualizations based how they would like trust the model. For the SST-2 dataset, we show subjects the predictions of the fine-tuned BERT model, and the explanations generated by SOC, SCD, ACD and GradSHAP algorithms for phrases. The phrase polarities are visualized in hierarchy with the provided parsing tree of each sentence in the dataset. For the TACRED dataset, we show the expla- nations provided by SOC, SCD, CD and Direct Feed algorithms on the LSTM model. We binarilize the importance of a phrase by calculating the difference between its importance to the predicted class and the its top importance to other classes, and the hierarchies are constructed automatically with agglomerative clustering (Singh et al. 2019). Figure 5 showw average ranking score of explanations, where 4 for the best, and for the worst. On the SST-2 dataset, SOC achieves significantly higher ranking score than ACD and GradSHAP, showing p-value less than 0.05 and 0.001 respectively. On the TACRED dataset, SCD achieve the best ranking score, showing significantly better ranking score than CD and Direct Feed with p-value less than 60-6
8

---

Published as a conference paper at ICLR 2020
GradSHAP
2.19
DirectFeed
1.60
ACD
2.40
CD
2.43
SCD
2.64
SCD
3.23
SOC
2.77
SOC
2.80
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Mean Trust Score
Mean Trust Score
(a) SST-2
(b) TACRED
Figure 5: Results for human evaluation on the Transformer model trained on SST-2 sentiment anal- ysis dataset (between SOC, SCD, ACD, GradSHAP) and the LSTM model trained TACRED relation extraction dataset (between SOC, SCD, CD, DirectFeed).
0.49
0.46
0.53
0.50
0.83
0.48
Fudding canpoc
Fadding
0.35
0.2
032
vcaress
0.60
22
0.00
22
14
3323
(a) SCD, word A BERT (b) SCD, phrase p. BERT (c) SOC, word p. BERT (d) SOC, phrase p. BERT
Figure 6: Word p and phrase p curves the size of the context region N and the number of samples K change on the BERT model trained on the SST-2 dataset. Dash line notes for the performance of padding the context words instead of sampling.
4.4 PARAMETER ANALYSIS
Both SOC and SCD algorithms require specifying the size of the context region N and the number of samples K. In Figure 6 (also Figure in Appendix) we show the impact of these parameters. We also plot the performance curves when we pad the contexts instead of sampling. We see sampling the context achieves much better performance than padding the context given a fixed size of the context region N. We also word p and phrase increase the number of samples K increases. The overall performance also increases as the size of the context region increases at the early stage, and saturates when N grows large, which is consistent with hypothesis that words phrases usually do not interact with the words that are far away them in the input.
RELATED WORKS
Interpretability of neural networks has been studied with vairous techniques, including probing learned features with auxiliary tasks Tenney et al. 2019), or designing models with inherent in- terpretability (Bahdanau et al., 2015 Lei et al. 2016). A major line of work, local explanation algorithms, explains predictions by assigning importance scores for input features. This line of work include input occlusion (Kádár et al. 2017), gradient based algorithms (Simonyan et al. 2013 Hechtlinger 2016; Ancona et al. 2017) ). additive feature attribution methods (Ribeiro al., Shrikumar et al.) 2017; Sundararajan et al. (017). among which Shapley value based ap- proaches (Lundderg Lee, (017) have been studied intensively because its good mathematical properties. In the context of input occlusion explanations, researchers also study how to efficiently marginalize over input features to be explained (Zintgraf et al.) 2017 Chang et al. 2019), while our research show extra focus could be placed of models with structured inputs, Chen et al. (2019) propose L-Shapley and C-Shapley for efficient approximation of Shapley values, with similar hypothesis with us that the importance of word usually only strongly dependent on its neighboring contexts. Chen et al. (2018) propose a feature selection based approach for explanation in information theoretic perspective.
2016
marginalizing over contexts. Regarding explanations

---

Published as a conference paper at ICLR 2020
On the other hand, global explanation algorithms (Guidotti et al., 2018) have also been studied for identifying generally important features, such as Feature Importance Ranking Measure (Zien et al, 2009). Accumulated Local Effects (Apley) 2016), while usually restricted in the domain of tabular data. We note that the context independence property in our proposed methods implies we study hi- erarchical explanation as global explanation problem (Guidotti al.. 2018). Compared with local explanation algorithms, global explanation algorithmss are less studied for explaining individual pre- dictions (Poerner al. 2018), because they reveal the average behavior of models. However, with hierarchical organization, we show global explanations are also powerful at explaining individual predictions, achieving better human evaluation scores and could explain compositional semantics where local explanation algorithms such as additive feature attribution algorithms totally fail. More- over, we note that the use of explanation algorithms is not exclusive; we may apply explanation algorithms of different categories to make a more holistic explanation of model predictions.
Another closely related field is statistical feature interaction detection (Hooker [004 (Sorokina et al.) 2008 Tsang al. 2017) from learned models, which usually focus tabular data only. An excep- tion iS sang et al. 2018), which also studies word interactions in neural sequence model predic- tions, but does not study interactions in phrase level.
CONCLUSION
In this work, we identify two desirable properties for informative hierarchical explanations of pre- dictions, namely the non-additivity and context-independenee. We propose a formulation to quantify context independent importance of words and phrases that satisfies the properties above. We revisit the prior line of works on contextual decomposition algorithms, and propose Sampling and Contex- tual Decomposition (SCD) algorithm and Sampling and Occlusion algorithm (SOC). Experiments on multiple datasets and models show that our explanation algorithms generate informative hierar- chical explanations, help extract classification rules from models, and enhance human trust of models.
ACKNOWLEDGEMENTS
This research is based upon work supported in part by NSF SMA 18-29268 and JP Morgan AI Research Award, We would like to thank all the collaborators USC INK research lab for their constructive feedback on the work.
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104 2017.
Daniel W Apley. Visualizing the effects of predictor variables in black box supervised learning models. arXiv preprint arXiv.1612.08468. 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
org abs 1409,440900 ( 0473
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin. Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference Learning Representations, 2019. MtRLitttss///penreet
net. httt/foormmiiiie.e MMx20
Jianbo Chen, Le Song. Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretie perspective model interpretation. In ICML. 2018.
10

---

Published as a conference paper at ICLR 2020
Jianbo Chen, Le Song. Martin J. Wainwright, and Michael I. Jordan. L-shapley and e-shapley: Efficient model interpretation for structured data. In Interattoonal Conference on Learning Rep- resentations, 2019. URL http::t:pptppeetaaaaaaaa http://eeeetttoomeeeseee11-
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
Katsushige Fujimoto, Ivan Kojadinovic, and Jean-Luc Marichal. Axiomatic characterizations of
probabilistic and pardinal-probabilitic interaction indices. Games and Economic Behavior, 55 (1):72-99, 2006.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):93, 2018.
Yotam Hechtlinger. Interpretation of prediction models using the input gradient. arXiv preprint 16Xi::611.07000 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Giles Hooker. Discovering additive structure in black box functions. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 575-580, 2004.
Akos Kádár, Grzegorz Chrupata, and Afra Alishahi. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780, 2017.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In EMNLP, 2016.
Jiwei Li, Will Monroe, and Dan Jurafsky, Understanding neural networks through representation erasure. arXiv preprint / 2016.
Hui Liu, Qingyu Yin, and William Yang Wang. Towards explainable NLP: generative explanation framework for text classification. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 5570-5581, Florence, Italy, July 2019a. Association for Compu-
tational Linguisties. URL https: / www. acl www.wb.ww..b........... http://wwwwwww..////tt..eoe///.e wwt..//ww...rgg/.attooooo///////
1560
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoee, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907 2019b.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4774,, 2017.
Sina Mohseni, Niloofar Zarei, and Eric D Ragan. A survey of evaluation methods and measures for interpretable machine learning, arXiv preprint arXiv: 18t.::881..11999 2018.
W James Murdoch, Peter Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. In ICLR, 2018.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In P Proceedings of the 2014 conference empiricul methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint wrXiv:1802.05365, 2018.
Nina Poerner, Hinrich Schütze, and Benjamin Roth. Evaluating neural network explanation methods using hybrid documents and morphological agreement. arXiv preprint arXiv: 0.6422, 2018.

---

Published as a conference paper at ICLR 2020
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13- 2016, pp- 1145-1144, 2016.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28):307- 317, 1953.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference Ma- chine Learning-Volue Volume 70, pp. 3145-3153, JMLR. org, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6034 2013.
Chandan Singh, W James Murdoch, and Bin Yu. Hicrarchical interpretations for neural network
predictions. In ICLR, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631 1631-1642, 2013.
Daria Sorokina, Rich Caruana, Mirek Riedewald, and Daniel Fink. Detecting statistical interactions with additive groves of trees. In Proceedings of the 25th international conference Machine learning, pp. 1000-1007. ACM, 2008.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Voile Volume 70, pp. 3319- 3328. JMLR. org, 2017.
lan Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Associa- tion for Computational Linguistics, 2019. httlaattt/:aaxxtt.. wwwpiiivv..///a..oooo abs 905.0050000 05950
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. arXiv preprini arXiv: 1705: 705.04977, 2017.
Michael Tsang, Youbang Sun, Dongxu Ren, and Yan Liu. Can i trust you more? model-agnostic hierarchical explanations. ArXiv, (tss/812..488) 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Kyunghyun Cho, and CIFAR Azrieli Global Scholar. Bert has mouth, and it must speak: Bert as markov random field language model. NAACL HLT 2019, pp. 30, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pp. 35-45, 2017. URL F
Alexander Zien, Nicole Krämer, Sören Sonnenburg, and Gunnar Rätsch. The feature importance ranking measure. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 694-709. Springer, 2009.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, 2017.
12

---

Published as a conference paper at ICLR 2020
0.55
0.90
0.49
0.36
066
0.4
Fxd81g
Fx68
0.22
carpap
0.52
0.2
(a0li (mak)
(a) SCD, word p. LSTM (b) SCD, phrase p. LSTM (c) SOC, word p. LSTM (d) SOC. phrase p. LSTM
Figure 7: Word p and phrase p curve as the size of the context region and the number of samples K change L.TT trained the SST-2 dataset. Dash line notes for the performance of padding the context words instead of sampling.
A IMPLEMENTATION DETAILS
Our LSTM classifiers layer unidirectional LSTM and the number of hidden units is set to 128, 500, and 300 for SST-2, Yelp, and TACRED dataset respectively. For all models, we load the pretrained 300-dimensional Glove word vectors (Pennington al. 2014). The language model sam- pler is also built on LSTM and have the same parameter settings as the classifiers. Our Transformer models are fine-tuned from pretrained BERT models (Devlin al.) 2018), which have 12 layers and 768 hidden units of per representation. On three datasets, LSTM models achieve 82% accuracy, 95% accuracy, and 0.64 F1 score on average. The fine-tuned BERT models achieve 92% accuracy, 96% accuracy, and 0.68 F1 score on average. We use the same parameter settings between LSTM classifiers and language models on three datasets. Following Murdoch et al. (2018), we randomly sample subset (set to 500 instances) of sentences of length at most 40 words for explanation on Yelp dataset, and also for TACRED dataset. On TACRED dataset, we generate explanations only for correctly predicted instances with label other than
B PERFORMANCE ON ADVERSARIAL. MODELS
For computing context independent importance of phrase. intuitive and simple alternative ap- proach, which nevertheless neglected in prior literature, is to only feed the input to the model and treat the prediction score the explanation. In Table while the score of the Direct Feed is lower than that of the best performing algorithms, the score is rather competitive.
The potential risk of this explanation is that it assumes model performs reasonably incomplete sentence fragments that are significantly out of the data distribution. As result, the explanation of short phrases can be misleading. To simulate the situation, we train LSTM model on inversed labels on isolate words, in addition to the original training instances. The model could achieve the same accuracy the original LSTM model. However, the word p and the phrase p of Direct Feed drop by a large margin, showing word of -0.38 and 0.09. SOC and SCD are still robust on the adverse L.TM model, both showing a word p and phrase p of more than 0.60 and 0.55.
The masking operation could also cause performance drop because the masked sentence can be out of data distribution when explaining long phrases. For SOC, the risk can be resolved by implement- ing the masking operation of the phrase by another round of sampling from language model conditioned on its context x x_p: p: but we do not find empirical evidence showing that improves performance.
C EXPLANATION HEATMAPS
13

---

Published as a conference paper at ICLR 2020
Model
BERT
LSTM
Metric
word Pave
phrase Pavg
word Pave
phrase Pave
Input Occlusion
0.2947
0.4365
0.7150
0,4968
GradSHAP
0.5444
0.6078
0.7360
0.5449
CD
0.3642
0.3755
0,6962
0,4777
ACD
0.3023
0.3913
0.3210
0.2110
SCD
0.5601
0.6072
0.7348
0.5685
SOC
0.6282
0.6706
0.7430
0,5675
Table 2: Correlation between averaged word phrase importance attribution and linear model coefficients and human annotations over all 2210 test instances in SST-2 dataset. The relative per- formance similar to the case without score averaging as shown in Table
per.age
S-CESON
man
from
Canada
accused of aoosti Aquash
in
the
head
D HirBac
man
Aquash In head
O HUNBG
man
man
gegttpp_memberr//mplllyees
ather
side
passage
0-PEROO
the BECEEES
O-RERSON
side
O-PERSON
mhsiman
the 2-10 ELLEIOO
pred
side
O-PERSON
chalman of the pred ctod
per:sibiings
The
under
arrest
brother
face charges
Mafia
Jele
Figure 8: More examples about rule extraction from LSTM models trained on TACRED relation extraction dataset with SOC. Red indicate evidence for predicting the class, and blue indicate dis- tractor for predicting the class. By applying the agglomerative clustering algorithm and defining threshold score, we effectively extract classification rules from LSTM models. The ground truth label noted on the top
Dataset
Label
Pattern
Positive
frighteningly evocative; insight and honesty neither funny nor provocative; kill the suspense
SST-2
Negative
person:age
[PERSON], [NUMBER], was; a [NUMBER] man chief engineer of the [ORGANIZATION] [Nationality] citizen; tribal member from [COUNTRY]
TACRED
organization:too-member
person:origin
Table Phrase-level classification patterns extracted from models. We show the results of SCD and SOC respectively for the SST-2 and the TACRED dataset.
14

---

Published as a conference paper at ICLR 2020
sustined
*ouunders
Meaning
chis
and
quest
Deeper
Meaning
sustained
Meaning
sustained
fluunders
Selierr
some
chil
and
sustained unease
but
delvers
suaaained unease
but
quess
For
Deeper Maaning
(a) SOC
deivers
MOTT
chill
and
sindained LTTE
but Bounderr in Rx queal For
Desper Meunin
QuesS Delper Meaning
sustained
Meaning
Gelvers some
chilk and sustained uneeee
Blurdders In queett For Deeper Meening
wid sundaired uTTeA
but
sustained
*ounnerrs quest For
Meaning
delivers some
chil and
sustained unease
but flounders In
Deeper Meaning
(b)
SCD
50me
guest Meaning
chis und
sustained
for
Deeper Meaning
Solivers s0me
sustained unesse
quest for
Moaning
delivers
SORT chilk and sindained
uneeaa
bui
Bounderrs in
quess For Deeper Memnig
delivers some sustained
but
quesst for Deeper Meaning
(c) CD
deivers chilk
and Sind
but flounders
For
Dewper Meining
chils
und
Ra quest
Deeper Mouning
Meaning
chik
susdained
Hounddrr
Meaning
Selivers
50me
chis
3nd
sustained unesse
but
20me
sustained
QUeSt
Meaning
sustained
"lounders
Meaning
(d) GradSHAP
Figure 9: Explanation heatmaps generated by SOC, SCD, CD, and GradSHAP on a negatively predicted sentence by BERT Transformer model in SST-2 dataset. Only SOC and SCD captures adversarial conjunction connected by "but".
15


Reviews:
[{"review_id": "r1gOVjXf5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors proposed a method for generating hierarchical importance attribution for any neural sequence models (LSTM, BERT, etc.) Towards this goal, the authors propose two desired properties: 1) non-additivity, which means the importance of a phrase should be a non-linear function over the importance of its component words; 2) context independence, which means that the attribution of any given phrase should be independent of its context. For example, in the sentence \"the film is not interesting\", the attribution of \"interesting\" should be positive while the attribution of \"not interesting\" should be negative.\n\nFollowing these two properties, the authors designed three algorithms to post-hoc analysis the importance of a given phrase p.\n1. [Sec 3.2] eq 4. expected differences in model predictions between the a sentence that contains p and the same sentence with p removed. The expectation is computed over the conditional probability Prob(sentence | p in sentence). In practice, the authors use eq 3 as a proxy to eq 4.\n2. [Sec 3.3] eq 5.  expected differences in the activation values of each layer. The expectation is computed over the conditional probability Prob(context-dependent representations | phrase-dependent representations).\n3. [Sec 3.4] eq 8. similar to 1 but we replace the phrase p with padded tokens.\n\nThe authors conducted experiments on SST and Yelp. Results show that their proposed context-independent attribution correlates better with a trained linear model's coefficient, achieves higher human trust.\n\nDecision: reject.\n\nWhile I found the idea of marginalizing out the local context interesting, I think the paper still needs more work on its formulation, experiments and writing.\n\nFormulation:\n1. In eq 3, the expectation is taken over the difference between the prediction on the sampled sentence and the one with the phrase removed. This may be problematic for longer inputs (a pargraph), where the overall prediction may not change a lot when you remove a single phrase (since the evidence is everywhere). For example, consider the input: \"The movie is the best that I have ever seen. It is remarkable!\". Removing the word \"best\" alone doesn't alter the prediction much. \n2. In eq 5, the expectation is computed over P(h | beta). It is NOT THE SAME as sampling words p(x_{\\delta} | x_{-\\delta}) and then consider their hidden states.\n3. In Sec 3.1, you mentioned that CD is limited since the decomposition of activation sigma evolves context information gamma, and you resolved this by marginalization. But it seems to me that the computation of element wise multiplication also evolves context information. How do you deal with these?\n3. What's the difference between eq3 and eq8? Are you just changing from remove the phrase completely to replace it by mask?\n\nExperiments:\n1. The performance of CD in Table 1 seems very different to the original CD paper (which is 0.758 for SST and 0.520 for Yelp). I am not sure what contributes to this big difference. Is it the trained model or data splits?\n2. Table 1 shows that your methods achieves higher correlation to linear model's coefficients. But why shall we consider linear model's coefficients as the ground truth for the learned neural model? For example, the fine-tuned BERT achieves lower correlation corresponding to the LSTM. Does that mean the BERT model performs worse than LSTM?\n\nMissing related references:\n1. Explaining Image Classifiers by Counterfactual Generation\n2. Rationalizing Neural Predictions\n3. Learning to Explain: An Information-Theoretic Perspective on Model Interpretation\n4. L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgBoPd0Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - The paper addresses the problem of hierarchical explanations in deep models that handle compositional semantics of words and phrases. The paper first highlights desirable properties for importance attribution scores in hierarchical explanations, specifically, non-additivity and context independence, and shows how prior work on additive feature attribution and context decomposition doesn’t accurately capture these notions. After highlighting the said properties in context of related work, the authors propose an approach to calculate the context-independent importance of a phrase by computing the difference in scores with and without masking out the phrase marginalized over all possible surrounding word contexts (approximated by sampling surrounding context for a fixed radius under a language model). Furthermore, based on the above, the authors propose two more score attribution approaches -- based on integrating the above sampling step with (1) the contextual decomposition pipeline and (2) the input occlusion pipeline. Experimentally, the authors find that the attribution scores assigned by the proposed approach are more correlated with human annotations compared to prior approaches and additionally, the generated explanations turn out to be more trustworthy when humans evaluate their quality.\n\nStrengths\n\n- The paper is well-written and generally easy to follow. The authors do a good job of motivating and highlighting the desired properties of importance attribution scores and developing the proposed scoring mechanism. The proposed scoring mechanism ties in seamlessly with the existing contextual decomposition and occlusion pipelines and leads to improved performance when the generated explanations are evaluated.\n\n- The proposed approach involving masking out the phrase and marginalizing over possible surrounding word-concepts is novel and offers an interesting perspective on how to approach context independent scoring of phrases -- (1) phrases don’t exist independent of the surrounding context and therefore marginalizing over all possible surrounding concepts makes sense and (2) replacing the intractable enumeration over all possible surrounding concepts with samples from a language model makes the score attribution process faster and more scalable modulo the learnt language model.\n\n- Sec. 4.4 offers interesting insights. I like that the authors performed this ablation given that the expectation over surrounding contexts is computed approximately via samples under a language model. There’s a clear increase in terms of the attribution scores as the number of samples increases and the neighborhood size is increased. It is interesting to note that there is an approaching plateau region where increasing the neighborhood size won’t affect the assigned scores. This experiment provides a holistic picture of the behavior of the interpretability toolkit (manifesting in terms of attribution scores) given the approximations involved. I would encourage the authors to flesh this out even more. \n\nWeaknesses\n\nHaving said that, there are some minor comments that I’d like to point out / get the authors’ opinion on. Highlighting these below:\n\n- While SOC and SCD don’t always end up outperforming other approaches (specifically Statistic) on the SST-2, Yelp and TACRED datasets (Table. 1), for the human evaluation experiments, the authors only compare with CD, Direct Feed, ACD and GradSHAP. Do the authors have any insights on how well does Statistic perform on the human-evaluation set of experiments?\n\n- While inspiring trust in users is one aspect of evaluating explanations via humans, it’s slightly unclear what ‘trust’ in this sense inherently identifies. Although, it might implicitly capture some notion of reliability (and predictability of the explanations by humans), asking users to rank explanations across a spectrum of ‘best’ to ‘worst’ doesn’t explicitly capture that. Another possible aspect to look into could be -- ‘’Do the generated explanations help humans predict the output of the model?’’ This captures reliability in a very explicit sense. Do the authors have any thoughts on this and potential experiments that might address this? I don’t think not addressing this is necessarily detrimental to the paper but I’m curious to hear the thoughts of the authors on the same. \n\nReasons for rating\n\nBeyond the above points of discussion, I don’t have major weaknesses to point \tout. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. The proposed phrase attribution scoring mechanism is motivated from a novel perspective and has a reasonable approximation characterized appropriately by the ablations performed. The strengths and weaknesses highlighted above form the basis of my rating.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJey9CJQqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a hierarchical decomposition method to encode the natural language as mathematical formulation such that the properties of the words and phrases can encoded properly and their importance be preserved independent of the context. This formulation is intuitive and more efficient compared to blindly learning contextual information in the model. The proposed method is a modification of contextual decomposition algorithm by adding a sampling step. They also adapt the proposed sampling method into input occlusion algorithm as another variant of their method. The proposed method is tested on LSTM and BERT models over sentiment datasets of Stanford Sentiment Treebank-2 and Yelp Sentiment Polarity and TACRED relation extraction dataset and showed more interpretable generated hierarchical explanations compared to baselines.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a hierarchical explanation model for understanding the underlying representations produced by LSTMs and Transformers.  Using human evaluation, they find that their explanations are better, which could lead to better trust of these opaque models.

The reviewers raised some issues with the derivations, but the author response addressed most of these.  
Decision: Accept (Spotlight)