Asymmetric feature interaction for interpreting model predictions
Year: 2023

Abstract:
In natural language processing (NLP), deep neural networks (DNNs) could model complex interactions between context and have achieved impressive results on a range of NLP tasks. Prior works on feature interaction attribution mainly focus on studying symmetric interaction that only explains the additional influence of a set of words in combination, which fails to capture asymmetric influence that contributes to model prediction. In this work, we propose an asymmetric feature interaction attribution explanation model that aims to explore asymmetric higher-order feature interactions in the inference of deep neural NLP models. By representing our explanation with an directed interaction graph, we experimentally demonstrate interpretability of the graph to discover asymmetric feature interactions. Experimental results on two sentiment classification datasets show the superiority of our model against the state-of-the-art feature interaction attribution methods in identifying influential features for model predictions.

heading: 1 Introduction
text: Deep neural networks (DNNs) have demonstrated impressive results on a range of Natural Language Processing (NLP) tasks. Unlike traditional models (e.g. CRFs and HMMs) that optimize weights on human interpretable features, deep neural models operate like a black box by applying multiple layers of non-linear transformation on the vector representations of text data, which fails to provide insights to understand the inference process of deep neural models over the features (e.g. words and phrases) involved in modeling.
Interpreting the prediction of a black box model could help understand model inference behaviors and increase user trust in applying the model to realworld applications. Prior efforts in NLP mainly focus on quantifying the contributions of individual
∗ Corresponding author.
word or word interactions to the prediction. Fig.1 demonstrates word-level and pairwise word interaction explanations for a sentiment classification task, where the word “not” and the interaction between “not” and “funny” contribute positively to the prediction Negative.
Studying word interaction could help identify to what extent a set of words exert influence in combination as opposed to independently. However, most interaction attribution methods assume symmetric interaction, which may fail to capture asymmetric influence that contributes to model prediction. Fig. 2 presents some symmetric pairwise interactions with graph representation for the instance in Fig. 1, where words becomes nodes and edges
4666
between words represent interaction. In individuallevel explanation “funny” has negative influence while the symmetric interaction between “funny" and “not" produces positive influence to model prediction. Therefore the influence of the presence of “not” to “funny” is not the same as that of the presence of “funny” to “not". “funny” has weak positive contribution in the presence of “not” and “funny” further modifies “not”, the interaction influence of “funny” → “not” could be stronger than the interaction of “not” → “funny”. For the ideal asymmetric interaction, the presence of other features should not negate the positive influence of the important feature and other features would lose their influence when important features are present. Constructing asymmetric interaction graph for the predicted instance could help human have a nuanced understanding toward the inference of deep NLP models.
In this paper, our work aims to provide the explanation that incorporates asymmetric feature interaction 1. The contributions are summarized as follows:
• We propose an asymmetric feature interaction attribution method that incorporates asymmetric higher-order feature interactions toward explaining the prediction of deep neural NLP models.
• We investigate three different sampling strategies in NLP field for computing marginal contribution of our defined asymmetric feature interaction attribution score, and empirically show that none is generally better than the others or more broadly applicable.
• We evaluate the proposed model on two sentiment classification datasets with BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019), and the experimental results demonstrate the faithfulness of our explanation model.

heading: 2 Related work

heading: 2.1 Feature attribution explanation
text: Most explanation methods mainly focus on modelagnostic explanations and study how to effectively measure the importance of features on the prediction. For example, LIME (Zhang et al., 2019) evaluates the contribution of each feature by learning
1Our code is available at https://github.com/ StillLu/ASIV.
a linear model locally around a instance. Shapley value (Shapley, 1997; Lundberg and Lee, 2017) estimates the influence of a feature by averaging its marginal contribution among all permutations. Since the computation of Shapley value is computationally expensive, popular variants like Kernel SHAP (Lundberg and Lee, 2017) and Quasirandom and adaptive sampling (Štrumbelj and Kononenko, 2014) are proposed to efficiently approximate Shapley value. However, these methods do not explain how feature interactions contribute to model predictions, which fails to address model’s learning capability from high-order feature interactions.

heading: 2.2 Feature interaction explanation
text: There are increasingly research on studying feature interaction explanation methods. For example, Shapley interaction index (Grabisch, 1997) and Shapley Taylor interaction index (Dhamdhere et al., 2019) are proposed to measure the interaction between multiple players. Integrated Directional Gradients (IDG) (Sikdar et al., 2021) borrowed axioms from Integrated Gradients (IG) (Sundararajan et al., 2017), where the desirable characteristics are satisfied by IG and Shapley value in cooperative game theory. Tsang et al. (2020) proposed an efficient framework Archipelago to combine feature interaction detector (ArchDetect) and feature attribution measure (ArchAttribute).
Further, feature interaction could be explained in a hierarchical structure. Agglomerative contextual decomposition (ACD) (Singh et al., 2018) builds the hierarchical explanations in a bottom-up way by starting with individual features and iteratively combining them based on the generalized CD scores. Jin et al. (2019) addressed context independent importance that is ignored in ACD, and proposed an easy and model-agnostic Sampling and Occlusion (SOC) algorithm to incorporate conditional context information given the specified text sequence. HEDGE (Chen et al., 2020) designs a top-down framework to construct hierarchical explanations by detecting the weakest interaction point and selecting important sub-span for a given text span.
The above feature interaction explanation methods only focus on symmetric interaction. We note that concurrent work by Masoomi et al. (2021) addressed directed pairwise interaction. However, their Shapley value based formulation also intro-
duces noisy interaction as different subsets may contain several same elements. Moreover, this solution ignores asymmetric high-order interaction.

heading: 3 Asymmetric Shapley interaction value
text: This section first revisits Shapley value for explaining model prediction, then describes the definition of asymmetric Shapley interaction value and the corresponding approximating computation.
The following notations will be used throughout the paper. For a classification task, given a text sequence x = (x1, ..., xn) and a trained model f , ŷ is the prediction label and f(·) denotes the model output probability on ŷ.

heading: 3.1 Shapley value for model interpretability
text: In cooperative game theory Shapley value measures the marginal contribution that a player makes upon joining the group by averaging over all possible permutations of players in the group.
The Shapley value of ith word in x for the model prediction ŷ is weighted and summed over all possible word combinations:
ϕ(i) = ∑
S⊆N\i (n−1−|S|)!|S|!
n! [v(S ∪ i)− v(S)] , (1)
where S is the subset of feature indices. The equivalent formulation is
ϕ(i) = 1
n!
∑
O∈π(n) v(prei(O) ∪ i)− v(prei(O)),
(2) where π(n) denotes all permutation of the word indexes {1, 2, ..., n}. prei(O) is the set of all indices that precede i in the permutation O ∈ π(n). v(S) is the value function that characterizes the contribution of the subset S to the prediction ŷ:
v(S) = E(f |xS ∪ x′S̄)− E(f |x′), (3)
where x′ denotes the text sequence with the same length as x, S̄ ⊆ N \ S. E(f |xS ∪ x′S̄) is the expectation of f(·) over possible x′ where only the subset values xS unchanged.

heading: 3.2 Definition of Asymmetric Shapley interaction value
text: The Shapley value above can quantify the contribution of a single word or phrase to the model prediction. The proposed asymmetric Shapley interaction value (ASIV) measures the asymmetric
interaction between two different subsets T1 and T2 that attributes to the model prediction. That is, ASIV determines the contribution of T1 conditioned on the presence of T2 to the prediction ŷ. By treating T1 and T2 as two singletons among the players, ASIV is defined as
ϕT2(T1) = C1 ∑
T2⊆S⊆N\{T1} ∆T1v(S)−∆T1v(S\T2),
(4) where C1 =
(n−|T1|−|T2|+1−|S|)!|S|! (n−|T1|−|T2|+2)! . ∆T2v(S) and
∆T1(S) are given as
∆T1v(S) = v(S ∪ T1)− v(S), (5)
∆T1v(S\T2) = v(S\T2 ∪ T1)− v(S\T2). (6) ∆T1v(S) − ∆T1v(S\T2) computes the difference of marginal contribution of T1 to the coalition S with and without participation of the subset T2, which aims to capture directional interaction influence between T1 and T2. The equivalent formulation is
ϕT2(T1) = C2 ∑ O ∆T1v(pre T1(O))−∆T1v(preT1T2(O)), (7)
where O denotes the possible permutation where T2 precedes T1, and C2 = 1(n−|T1|−|T2|+2)!/2 . ∆T1v(pre T1(O)) and ∆T1v(preT1T2(O)) are defined as
∆T1v(pre T1(O)) = v(preT1(O) ∪ T1)− v(preT1(O)), (8) ∆T1v(pre T1 T2 (O)) = v(preT1T2(O) ∪ T1)− v(pre T1 T2 (O)), (9) where preT1 denotes the set of all indices that precede T1 while preT1T2 excludes T2 from this set.
If both T1 and T2 contain a single element (i.e. T1 = {i} and T2 = {j}), the directed pairwise relationship could be obtained by
ϕj(i) = C3 ∑
j⊆S⊆N\{i} v(S ∪ {i})− v(S) − ( v(S\ {j} ∪ {i})− v(S\ {j}) ) , (10)
where C3 = (n−1−|S|)!|S|!
n! .

heading: 3.3 Approximating computation
text: Computing asymmetric Shapley interaction value has to estimate value function v(S) over all possible permutations (or subsets). First, we investigate
three different sampling strategies of computing E(f |xS ∪ x′S̄).
Marginal Expectation (ME): In applying Shapley value to explain predictions of NLP models, prior research assumes individual features are mutually independent. Then E(f |xS ∪ x′S̄) is computed as
E(f |xS ∪ x′S̄) = Ep(x′S̄)(f |xS ∪ x ′ S̄), (11)
where x′ S̄
could be randomly sampled from training data or a sequence with all ⟨pad⟩ tokens. In computing f(xS ∪ x′S̄), the combination of{ xS ∪ x′S̄ } may be incompatible. For example, given the instance x “the issue of faith is not explored deeply” and the subset xS “the issue of faith”, random sampling could generate the sequence “the issue of faith time changer may not”. Such incoherence lie off the data manifold (Frye et al., 2020) where v(S) may fail to capture model’s dependence on the whole context information.
Conditional Expectation (CE): The expectation of (f |xS ∪x′S̄) with respect to the distribution x′ S̄ conditioning on xS is computed as
E(f |xS ∪ x′S̄) = Ep(x′S̄ |xS)(f |xS ∪ x ′ S̄), (12)
where x′ S̄ |xS could be sampled from a pre-trained language model. In conditional expectation, the combined input xS ∪ x′S̄ is more coherent. For example, given the subset xS “the issue of faith”, the generated complete sequence could be “the issue of faith is not very important”.
However, both marginal expectation Ep(x′
S̄ )(f |xS ∪ x′S̄) and conditional expecta-
tion Ep(x′ S̄ |xS)(f |xS ∪ x′S̄) have to evaluate f on some out-of-domain data, where the model f is forced to extrapolate to an unseen part of the feature space. Hooker et al. (2021) conducted many simulation experiments showing that the permutation-based feature attributions are sensitive to these edge cases.
In-domain Expectation (IDE): To enable f to evaluate on in-domain data, we pretrain a language model on the training data x = {x1, ...,xN} to model the underlying data distribution p(x). For a text sequence x′ with known subset x′S = xS , by sampling the remaining subset x′
S̄ from p(x),
E(f |xS ∪ x′S̄) is computed as
E(f |xS ∪x′S̄) = Ex′S̄ |xS∼p(x)(f |xS ∪x ′ S̄). (13)
Second, by enumerating all possible permutations, Eq.(7) could be replaced with
ϕT2(T1) = C2 ∑
O
∑
w
p(w)f(w[wj=xj ,j∈preT1 (O)∪T1])
− f(w[wj=xj ,j∈preT1 (O)]) − [ f(w
[wj=xj ,j∈preT1T2 (O)∪T1] )
−f(w [wj=xj ,j∈preT1T2 (O)] )
] ,
(14) where w could be obtained from the above three different sampling strategies.
Let VO,w for all permutation/instance pairs,
VO,w =f(w[wj=xj ,j∈preT1 (O)∪T1])− f(w[wj=xj ,j∈preT1 (O)])
− [ f(w
[wj=xj ,j∈preT1T2 (O)∪T1] )− f(w [wj=xj ,j∈preT1T2 (O)] )
] ,
(15) we adopt Monte Carlo sampling (Štrumbelj and Kononenko, 2014) to approximate ϕT2(T1) as
ϕT2(T1) = 1
m
m∑
j=1
Vj . (16)

heading: 4 Experiments
text: We evaluate explanation methods on text classification tasks with BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) models.

heading: 4.1 Datasets and classification models
text: We use Stanford Sentiment Treebank-2 (SST-2) dataset with 6920/1821 in train/test sets (Socher et al., 2013) and the Yelp Sentiment Polarity (Yelp2) dataset with 560, 000/38, 000 in train/test sets (Zhang et al., 2015). The average review length in SST-2 test set is 19.25 and 136.49 in Yelp2 test set. Also, we employ pretrained BERTbase and RoBERTa-base models, and then finetune them in the downstream classification tasks. BERT achieves accuracy 91.15% on SST-2 and 96.39% on Yelp-2, and RoBERTa achieves accuracy 94.39% on SST-2 and 96.94% on Yelp-2.

heading: 4.2 Baselines
text: We compare our method against five modelagnostic feature interaction attribution methods: HEDGE (Chen et al., 2020), SOC (Jin et al., 2019), Archipelago (Tsang et al., 2020), Shapley interaction index (Grabisch, 1997) and Bivariate Shapley value (Masoomi et al., 2021). More implementation details of the baselines and ASIV could be found in Appendix A.

heading: 4.3 Evaluation
text: For asymmetric interaction between features, the important features are expected to have more positive incoming edges and less outgoing edges. Therefore to evaluate faithfulness of feature interaction attribution methods in deriving feature interaction relationship, in this paper we focus on pairwise interaction and apply PageRank (Page et al., 1999) algorithm to obtain feature importance ranking. Two evaluation metrics (Chen et al., 2020; Nguyen, 2018; Shrikumar et al., 2017) are employed to evaluate influential features as follows:
The area over the perturbation curve (AOPC): average change in the model output probability on the predicted class among the test data by deleting top k influential words from each text sequence.
AOPC(k) = 1
N
N∑
i=1
f(xi)− f(x̃i), (17)
where x̃i is obtained by dropping the k% topscored words from xi. The higher AOPC, the more important deleted words for model prediction.
Log-odds (LOR): average the difference of negative logarithmic probabilities on the predicted class over the test data before and after masking the top k% influential words from the text sequence.
LOR(k) = 1
N
N∑
i=1
log f(x′i) f(xi) , (18)
where x′i is obtained by masking the k% top-scored words from xi. The lower LOR, the more important deleted words for model prediction.

heading: 4.4 Qualitative Analysis
text: We first demonstrate interpretability of the feature asymmetric interaction graph by presenting a test example “you might not buy the ideas” from SST2 that is predicted as Negative by BERT model. More examples are shown in Appendix C.
Fig. 3 focuses on two words “not” and “might” that are more consistent with human explanation and describes three feature interaction graphs estimated by the typical feature interaction attribution methods. Bivariate Shapley value also estimates asymmetric interaction relationship and Shapley interaction index models symmetric feature interaction. Here ASIV is estimated with random sampling as it performs best in SST-2 dataset. In the directed weighted interaction graph, as shown in
Fig. 3 (a) and (b), word1 0.05→ word2 denotes that in the presence of word1, the influence score of word2 to model prediction is 0.05. Fig. 3 (c) shows an undirected weighted interaction graph where the edge weight is the symmetric interaction influence to model prediction.
For Negative prediction, we take two pairs [“not”, “might”] and [“not”, “buy”] as examples.
In ASIV estimation, the interaction influence of “might” → “not” is positively stronger than that of “not” → “might”, while in Bivariate Shapley and Shapley interaction estimation, both asymmetric and symmetric interaction influences are negative. Intuitively, the interaction between “not” and “might” could contribute positively to Negative prediction. Compared with “not”, “might” does not convey much information against Negative. Therefore ϕnot(might) < ϕmight(not). Similarly, the interaction influence between the pair [“not”, “buy”] is positive in ASIV while in Bivariate Shapley and Shapley interaction estimation the interaction influence is negative. In practice, human evaluation tends to attribute more importance to the pair [“not”, “buy”] for model prediction. “not” further modifies “buy” so in ASIV the interaction influence of “not” → “buy” could be positive and stronger than that of “buy” → “not”.

heading: 4.5 Quantitative Analysis
text: We follow the prior works (Chen et al., 2020; Guerreiro and Martins, 2021) that set k to 20 in sentiment classification task. The sampling size m is set to 500. Due to increasing computational cost in computing each pairwise interaction (statistics of the datasets in Appendix B), we randomly choose 1000 samples from SST-2 dataset and 100 samples from Yelp-2 dataset that the review length is restricted to be less than 100 words. Further, to estimate conditional expectation and in-domain expectation, in each permutation, for each token in
x′ S̄
only the most likely word is sampled from the pretrained language model given xS .

heading: 4.5.1 Comparison with baselines
text: Table 1 shows the evaluation performance of BERT and RoBERTa models on two different datasets. We observe that the proposed ASIV consistently outperforms the compared baselines in identifying influential features for the predictions of BERT and RoBERTa. ASIV with random sampling strategy performs better in SST-2 while ASIV with in-domain sampling demonstrates its effectiveness for Yelp-2 classification with RoBERTa.
We first analyze the baseline as follows: for the univariate feature attribution methods (i.e. Hedge and SOC), the explanation heavily depends on their attribution estimation. For example, compared with Shapley values, SOC could assign different values to the equivalent important features, therefore the corresponding estimated feature ranking may not be faithful. For interaction attribution methods, derived undirected interaction graph (i.e. from Archipelago and Shapley interaction index) does not emphasize the importance of some nodes in symmetric interaction. As shown in Fig. 3 (c), the importance of “not” is ignored in the interaction with “might”. Bivariate Shapley defines a directed interaction graph but its formulation introduces noisy interaction relationship, which may result in false estimation. For example, both the directional interaction between “not” and “might” are negative in Fig. 3 (b). So with PageRank algo-
rithm the obtained node importance ranking could be misleading.
For ASIV evaluated in SST-2 dataset, conditional expectation based estimation performs better than in-domain expectation and marginal expectation with padding operation while it is inferior to marginal expectation with random sampling. Since the review length in SST-2 is relatively short (see Appendix B), compared with conditional sampling, random sampling also can generate smooth and indomain-like text sequence. For example, given the test instance in SST-2 “the cast is uniformly excellent and relaxed” and the corresponding xS “the [MASK] is uniformly excellent [MASK] [MASK] ”, conditional sampling generates the most likely sequence “the interior is uniformly excellent throughout .” and random sampling produces “the movie is uniformly excellent and predictable”.
In Yelp-2 dataset with RoBERTa classification, both ASIV-CE and ASIV-IDE performs better than ASIV with random sampling . Different from SST2 reviews, the review length in Yelp-2 is quite long (see Appendix B). Random sampling is more likely to produce long disorganized x′
S̄ in some
permutations. Compared with in-domain sampling where the quality of pretrained language model is restricted by the limited training corpus, condi-
tional sampling could generate more smooth text sequences that modify the main idea of the review.

heading: 4.5.2 Sensitivity analysis
text: As shown in Fig. 4 and Fig. 5, we study the influence of k on the evaluation performance of BERT and RoBERTa models on SST-2 and Yelp-2 datasets.
We can see from Fig. 4 that ASIV always outperforms the baselines and ASIV with random sampling consistently achieves the best performance in AOPC and LOR metrics by varying k. With the increase of k, the curves of ASIV-IDE and ASIV-CE tend to overlap in BERT-based classification and ASIV-CE outperforms ASIV-IDE by a narrow margin in RoBERTa-based classification. In Yelp-2 dataset with the increase of k generally both conditional and in-domain sampling strategies are more effective than random and padding operation. ASIV-IDE with BERT model performs better in AOPCs and LORs in the scenario k < 30 and ASIV-CE with RoBERTa model performs better with k < 30.
As addressed in the above subsection that in SST-2 the average number of words of the review is limited, random sampling could enforce the classification model to focus on the specific text span
that is essential for predicting the short text sequence, while the smooth context produced by conditional and in-domain sampling strategies may contain confusing information for prediction. For predicting long text sequence, the classification model has to rely on the whole context to capture the main idea of the text. Therefore in computing marginal contribution both conditional and indomain sampling could be more applicable for long text sequences.

heading: 5 Conclusion
text: In this paper we propose an asymmetric feature interaction attribution method to explain asymmetric higher-order feature interactions in the inference of deep neural NLP models. We extend Shapley value to asymmetric Shapley interaction value and investigate three different sampling strategies in computing marginal contribution of value function in NLP field. By evaluating our proposed model and five model-agnostic feature interaction attribution methods on two sentiment datasets with BERT and RoBERTa, our model achieves the best performance in identifying the influential words for model prediction. Also, for the three different sampling strategies we empirically show that none is generally better than the others or more broadly applicable, which could provide guidelines for the
selection of reference distribution in NLP field. For example, random sampling strategy is effective for the short text sequence and for the long text sequence in-domain sampling could produce more smooth and domain-dependent context. In the future, we consider generating sparse and differential causal structure for explaining model prediction.

heading: 6 Limitations
text: The proposed asymmetric Shapley interaction value could estimate asymmetric feature interaction in explaining the prediction of deep models. There are two major concerns regarding the time complexity: estimation of marginal contribution and construction of hypergraphs. In computing value function we have to consider more permutations to reduce approximation errors. Also, before estimating the contribution of asymmetric interaction, interaction graph with different orders should be constructed. We could resort to effective approximation methods in computing marginal contribution and prior knowledge in building hypergraph.

heading: Acknowledgements
text: Research reported in this publication was supported by the National Natural Science Foundation of China under Grant 62202122. We would like to thank the anonymous reviewers and Dr. Yang Dong
for their valuable comments.
Ethics Statement
The authors declare that they have no conflicts of interest. This paper does not contain any studies involving business data and personal information.

heading: B Statistics of SST-2 and Yelp-2 datasets
text: Here we present the statistics of the test datasets. In Yelp-2 test set the the average review length 136.49, to reduce computational cost in estimating pariwise interaction, we have to sample from the test with restriction to review length (i.e. review length ≤ 100).
C Examples of the estimated feature interaction graph with matrix demonstration
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we consider generating sparse and differential causal structure for explaining model prediction