SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The presented method, ABCD, is susceptible to two sources of error affecting its accuracy. The first is prediction errors of the Transformer neural network, and the second is errors from the constraint-based causal discovery algorithm that uses the covariance matrix calculated in the Transformer. The presence of errors from the second source depends on the first. For a Transformer with perfect generalization accuracy, and when causal Markov and faithfulness assumptions are not violated, the second source of errors vanishes, and a perfect result is returned by the presented ABCD method.
From Assumption 1, the deepest attention layer is expected to produce embeddings for each input symbol, such that the input symbol can be correctly predicted from its corresponding embedding (one-to-one embedding). However, except for trivial cases, generalization accuracy is not perfect. Thus, the attention matrix of an input sequence, for which the Transformer fails to predict the desired outcome, might include errors. As a result, the values of correlation coefficients calculated from the attention matrix might include errors.
Constraint-based algorithms for causal discovery are generally proved to be sound and complete when a perfect CI-test is used. However, if a CI-test returns an erroneous result, the constraint-based algorithm might introduce additional errors. This notion is called stability (35), which is informally measured by the number of output errors as a function of the number of CI-tests errors. In fact, constraint-based algorithm differ from one another in their ability to handle CI-test errors and minimize their effect. Thus, inaccurate correlation coefficients used for CI-testing might lead to erroneous independence relations, which in turn may lead to errors in the output graph.
We expect that as Transformer models become more accurate with larger training datasets, the accuracy of ABCD will increase. In this paper's experiments we use existing pre-trained Transformers and common datasets, and use the ICD algorithm (30) that was shown to have state-of-the-art accuracy.
5 ABCD with Application To Explaining Predictions
One possible application of the proposed ABCD approach is to reason about predictions by generating causal explanations from pre-trained self-attention models such as BERT (9). Specifically, we seek an explaining set that consists of a subset of the input symbols that are claimed to be solely responsible for the prediction. That is, if the effect of the explaining set on the prediction is masked, a different (alternative) prediction is provided by the neural network.
It was previously claimed that attention cannot be used for explanation (16); however, in a contradicting paper (41), it was shown that explainability is task dependent. A common approach is to use the attention matrix to learn about input-output relations for providing explanations (33; 5; 4). These often rely on the assumption that inputs having high attention values with the class token influence the output (44; 6). We claim that this assumption considers only marginal statistical dependence and ignores conditional independence and explaining-away that may arise due to latent confounders.
To this end, we propose CLEANN (CausaL Explanations from Attention in Neural Networks). A description of this algorithm is detailed in Appendix B, and an overview with application to explaining movie recommendation is given in Figure 4. See also Nisimov et al. (23) for more details.
In this section we demonstrate how a causal graph constructed from a self-attention matrix in a Transformer based model can be used to explain which specific symbols in an input sequence are the causes of the Transformer output. We experiment on the tasks of sentiment classification, which classifies an input sequence, and recommendation systems, which generates a candidate list of recommended symbols (top- $k$ ) for the next item. For both experiments in this section we compare our method against two baselines from (38): (1) Pure-Attention algorithm (Pure-Atten.), that uses the attention weights directly in a hill-climbing search to suggest an explaining set, (2) Smart-Attention algorithm (Smart-Attn.), that adapts Pure-Attention, where a symbol is added to the explanation only if it reduces the score gap between the original prediction's score and the second-ranked prediction's score. Implementation tools are in https://github.com/IntelLabs/causality-lab.
6.1 Evaluation metrics
We evaluate the quality of inferred explanations using two metrics. One measures minimality of the explanation, and the other measures how specific is the explanation to the prediction.
6.1.1 Minimal explaining set
An important requirement is having the explaining set minimal in the number of symbols (40). The reason for that is twofold. (1) It is more complicated and less interpretable for humans to grasp the interactions and interplay in a set that contains many explaining symbols. (2) In the spirit of occum's razor, the explaining set should not include symbols that do not contribute to explaining the prediction (when faced with a few possibles explanations, the simpler one is the one most likely to be true). Including redundant symbols in the explaining set might result in a wrong alternative predictions when the effect of this set is masked.
6.1.2 Influence of the explaining set on replacement prediction
The following metric is applicable only to tasks that produce multiple predictions, such as multiple recommendations by a recommender system (e.g., which movie to watch next), as opposed to binary classification (e.g., sentiment) of an input. Given an input sequence consisting of symbols,  $s = \{s_1, \ldots, s_n\}$ , a neural network suggests  $\tilde{s}_{n+1}$ , the  $1^{\text{st}}$  symbol in the top- $k$  candidate list. CLEANN finds the smallest explaining set within  $s$  that influenced the selection of  $\tilde{s}_{n+1}$ . As a consequence, discarding this explaining set from that sequence should prevent that  $\tilde{s}_{n+1}$  from being selected again, and instead a new symbol should be selected in replacement (replacement symbol). Optimally, the explaining set should influence the rank of only that  $1^{\text{st}}$  symbol (should be downgraded), but not the ranks of the other candidates in the top- $k$  list. This requirement implies that the explaining set is unique and important for the isolation and counterfactual explanation of only that  $1^{\text{st}}$  symbol, whereas the other symbols in the original top- $k$  list remain unaffected, for the most part. It is therefore desirable that after discarding the explaining set from the sequence, the new replacement symbol would be one of the original (i.e. before discarding the explaining set) top- $k$  ranked symbols, optimally the  $2^{\text{nd}}$ .
6.2 CLEANN for Sentiment classification
We exemplify the explainability provided by our method for the task of sentiment classification of movie reviews from IMDB (20) using a pre-trained BERT model (9) that was fine-tuned for the task. The goal is to propose an explanation to the classification of a review by finding the smallest explaining set of word tokens (symbols) within the review. Figure 2 exemplifies the explanation of an input review that was classified as negative by the sentiment classification model. We compare between our method and two other baselines on finding a minimal explaining set for the classification. We see that both baselines found the word 'bad' as a plausible explanation for the negative sentiment classification, however in addition, each of them finds an additional explaining word ('pizza', 'cinema') that clearly has no influence on the sentiment in the context it appears in the review. Contrary to that, our method finds the words 'bad' and 'but' as explaining the negative-review classification. Indeed, the word 'but' may serve as a plausible explanation for the negative review at the  $2^{\mathrm{nd}}$  part of the sentence, since it negates the positive  $1^{\mathrm{st}}$  part of the sentence (about the pizza). Additionally, Figure 2(b) shows
Algorithm	Review Length
20	30	40	60
Pure-Attention	5.52 (4.34)	8.88 (7.59)	12.71 (10.30)	13.14 (12.47)
Smart-Attention	3.62 (2.45)	4.63 (3.11)	6.03 (3.34)	6.41 (4.74)
CLEANN	2.82 (1.71)	3.48 (2.18)	3.58 (1.83)	3.53 (1.54)
Table 1: Mean and standard deviation (in parenthesis) of explanation set sizes for different review lengths (number of tokens in the input sequence). Each column is calculated using 25,000 reviews. CLEANN provides explaining sets having the smallest size on average, with statistical significance tested using the Wilcoxon signed-ranks test at significance level  $\alpha = 0.01$  (indicated in bold).
the corresponding attention map at the last attention layer of the model for the given input review. In addition, in Figure 2(c) we present the corresponding matrix representation of the learned graph (PAG). We can see that despite high attention values of the class token [cls] with some of the word tokens in the review (in the first row of the attention map), some of these words are found not to be directly connected to the class token in the causal graph produced by our method, and therefore may not be associated as influencing the outcome of the model.
Table 1 and Figure 3 show how the length of a review influences the explaining set size. CLEANN produces the smallest explaining sets on average, where statistical significance was tested using Wilcoxon signed-ranks test at significance level  $\alpha = 0.01$ . For an increasing review length it is evident that the two baselines increase their explaining set, correspondingly. Even if the additional explaining tokens are of negative/positive context, they are not the most essential and prominent word token of this kind. Contrary to that, the explaining set size produced by our method is relatively stable, with a moderate increase in the explaining set size, meaning that it keeps finding the most influential words in a sentence that dominates the decision put forward by the model.
Figure 2: Comparison between the explaining sets produced by the methods given a negative review (a) Notice the implausible explaining tokens 'cinema' and 'pizza' found by the baseline methods. In contrast, in addition to the token 'bad', CLEANN found the token 'but', which is a plausible explanation as it is a negation to positive part of the sentence that precedes it. (b) Attention matrix produced for the review by the model. (c) A matrix representation of the learned graph (PAG) produced by CLEANN (green: arrow head, yellow: arrow tail, blue: circle mark).
(a)
Figure 3: Comparison between CLEANN and the baseline methods on the distribution of explaining set size over the various reviews. The reviews are sorted by their corresponding explanation size on the horizontal axis. CLEANN consistently finds smaller sets for: (a) review length  $= 20$ , (b) review length  $= 30$ , (c) review length  $= 40$ , (d) review length  $= 60$ .
Figure 4: An overview of the presented approach. As an example,  $\mathcal{M}$  is a pre-trained BERT4Rec recommender (36). (a) Given a set of  $n$  movies (symbols)  $[s_1,\dots ,s_n]$  the user interacted with, and the  $(n + 1)^{th}$  movie masked, the recommender  $\mathcal{M}$  recommends  $\tilde{s}_{n + 1}$ . Next, in the abductive inference stage (b) the recommendation is treated as input in addition to the observed user-movie interactions, that is, input:  $[s_1,\ldots ,s_n,\tilde{s}_{n + 1}]$ , and the attention matrix is extracted. (c) A causal structure learning stage in which the attention matrix is employed for testing conditional independence in a constraint-based structure learning algorithm. The result is a Markov equivalence class. Here, as an example, a causal DAG (a member of the equivalence class) is given, where  $H_{1}$  and  $H_{2}$  are latent confounders. (d) A causal explanation reasoned from the causal graph.
6.3 CLEAN for recommendation system
We assume that the human decision process, for selecting which items to interact with, consists of multiple decision pathways that may diverge and merge over time. Moreover, they may be influenced by latent confounders along this process. Formally, we assume that the decision process can be modeled by a causal DAG consisting of observed and latent variables. Here, the observed variables are user-item interactions  $\{s_1,\dots ,s_n\}$  in a session  $S$  , and latent variables  $\{H_{1},H_{2},\ldots \}$  represent unmeasured influences on the user's decision to interact with a specific item. Examples for such unmeasured influences are user intent and previous recommendation slates presented to the user.
We exemplify the explainability provided by our method for the task of a recommendation system, and suggest it as a means for understanding of the complex human decision-making process when interacting with the recommender system. Using an imaginary session that includes the recommendation, we extract an explanation set from the causal graph. To validate this set, we remove it from the original session and feed the modified session into the recommender, resulting in an alternative recommendation that, in turn, can also be explained in a similar manner. Figure 4 provides an overview of our approach.
For empirical evaluation, we use the BERT4Rec recommender (36), pre-trained on the MovieLens 1M dataset (15) and estimate several measures to evaluate the quality of reasoned explanations.
Minimal explaining set: Figure 5(a) compares the explaining set size for the various sessions produced by CLEANN and the baseline methods. It is evident that the set sizes found by CLEANN are smaller. Figure 5(b) shows the difference between the explaining set sizes found by the baseline method and CLEANN, as calculated for each session, individually. Approximately  $25\%$  of the sessions are with positive values, indicating smaller set sizes for CLEANN, zero values show equality between the two, and only  $10\%$  of the sessions are with negative values, indicating smaller set sizes for Pure-Attention.
Influence of the explaining set on replacement recommendation: Figure 6(a) compares the distribution for positions of replacement recommendations, produced by CLEANN and the baseline methods, in the original top-5 recommendations list. It is evident that compared to the baseline methods, CLEANN recommends replacements that were ranked higher (lower position) in the original top-5 recommendations list. In a different view, Figure 6(b) shows the relative gain in the number of sessions for each position, achieved by CLEANN compared to the baseline methods. There is a trend line indicating higher gains for CLEANN at lower positions. That is, the replacements are more aligned with the original top-5 recommendations. CLEANN is able to isolate a minimal explaining set that influences mainly the  $1^{\text{st}}$  item from the original recommendations list.
Figure 5: Explaining set size. (a) Comparison between CLEANN and the baseline methods on the explaining set size (sorted by size along the horizontal axis) for the various sessions. Set sizes found by CLEANN are smaller. (b) The difference between set sizes found by the baseline method and CLEANN, presented for each individual session. Positive values are in favor of CLEANN.
Figure 6: Positions of the replacement recommendations within the original top-5 recommendations list. (a) Position distribution of the replacements: compared to the baseline methods, CLEANN recommends replacements which are ranked higher in the original top-5 recommendations. (b) The relative gain in the number of sessions for each position, achieved by CLEANN. There is a trend line indicating higher gains for CLEANN at lower positions (replacements are more aligned with the original top-5 recommendation).
We presented a relation between the self-attention mechanism used in the Transformer neural architecture and the structural causal model often used to describe the data-generating mechanism.
One result from this relation is that, under certain assumptions, a causal graph can be learned for a single input sequence (ABCD). This can be viewed as utilizing pre-trained models for zero-shot causal-discovery. An interesting insight is that the only source of errors while learning the causal structure is in the estimation of the attention matrix. Estimation is learned during pre-training the Transformer. Since in recent years it was shown that Transformer models scale well with model and training data sizes, we expect the estimation of the attention matrix to be more accurate as the pre-training data increases, thereby improving the accuracy of causal discovery.
Another result is that the causal structure learned for an input sequence can be employed to reason about the prediction for that sequence by providing causal explanations (CLEANN). We expect learned causal graphs to be able to answer a myriad of causal queries (24), among these are personalized queries that can allow a richer set of predictions. For example, in recommender systems, assuming that the human decision process consists of multiple pathways that merge and split, by identifying independent causal pathways, recommendations can be provided for each one, independently.
The results of this paper contribute to the fields of causal inference and representation learning by providing a relation that bridges concepts from these two domains. For example, they may lead to 1) causal reasoning in applications for which large-scale pre-trained models or unlabeled data exist, and 2) architectural modifications to the Transformer to alleviate causal inference.
In this section we provide Table 2, which describes the main symbols used in the paper.
Table 2: Notations used in the paper. The first set of symbols describes entities in the structural causal model, and the second set of symbols describes entities in the Transformer neural network.
Symbol	Description
Xi	a random variable representing node i in an SCM
Ui	latent exogenous random variable i in an SCM
G	weighted adjacency matrix of an SCM
G	causal graph (unweighted, directed-graph structure)
Zi	output embedding of input symbol i, Zi ≡ Z(i, ·), in attention layer
Vi	value vector corresponding to input i, Vi ≡ V(i, ·), in attention layer
A	attention matrix
M	Transformer neural network
WV, WQK	learnable weight matrices in the Transformer neural network
B Finding Possible Explanations from a Causal Graph
This appendix includes the pseudo-code for the CLEANN (CausaL Explanations from Attention in Neural Networks) algorithm and explanatory figures. By abduction, we replace the masked symbol with a predicted one, creating an imaginary sequence. We then feed-forward this sequence and extract the resulting attention matrix (Algorithm 1-lines 2-4). Using this attention matrix we learn a causal graph as discussed in Section 3.2 (Algorithm 1-lines 5-7).
Given a causal graph, various "why" questions can be answered (26). We follow (38) explaining a prediction using the symbols from the input sequence. That is, provide the minimal set of symbols that led to a specific prediction and provide an alternative prediction. To this end, we define the following.
Definition 1 (PI-path). A potential influence path from  $A$  to  $Z$  in PAG  $\mathcal{G}$ , is a path  $\Pi(A, Z) = \langle A, \ldots, Z \rangle$ , such that for every sub-path  $\langle U, V, W \rangle$  of  $\Pi(A, Z)$ , where there are arrow heads into  $V$  and there is no edge between  $U$  and  $W$  in  $\mathcal{G}$ .
Essentially, a PI-path ensures dependence between its two end points when conditioned on every node on the path (a single edge is a PI-path).
Definition 2 (PI-tree). A potential influence tree for  $A$  given causal PAG  $\mathcal{G}$ , is a tree  $\mathcal{T}$  rooted at  $A$ , such that there is a path from  $A$  to  $Z$  in  $\mathcal{T}$ ,  $\langle A, V_1, \ldots, V_k, Z \rangle$ , if and only if there is an PI-path  $\langle A, V_1, \ldots, V_k, Z \rangle$  in  $\mathcal{G}$ .
Definition 3 (PI-set). A set  $E$  is a potentially influencing set (PI-set) on item  $A$ , with respect to  $r$ , if and only if:
1.  $|\pmb {E}| = r$
2.  $\forall E\in E$  there exists a PI-path,  $\Pi (A,E)$  such that  $\forall V\in \Pi (A,E),V\in E$
3.  $\forall E\in E,E$  temporally precedes  $A$
An example for identifying PI-sets is given in Figure 7. Although a PI-set identify nodes that are conditionally dependent on the prediction, it may not be a minimal explanation for the specific prediction. That is, omitting certain symbols from the session alters the graph and may render other items independent of the prediction. Hence, we provide an iterative procedure ("FindExplanation" in Algorithm 1-lines 10-18), motivated by the ICD algorithm (30), to create multiple candidate explanations by gradually increasing the value of  $r$ . See an example in Figure 8. Note that from conditions (1) and (2) of Definition 3,  $r$  effectively represents the maximal search radius from the prediction (in the extreme case, the items of the PI-set lie on one PI-path starting at the prediction). The search terminates as soon as a set qualifies as an explanation (Algorithm 1-line 17). That is, as soon as a different prediction is provided. If no explanation is found (Algorithm 1-line 19) a higher value of  $\alpha$  should be considered for the algorithm.
Algorithm 1: CLEANN: CausaL Explanations from Attention in Neural Networks
Input:  $s$  : a sequence  $s = [s_1,\dots ,s_n,\langle \text{mask}\rangle ]$ $\mathcal{M}$  : a pre-trained attention-based model   
 $\tilde{s}_{n + 1}$  : prediction by  $\mathcal{M}$ $\alpha$  : significance level for CI-testing   
Output:   
 $E$  : an explanation set for the prediction   
 $\tilde{s}_{n + 1}^{\prime}$  : an alternative prediction (after masking the effect of the explanation)   
 $\mathcal{G}$  : a causal graph of  $[s_1,\ldots ,s_n,\tilde{s}_{n + 1}]$    
Function Main  $(s,\mathcal{M},\tilde{s}_{n + 1})$  ..  $\tilde{s}\gets [s_1,\dots ,s_n,\tilde{s}_{n + 1}]$  replace mask with prediction forward pass:  $\mathcal{M}(\tilde{s})$  get  $\tilde{\mathbf{A}}$  , the attention matrix in the last layer of  $\mathcal{M}(\tilde{\mathbf{S}})$  define correlation matrix  $\rho_{\tilde{S}}:\rho_{\tilde{S}}(i,j) = \mathbf{C}(i,j) / \sqrt{\mathbf{C}(i,i)\mathbf{C}(j,j)}$  where  $\mathbf{C} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}^{\top}$  define Ind  $(\rho_{\tilde{S}})$  : a conditional independence test based on partial correlation  $\mathcal{G}\gets$  CausalDiscovery (Ind  $(\rho_{\tilde{S}}),\alpha)$ $E,\tilde{s}_{n + 1}^{\prime}\gets$  FindExplanation  $(\mathcal{G},\tilde{s}_{n + 1},\mathcal{M},s)$  return  $E,\tilde{s}_{n + 1}^{\prime},\mathcal{G}$    
Function FindExplanation  $(\mathcal{G},\tilde{s}_{n + 1},\mathcal{M},s)$  .. create  $\mathcal{T}$  : a PI-tree for  $\tilde{s}_{n + 1}$  given  $\mathcal{G}$  for  $r$  in  $\{1,\dots ,n - 1\}$  do create  $\mathcal{E}$  , subsets of Nodes(T)  $\backslash$ $\tilde{s}_{n + 1}$  such that  $\forall E\in \mathcal{E}$ $E$  is a PI-set on  $\tilde{s}_{n + 1}$  in  $\mathcal{G}$  with respect to  $r$  for  $E\in \mathcal{E}$  do  $s^{\prime}\gets s\setminus E$ $\tilde{s}_{n + 1}^{\prime}\gets \mathcal{M}(s^{\prime})$  get an alternative prediction if  $\tilde{s}_{n + 1}^{\prime}\neq \tilde{s}_{n + 1}$  then return  $E,\tilde{s}_{n + 1}^{\prime}$  return immediately ensuring smallest explanation set return  $\emptyset ,\emptyset$  no explanation found at the current significance level
In this section we provide proofs for the results described in Section 4.2. Let  $(\Omega_n, \mathcal{F}_n, P_n)$  be an SCM-consistent probability space, and  $\mathcal{M}$  trained on sequences from this probability space, complying with Assumption 1.