Causal Interpretation of Self-Attention in Pre-Trained Transformers
Authors: Raanan Yehezkel Rohekar
Yaniv Gurwicz
Shami Nisimov
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-Attention
Causal Discovery
Reasoning
Explainability
Zero-shot
Transformer
TL;DR: We propose a causal interpretation of self-attention in the Transformer neural network architecture, and zero-shot causal-discovery utilizing existing pre-trained Transformers.

Abstract: We propose a causal interpretation of self-attention in the Transformer neural network architecture. We interpret self-attention as a mechanism that estimates a structural equation model for a given input sequence of symbols (tokens). The structural equation model can be interpreted, in turn, as a causal structure over the input symbols under the specific context of the input sequence. Importantly, this interpretation remains valid in the presence of latent confounders. Following this interpretation, we estimate conditional independence relations between input symbols by calculating partial correlations between their corresponding representations in the deepest attention layer. This enables learning the causal structure over an input sequence using existing constraint-based algorithms. In this sense, existing pre-trained Transformers can be utilized for zero-shot causal-discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.

Causal Interpretation of Self-Attention in Pre-Trained Transformers
Raanan Y. Rohekar
Yaniv Gurwicz
Shami Nisimov Intel Labs
Intel Labs
Intel Labs yaniv. wwiiv.gurwicziineel.cm com
com
shami www.iiisiviiieee..ommm
Abstract
We propose causal interpretation of self-attention in the Transformer neural network architecture. We interpret self-attention as a mechanism that estimates structural equation model for a given input sequence of symbols (tokens). The structural equation model can be interpreted, in turn, the input symbols under the specific context of the input sequence. Importantly, this interpretation remains valid in the presence of latent confounders. Following this interpretation, we estimate conditional independence relations between input symbols by calculating partial correlations between their corresponding represen- tations in the deepest attention layer. This enables learning the causal structure over an input sequence using existing constraint- existing pre-trained Transformers can be utilized for zero-shot causal-discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.
causal structure over
algorithms. In this sense,
Introduction
Causality plays an important role in many sciences such as epidemiology. social sciences, and finance (25 34).. Understanding the underlying causal mechanisms is crucial for tasks such explaining phenomenon, predicting, and decision making. An automated discovery of causal structures from
observed data alone is important problem in artificial intelligence. It particularity challenging when latent confounders may exist, One family of algorithms (35) 7 46 30), called constraint- based, recovers in the large sample limit an equivalence class the true underlying graph. However, they require a statistical test to be provided. The statistical test is often sensitive to small sample size, and in some cases is not clear which statistical method assume that data samples are generated from a single causal graph, and are designed to learn single equivalence class. There are cases where data samples may generated by different causal mechanisms and it is not clear how to learn the correct causal graph for each data sample. For example, decision process may different among humans. Data collected about their actions may not be represented by a single causal graph. This is the case for recommender systems that are required to provide personalized recommendation for a user, based on her past actions. addition, it is desirable that such automated systems will provide tangible explanation to why the specific recommendation was given.
is suitable for the data. Moreover, these
Recently, deep neural networks, based on the Transformer architecture 39 have been shown to achieve state-of-the- accuracy in domains such as natural language processing
45 vision (I0) 19), and recommender systems (36) [8). The Transformer is based on the attention mechanism, where it calculates context-dependent weights for a given input (32). Given an input consisting of sequence of symbols (tokens), the Transformer is able to capture complex dependencies, but it is unclear how they are represented in the Transformer nor how to extract them.
* All authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this paper we bridge between structural causal models and attention mechanism used in the Transformer architecture. We show that self-attention a mechanism that estimates a linear structural equation model in the deepest layer for each input sequence independently, and show that it represents a causal structure over the symbols in the input sequence. In addition, we show that an equivalence class of the causal structure over the input can be learned solely from the Transformer's is estimated attention matrix. This enables learning the causal structure over a single input sequence, using existing constraint-based algorithms, and utilizing existing pre-trained Transformer models for zero-shot causal discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.
2 Related Work
In recent years, several advances in causal reasoning using deep neural networks were presented. One line of work is causal modeling with neural networks (2). Here, neural network architecture follows a given causal graph structure. This neural network architecture constructed to model the joint distribution over observed variables (12) to explicitly learn the deterministic functions of a structural causal model (42). It was recently shown that a neural network architecture that explicitly learns the deterministic functions of an SCM can be used for answering interventional and counterfactual queries (43). That is, inference in rungs 2 and 3 in the ladder of causation (26), also called layers 2 and in the Pearl causal hierarchy (2). Nevertheless, this architecture requires knowing the causal graph structure before training, and supports a single causal graph.
In another line of work, an attention mechanism is added before training, and causal graph is inferred from the attention values at inference (7). Attention values are compared against a threshold, and high values are treated indication for potential causal relation. Nauta et al. (2D validate these potential causal relations by permuting the values of the potential cause and measuring the effect. However, this method is suitable only if a single causal model underlay the dataset.
3 Preliminaries
First, we provide preliminaries for structural causal models and the attention mechanism in the Transformer-basee neural architecture. Throughout the paper we denote vectors and sets with bold- italic upper-case (e.g.. V), matrices with bold upper-case (e.g., A), random variables with italic upper-case (e.g., X), and models in calligraphic font (e.g., MM. Different sets of letters used when referring to structural causal models and attention (see Appendix A-Table for a list).
3.1 Structural Causal Models
A structural causal model (SCM) represents causal mechanisms in a domain (24 35 28). An SCM is a tuple (U, F,F, P(U)), where U = .U,, }m is a of latent exogenous random variables, X (XI,..,,x, } is a set of endogenous random variables, F = {ff..... fo. is a set of deterministic functions describing the values X given their direct causes, and P(U) the distribution over U. Each endogenous variable X, has exactly one unique exogenous cause U2 (m = n). The value of an endogenous variable X,, Vi E [I....,nii is
X, f((,,,,
(the left-arrow indicates an assignment due to cause-effect relation), where Pa; is the set of direct causes (parents in the causal graph) of X,. A graph corresponding to the SCM consists of node for each variable, and directed edge for each direct mouse-and-effee relation, as evident from F. If the graph is directed and acyclic (DAG) then the SCM is called recursive.
In a special subclass of SCMs, referred in this paper combination of its parents, and exogenous nodes are independently and normally distributed additive noise. Let G be weight matrix, where (i is the weight of parent node X, determining the child node X, and G(i, if X₁ is not a parent of X,, and A be diagonal matrix, where A is the weight of exogenous node U.. Then, Vi E .n]]
SCM, a node value is a linear
X + G(i, X + Aii,
(2)

---

and in matrix form = (ancestors before their descendants). the weight matrix G is strictly lower triangular (zeros on the diagonal). In this paper we learn the causal graph, described by the non-7ero elements of G. using constraint-based causal discovery approach, which generally requires assuming the following.
= G. + AU. In the case of a recursive SCM, with nodes sorted topologically
Definition 1 (Causal Markov). In causally Markov graph, variable is independent of all other variables, except its effects, conditional on all its direct causes.
Definition 2 (Faithfulness). A distribution is faithful to graph if and only if every independence relation true in the distribution is entailed by the graph.
3.2 Self-Attention Mechanism
Attention is mechanism that predicts context dependent weights (32) for the input sequence. In this
paper we analyse the attention mechanism used by Vaswani et al. (39 Here, the input is a matrix Y where row vector (i, is the embedding of symbol 8; in an input sequence
[81,.... sn]. The attention matrix is calculated by A softmar YW QK Y ".) such that the rows of A sum to ! Values matrix is computed by V = YWv. v. where row V(i, is the value vector of input symbol 8;- We treat V as set of projection of Y column vectors Wv. Finally, the output embeddings are = AV, where the i-th output embedding Z; is the i-th row of Z.
4
A Link between Pre-trained Ateettion and a Causal Graph underlying an Input Sequence of Symbols
In this section we first describe self-attention as a mechanism that encodes correlations between symbols in an unknown structural causal model (Section 4.1). After establishing this relation, present a method for recovering an equivalence class of the causal graph underlying the input sequence (Section 4.2 We also extend the results to the multi-head and multi-layer architecture (Section 4.3)
4.1
Self-Attention as Correlations between Nodes in Structural Causal Model
We now describe how self-attention can be viewed as a mechanism that estimates the values of observed nodes of an SCM. We first show that the covariance over the outputs of self-attention is similar to the covariance over observed nodes of an SCM. Specifically, we model relations between symbols in input sequence using SCM, the values over endogenous nodes, in matrix form, are X = GX + AU, which means
SCM at the output of the attention layer. In an
X = (I - AU-
Since is a strictly lower trianuura weight matrix, - GI- is lower unitriangular matrix, which equal to the sum of geometric series
(I -- --G)--- c
(4)
k=
Thus, the (i,j) element represents the sum of directed-pan -pathe weights, for all directed paths from Xj X, having length up
The weight of directed path is the product of the weights of edges along the path. In case some of the nodes are latent confounders, then the matrix in Equation 3 may not be triangular, due to spurious associations. Equation 3aattionnrerreeee rrepresents system with input U, output X and weights (I G)- A. The covariance matrix of the output is
Cx = [ (x) 1
[I -)- (U HU)
(5)
-II-))- (U) (D) I(I-
= (1--))--ACC((--G))))
where Px = -I App. and Cu is the covariance matrix of exogenous variables U.
G)
In the paper by Vaswani et al. (39) the weight matrix is Wok = W where the weight matrices We Q and Wxr are learned explicitly and dx the number of columns in these matrices.
3

---

An attention layer in a Transformer architecture, estimates an attention matrix A and a values matrix V from embeddings Y of an input sequence symbols s ........ sn]". The output embeddings is calculated by Z AV. It is important to note that during self-attention training, an inductive bias used in the form of learning separate representations for the attention matrix and values matrix V. It can be seen that the j-th element of the i-th row-vector, Vii,j) is the projection of the symbol 8i input embedding (i, on column vector (v In other words, an embedding is projected on the direction of vector Wv(.,)). Thus, each column of V can be seen as projection of the input embeddings. Moreover, the attention matrix is shared with all projections for calculating the output embeddings. That is, for the j-th projection the output embedding is
((,))) AV(.,)).
(6)
Denote 1 ((-5) and V, = V(+.)) corresponding to projection j. Covariance of Zj is
Cz, Bi(Z; (Z; 1 1
= AB[((V; 1v, ,V, =AV,,A,,
where Cv is the covariance matrix of input embeddings projected on vector Wr..)). Since columns of V describe different projections corresponding to independent columns of W and from the central limit theorem it follows that Cv + I, which leads to
Cz AAT.
(8)
How does attention matrix A and a values vector V, translate into structural causal model? We interpret the attention matrix A and values vector V as components of an SCM, such that the covariance matrix of its endogenous variables is equal to the covariance matrix of the attention layer's output, Cx = Cz. From Equation 5) and Equation
AAT = - (CC
The attention matrix A measures the total effect each variable has on another variable, for any projection V(-,j) of the input embeddings. Similarly, matrix (I - G)- A in an SCM measures the effect one variable has on another variable through all directed paths in the causal graph (Equation 4 for any value of the exogenous variables U. Thus. form Equation p matrix A is analogous to (I-G)-1a 'A and V(..j) .) value assignment for U, which serves context. Both A and (I-G) are not required to be triangular as latent confounders may introduce spurious associations, such that variables might be required to estimate the values of variables earlier in the topological order. In
addition, the symbols in the sequence S are not assumed to be topologically ordered.
4.2 Attention-based Causal-Diseovery (ABCD)
In Section 4.1 we show that self-attention learns to represent pairwise associations between symbols of an input sequence, for which there exists a linear-Gaussin SCM that has the exact same pair-wise associations between its nodes. However, can we recover the causal structure G of the SCM solely from the weights of a pre-trained attention layer? We now present the Attention-Bant Based Causal- Discovery (ABCD) method for learning an equivalence class of the causal graph that underlies given input sequence.
Causal discovery (causal structure learning) from observed data alone requires placing certain assumptions. Here we assume the causal Markov (24) and faithfulness (35) assumptions. Under these assumptions, constraint- baaseed methods use tests of conditional independence (CI-tests) to learn the causal structure (27) (35 & 77, 300 31 (2)). A statistical Cl-test is used for deciding if two variables are statistically independent conditioned on a set of variables. Commonly, partial correlation is used for Cl-testing between continuous, normally distributed variables with linear relations. This test requires only a pair-wise correlation matrix (marginal dependence) for evaluating partial correlations (conditional dependence). We evaluate the correlation matrix from the attention matrix. From Equation S the covariance matrix output embeddings is Cz = AAT T and (pair- wise) correlation coefficients are Pi. = Cz(i,j)/ Cz(i,
Unlike bernel-based CI tests 37, 48), do not need to explicitly define estimate the kernel, as it is readily
[ 13
[4

---

available by a single forward-pass -psss of the input sequence in the Transformer. This implies the following. Firstly, our Cl-testing function is inherently learned during the training stage of a Transformer, by that enjoying the efficiency in learning complex models from large datasets. Secondly, since attention computed for each input sequence uniquely, Cl-testing is unique to that specific sequence. That is, conditional independence is tested under the specific context of the input sequence.
Finally, the learned causal graph represents an equivalence class in the form of a partial ancestral graph (PAG) (29; 47), which can also encode the presence of latent confounders. A PAG represents a set of causal graphs that cannot be refuted given the data. There are three types of edge-marks (at some node X): an arrow-head '-> X", a tail '-X', and circle --o X' which represent an edge-mark that cannot be determined given the data. Note that reasoning from a PAG is consistent with every member in the equivalence class it represents.
What is the relation between the equivalence class learned by ABCD and the causal graph underlying the input sequence $ = [st]....,._]?
Definition 3 (SCM-consistent probability-space) Let S2n = ([si........
8t E S} be a sample space, where S is countable set of symbols. Let Fn be an event space, and Pn be a probability measure. For finite N, the probability space ,n, Pr)) is called SCM-consistent if every sequence s E Dr, was generated by corresponding SCM with graph G..
Assumption 1. Attention mechanism M trained on sequences sampled from an SCM- probability space, such that each input symbol is uniquely represented by its output embedding.
Many training methods, such as MLM (,, satisfy Assumption I as they train the network to predict input symbols using their corresponding output embeddings in the deepest layer. This assumption is required for relating the causal graph learned from the deepest attention layer to the causal graph underlying the input sequence.
Theorem 1. If 8 is sequence from an SCM-consistent probability space used to train M, Gn is the causal graph of the SCM from which was generated, and A the atlention matrix for impul sequence s in the deepest layer, then Go is in the equivalence class learned by a sound meastrrintt-basee algorithu that uses covariance matrix Cz AA for partial correlation Cl-testing.
Corollary 1. Nodes X; and X, are d-separated in 9z conditioned on X' C X if and only symbols 8; and 8j are d-separated in Go conditioned on s', a subset containing symbols corresponding to X'.
Corollary is useful for cases in which the graph over the input symbols G.. as well as the structural equation model described by Gz, does not have a causal meaning. In these cases, one can reason about conditional independence relations among input symbols.
4.3 Multiple Attention Layers and Multi-Head Attention as Structural Causal Model
Commonly, the encoder module in the Transformer architecture (39) consists of multiple self attention layers executed sequentially. A non-linear transformation is applied to the the output embeddings of one attention layer before feeding it input to the next attention layer. The non-linear transformation applied to each symbol independently. In addition each attention layer consists of multiple self- attention heads, processing the input in parallel using head-specific attention matrix. The output embedding of each symbol is then linearly combined along the heads. It is important to note that embedding of each symbol is processed independently of embeddings of other symbols. The only part in which an embedding of one symbol is influenced by embeddings of other symbol is the multiplication by the attentnion matrix.
Thus, a mayte-layer, multi-head, architecture can be viewed as a deep graphical model, where exogenous nodes of an SCM are estimated by a previous attention layer. The different heads in an attention layer learn different graphs for the same input, where their output embeddings are linearly combined. This can be viewed as a mixture model. We recover the causal graph from the last (deepest) attention layer, and treat earlier layers as context estimation, which is encoded in the values of the exogenous nodes (Figure Tp.
5

---

deep context representation
encoded context
causal graph
layer
lyer
X1
X2
X3
X4
transformation type: Unear
nonlinear
linear
nonlinear
linear
Figure I: A network with three attention layers. Early attention layers (0 and 1) estimate context, which is encoded in the values of exogenous nodes for the last layer (2).
4.4 Limitations of ABCD
The presented method, ABCD, is susceptible to two sources of error affecting its accuracy. The first prediction errors of the Transformer neural network, and the second is errors from the constraint- based causal discovery algorithm that uses the covariance matrix calculated in the Transformer. The presence errors from the second source depends on the first. For a Transformer with perfect generalization accuracy, and when causal Markov and faithfulness assumptions are not violated, the second source of errors vanishes, and a perfect result is returned by the presented ABCD method.
From Assumption I the deepest attention layer is expected to produce embeddings for each input symbol, such that the input symbol can be correctly predicted from its corresponding embedding (one-to-one embedding). However, except for trivial cases, generalization accuracy is not perfect. Thus, the attention matrix of an input sequence, for which the Transformer fails to predict the desired outcome, might include errors. As a result, the values of correlation coefficients calculated from the attention matrix might include errors.
Constraint-basse algorithms for causal discovery are generally proved to be sound and complete when a perfect CI-test is used. However, if CI-test returns an erroneous result, the constraint- based algorithm might introduce additional errors. This notion is called stability (35), which is informally measured by the number of output errors function of the number of CI-tests errors. In fact, constraint- brased algorithm differ from one another in their ability to handle CI-test errors and minimize their effect. Thus, inaccurate correlation coefficients used for Cl-testing might lead to erroneous independence relations, which in turn may lead to errors in the output graph.
We expect that Transformer models become more accurate with larger training datasets, the accuracy of ABCD will increase. In this paper's experiments we use existing pre-trained Transformers and common datasets, and use the ICD algorithm (30) that was shown have state-of-the-art accuracy.
5 ABCD with Application To Explaining Predictions
One possible application the proposed ABCD approach is to reason about predictions by generating causal explanations from pre- rre-trined self-ateention models such BERT (9). Specifically, seek an explaining set that consists of a subset of the input symbols that are claimed to be solely responsible for the prediction. That is, if the effect of the explaining (alternative) prediction is provided by the neural network.
the prediction is masked, different
It was previously claimed that attention cannot be used for explanation (16); however, in contradict- ing paper (4T), it was shown that explainability is task dependent. A common approach is to use the attention matrix to learn about input-output relations for providing explanations (33: 5 4. These often rely on the assumption that inputs having high attention values with the class token influence the output (44) (6. We claim that this assumption considers only marginal statistical dependence and ignores conditional independence and explaining- away that may arise due to latent confounders.
To this end, we propose CLEANN (Causal. Explanations from Attention in Neural Networks). A description this algorithm detailed in Appendix B and an overview with application to explaining movie recommendation is given in Figure
See also Nisimov et al. (23 for more details.

---

Empirical Evaluation
In this section we demonstrate how a causal graph constructed from a self-attention matrix a Transformer based model can be used to explain which specific symbols in input sequence are the causes of the Transformer output. We experiment on the tasks of sentiment classification, which classifies an input sequence, and recommendation systems, which generates candidate list of recommended symbols (top-k) for the next item. For both experiments in this section we compare method against two baselines from (38): (1) Pure-Aftention algorithm (Pure-Atten.), that uses the attention weights directly in a hill-climbing search to suggest an explaining set, (2) Smart-Atentton algorithm (Smart-Autn.), that adapts Pure-Aftention, where symbol is added the explanation only it reduces the score gap between the original prediction's 's score and the second-ranked prediction's 's score. Implementation tools are in https: https:///ithhl com/ http://eelaaaabaauuaaattt-----a.
6.1 Evaluation metrics
We evaluate the quality of inferred explanations using two metrics. One measures minimality of the explanation, and the other measures how specific is the explanation to the prediction.
6.1.1 Minimal explaining set
An important requirement is having the explaining set minimal in the number symbols (40). The reason for that is twofold. (1) It is more complicated and less interpretable for humans to grasp the interactions and interplay in set that contains many explaining symbols. (2) In the spirit of occum's 's razor, the explaining set should not include symbols that do not contribute to explaining the prediction (when faced with a few possibles explanations, the simpler one is the one most likely to be true). Including redundant symbols in the explaining set might result in wrong alternative predictions when the effect of this set is masked.
6.1.2 Influence of the explaining set on replacement prediction
The following metric is applicable only to tasks that produce multiple predictions, such as multiple recommendations by recommender system (e.g., which movie watch next), as opposed to binary classification (e.g., sentiment) of an input. Given an input sequence consisting of symbols, {81,--- .1,.....y, 8n neural network suggests 8n-1, the 1st symbol in the top-k candidate list. CLEANN finds the smallest explaining set within & that influenced the selection of šn-1- As consequence, discarding this explaining set from that sequence should prevent that 8n+1 from being selected again, and instead a new symbol should be selected in replacement (replacement symbol). Optimally, the explaining set should influence the rank of only that 1 symbol (should be downgraded), but not the ranks of the other candidates in the top-k list. This requirement implies that the explaining set unique and important for the isolation and counterfactual explanation of only that 1 symbol, whereas the other symbols in the original top-k list remain unaffected, for the most part. It is therefore desirable that after discarding the explaining set from the sequence. the new replacement symbol would be one of the original (i.e. before discarding the explaining set) top-k ranked symbols, optimally the 2nd
6.2 CLEANN for Sentiment classification
We exemplify the explainability provided by our method for the task of sentiment elassification of movie reviews from IMDB (20) using pre-trained BERT model (9) that was fine-tuned for the task. The goal is to propose an explanation the classification of a review by finding the smallest explaining set of word tokens (symbols) within the review. Figure review that was classified as negative by the sentiment classification model. We compare between our method and two other baselines on finding minimal explaining set for the classification. We see that both baselines found the word 'bad' as a plausible explanation for the negative sentiment classification, however in addition, each of them finds an additional explaining word ('pizza', 'cinema') ') that clearly has no influence on the sentiment in the context appears the review. Contrary to that, our method finds the words 'bad' and 'but' explaining the negative-review classification. Indeed, the word 'but" may serve as plausible explanation for the negative review at the 2nd part of the sentence, since it negates the positive 1st part of the sentence (about the pizza). Additionally, Figure 2 shows
2 exemplifies the explanation of an input
7

---

Review Length
Algorithm
30
40
60
Pure- Attention
5.52 (4.34)
8.88 (7.59)
12.71 (10.30)
13.14 (12.47)
Amart-Attention
3.62 (2.45)
4.63 (3.11)
6.03 (3.34)
6.41 (4.74)
CLEANN
2.82 (1.71)
3,48 (2.18)
3,58 (1883
3.53 (1.54)
Table 1: Mean and standard deviation (in parenthesis) of explanation sizes for different review lengths (number of tokens in the input sequence). Each column is calculated using 25,000 reviews. CLEANN provides explaining sets having the smallest size average, with statistical significance tested using the Wilcoxon signed-ranks test at significance level 0.01 (indicated in bold).
the corresponding attention map at the last attention layer of the model for the given input review, In addition, in Figure we present the corresponding matrix representation of the learned graph (PAG). We can see that despite high attention values of the class token [cls] with some of the word tokens in the review (in the first row of the attention map), some of these words are found not be directly connected to the class token in the causal graph produced by our method, and therefore may not be associated influencing the outcome of the model.
Table D land Figure 3 show how the length of a review influences the explaining set size. CLEANN average, where statistical significance was tested using test at significance level a 0.01. For an increasing review length it is evident that the two baselines increase their explaining set, correspondingly. Even if the additional explaining tokens are of negative/positive context, they are not the most essential and prominent word token of this kind. Contrary to that, the explaining set size produced by our method is relatively stable, with moderate increase in the explaining set size, meaning that it keeps finding the most influential words in a sentence that dominates the decision put forward by the model.
produces the smallest explaining
Wilcoxon signed-ranke
Pure Amohtion
Iiked the pizza in cinema but the movie saw was bad
Smart Anwilon
Iiked the this oinea but the movis saw was bed.
CLEWNN
Iiked the this cinema but the movie saw was bad.
400
100
(
Figure 2: Comparison between the explaining sets produced by the methods given a negative review (a) Notice the implausible explaining tokens 'cinema' and 'pizza' found by the baseline methods. In contrast, in addition to the token 'bad', CLEANN found the token 'but', which is a plausible explanation as it is a negation to positive part the sentence that precedes it. (b) Attention matrix produced for the review by the model. (c) A matrix representation of the learned graph (PAG) produced by CLEANN (green: arrow head, yellow: arrow tail, blue: circle mark).
-Pan-Arma -Saun-na <CLBANN
-Pune -ww.tnaaaaaan -CLEANN
VHire Thh untellty expleationnti oer
milev sercc oxplreiiin
mytomctee
eewiew This unttd cxple selir
(d) Figure 3: Comparison between CLEANN and the baseline methods on the distribution of explaining set size over the various reviews. The reviews are sorted by their corresponding ex explanaiion size on the horizontal axis. CLEENN consistently finds smaller sets for: (a) review length 20, (b) review length = 30, (c) review length = 40, (d) review length 60.
(a)
(b)
(c)

---

Recommendation
Abductive Inference
Causal Discovery
Explain
5
%
Stor Vere Epeocs New Hope
Predect (1980)
Far Vhra Ecaodc' The reeuu
Becase you seaehho 11677
M
M
Aurssso Purc 11902
you would not have wircherlty
H
%/-
[mask]]
Sp01 Sus1
the Eutty (1966)
Sreuse V. The Faes
10000 For y (9980)
attention
(e)
(4)
(4)
Figure 4: An overview of the presented approach. As an example, M is pre-trained BERT4Rec recommender (36). (a) Given a set of movies (symbols) .s,....... s,]] the user interacted with, and the (n + 1)th movie masked, the recommender M recommends +n+1- Next, in the abductive inference stage (b) the recommendation is treated as input in addition to the observed user-movic interactions, that is, input: [si...., Sn, $n+1], and the attention matrix is extracted. (c) A causal structure learning stage in which the attention matrix is employed for testing conditional independence constraint- based structure learning algorithm. The result a Markov equivalence class. Here, as an example, causal DAG (a member of the equivalence class) is given, where H₃ and H2 are latent confounders. (d) A causal explanation reasoned from the causal graph.
6.3 CLEANN for recommendation system
We assume that the human decision process, for selecting which items to interact with, consists of multiple decision pathways that may diverge and merge over time. Moreover, they may be influenced by latent confounders along this process. Formally, we assume that the decision process can be modeled by a causal DAG consisting of observed and latent variables. Here, the observed variables are user-item interactions { 8x) unmeasured influences on the user's decision interact with a specific item. Examples for such unmeasured influences are user intent and previous recommendation slates presented to the user.
in a session S, and latent variables [,, Hy....) represent
We exemplify the explainability provided by our method for the task of recommendation system, and suggest as a means for understanding of the complex human decision- makiing process when interacting with the recommender system. Using an imaginary session that includes the recom- mendation, we extract an explanation set from the causal graph. To validate this set, we remove it from the original session and feed the modified session into the recommender, resulting in an alternative recommendation that, in turn, can also be explained in a similar manner. Figure provides an overview of our approach.
For empirical evaluation, we use the BERT4Rec recommender (36), pre-trained on the MovicLens IM dataset (15) and estimate several measures to evaluate the quality of reasoned explanations.
Minimal explaining set: Figure 5 (a) compares the explaining set size for the various sessions produced by CLEANN and the baseline methods. It is evident that the set sizes found by CLEANN are smaller. Figure (5tb) shows the difference between the explaining set sizes found by the baseline method and CLEANN, as calculated for each session, individually. Approximately 25% of the sessions are with positive values, indicating smaller set sizes for CLEANN, zero values shows equality between the two, and only 10% of the sessions are with negative values, indicating smaller set sizes for Pure-Aue Attention..
Influence of the explaining set on replacement recommendation: Figure 6(a) compares the distribution for positions of replacement recommendations, produced by CLEANN and the baseline methods, in the original top-5 recommendations list. It is evident that compared to the baseline methods, CLEANN recommends replacements that were ranked higher (lower position) in the original top-5 recommendations list. In different view, Figure bb shows the relative gain in the number of sessions for each position, achieved by CLEANN compared to the baseline methods. There is a trend line indicating higher gains for CLEANN at lower positions. That is, the replacements are more aligned with the original top-5 recommendations. CLEANN is able to isolate a minimal explaining set that influences mainly the 1st item from the original recommendations list.

---

-Pure-Atten. -Smart-Atten. -CLEANN
-Pure-Atten -Smmrt-Atten.
explanution size difference:
bascline -CLEANN
20
explanation
5
(a)
session IDs sorted by their explanation size
(b)
session sorted by explonation difference
Figure 5: Explaining set size. (a) Comparison between CLEANN and the baseline methods on the explaining set size (sorted by size along the horizontal axis) for the various sessions. Set sizes found by CLEANN are smaller. (b) The difference between set sizes found by the baseline method and CLEANN, presented for each individual session. Positive values are in favor of CLEANN.
porcestage of sessions (%6
Pare-AAtenn Smari-Altes CLEANN
change percentage in counts
0.6
Pure-Ali Arle-Alle. Smart-allen.
24
0.5
16
(a)
position of replacement recommendation
(b)
position of replacement recommendation
Figure 6: Positions of the replacement recommendations within the original top-5 recommendations list. (a) Position distribution of the replacements: compared to the baseline methods, CLEANN recommends replacements which are ranked higher in the original top-5 recommendations. (b) The relative gain in the number of sessions for each position, achieved by CLEANN. There is a trend line indicating higher gains for CLEANN at lower positions (replacements are more aligned with the original top-5 recommendation).
Discussion
We presented relation between the self-attention mechanism used in the Transformer neural architecture and the structural causal model often used to describe the data-generating mechanism.
One result from this relation is that, under certain assumptions, causal graph can be learned for a single input sequence (ABCD). This can be viewed as utilizing pre-trained models for zero-shot causal-discovery. An interesting insight is that the only source of errors while learning the causal structure is in the estimation of the attention matrix. Estimation is learned during gee-tranning the Transformer. Since in recent years it was shown that Transformer models scale well with model and training data sizes, we expect the estimation of the attention matrix to be more accurate as the pre-training data increases, thereby improving the accuracy of causal discovery.
Another result is that the causal structure learned for an input sequence can be employed to reason about the prediction for that sequence by providing causal explanations (CLEANN). We expect learned causal graphs to be able to answer a myriad of causal queries (24), among these are per- sonalized queries that can allow richer set of predictions. For example, in recommender systems, assuming that the human decision process consists of multiple pathways that merge and split, by iden- tifying independent causal pathways, recommendations can be provided for each one, independently.
The results of this paper contribute to the fields of causal inference and representation learning by providing relation that bridges concepts from these two domains. For example, they may lead to 1) causal reasoning applications for which large-scale pre-trained models or unlabeled data exist, and 2) architectural modifications to the Transformer to alleviate causal inference.
10

---

References
[1] Francis R Bach and Michael I Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jul):144) 2002.
[2] Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas leard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl, pages 507-556. 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[4] Chong Chen, Min Zhang. Yiqun Liu, and Shaoping Ma. Neural attentional rating regression with review-level explanations. In Proceedings of the 2018 World Wide Web Conference, pages 1583-1592, 2018.
[5] Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 765-774, 2019.
[6] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29, 2016.
[7] Tom Claassen, Joris M Mooij, and Tom Heskes. Learning sparse causal models is not NP-hard. In Uncertainty in Artificial Intelligence, page 172. Citeseer, 2013.
[8] Diego Colombo, Marloes Maathuis, Markus Kalisch, and Thomas S Richardson, Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294-321, 2012.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACI-HIT, pages 4171-4186, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
[11] Kenji Fukumizu, Francis R Bach, and Michael Jordan. Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5(Jan): 73-99, 2004.
[12] Olivier Goudet, Diviyan Kalainathan, Philippe Caillou, Isabelle Guyon, David Lopez-Paz, and Michele Sebag. Learning functional causal models with generative neural networks. Explainable and interpretable models in computer vision and machine learning, pages 39-80, 2018.
[13] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
[14] Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, and Bernhard Schölkopf. Kernel methods for measuring independence. Journal of Machine Learning Research, 6: 2075-2129, 2005.
[15] F. Maxwell Harper and Joseph A. Konstan. Trunsactions on Interactive Intelligent Systems (TūS), 5(4), dec 2015. ISSN 2160-6455. doi: 10.1145/2227872. URL sattps://dos orw/oogg///// 11....115/2227777
The movielens datasets: History and context. ACM

---

[16] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pages 3543-3556, 2019.
[17] Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Structural agnostic modeling: Adversarial learning of causal graphs. Journal of Machine Learning Research, 23(219):1-63 2022.
[18) Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pages 197-206. IEEE, 2018.
[19] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[20] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142- 150, Portland, Oregon, USA, June 201 1. Association for Computational Linguistics. URL http:///wwt awwweb.w............ htw.wweb.rggeoo..e..ooooo org/aatholoog//11111000
[21] Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention-basee based con- volutional neural networks. Machine Learning and Knowledge Extraction, 1(1):312-40, 2019.
[22] Shami Nisimov, Yaniv Gurwicz, Raanan Yehezkel Rohekar, and Gal Novik. Improving ef- ficiency and accuracy of causal discovery using hierarchical wrapper. In Uncertainty in Artificial Intelligence (UAI 2021), workshop on Tractable Probabilistic Modeling, 2021.
[23] Shami Nisimov, Raanan Y Rohekar, Yaniv Gurwicz, Guy Koren, and Gal Novik. CLEAR: Causal explanations from attention in neural recommenders. Conference on Recommender Systems (RecSys), workshop on Causality, Counterfactuals and Sequential Decision-Making (CONSEQUENCES), 2022.
[24] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge university press, second edition, 2009.
[25] Judea Pearl. An introduction to causal inference. The international journal of biostatistics, 6(2). 2010.
[26] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[27] Judea Pearl and Thomas Verma. theory inferred causation. International Conference on Principles of Knowledge Representation and Reasoning, pages 441-452, 1991.
[28] Peters, D. Janzing, and Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA, 2017.
[29] Thomas Richardson and Peter Spirtes. Ancestral graph markov models. The Annals of Statistics, 30(4):962-1030 2002.
[30] Raanan Y Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik. Iterative causal discovery in the possible presence of latent confounders and selection bias. Advances in Neural Information Processing Systems, 34:2454-2465, 2021.
[31] Raanan Yehezkel Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik. From tempo- ral to contemporaneous iterative causal discovery in the presence of latent confounders. In International Conference on Machine Learning, pages 39939-39950. PMLR, 2023.
[32] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
12

---

[33] Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In Proceedings of the eleventh ACM conference on recommender systems, pages 297-305, 2017.
[34] Peter Spirtes. Introduction to causal inference. Journal of Machine Learning Research, 11 (May):1643-1662, 2010.
[35] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction and Search. MIT Press, 2nd edition, 2000.
[36] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang, BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Pro- ceedings of the 28th ACM international conference on information and knowledge management, pages 1441-1450, 2019.
[37] Xiaohai Sun, Dominik Janzing, Bernhard Schölkopf, and Kenji Fukumizu. A kernel-based causal learning algorithm. In Proceedings of the 24th international conference on Machine learning, pages 855-862, 2007.
[38] Khanh Hiep Tran, Azin Ghazimatin, and Rishiraj Saha Roy. Counterfactual explanations for neural recommenders. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1627-1631, 2021.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[40] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. 2017. doi: 10.48550/ARXIV.7111 00399. URL.
htt.b/////...000 00399
[41] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (ICNLP, pages 11-20, 2019.
[42] Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal- connection: Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823-10836, 2021.
[43] Kevin Xia, Yushu Pan, and Elias Bareinboim. Causal models for counterfactual identification and estimation. In Proceedings of the 11th Eleventh International Conference on Learning Representations, forthcoming, 2023.
[44] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pages 2048-2057. PMLR, 2015.
[45] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[46] Raanan Ychezkel and Boaz Lerner. Bayesian network structure learning by recursive autonomy identification. Journal of Machine Learning Research, 10(7), 2009.
[47] Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(1 1877 2008.
[48] Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Sevene Conference Uncertainty in Artificial Intelligence, pages 2011.
13

---

A Notations
In this section we provide Table 2 which describes the main symbols used in the paper.
Table 2: Notations used in the paper. The first set of symbols describes entities in the structural causal model, and the second set of symbols describes entities the Transformer neural network.
Symbol
Description
X;
random variable representing node in SCM
U
latent exogenous random variable i in an SCM weighted adjacency matrix of SCM causal graph (unweighted, directed-graph structure)
G
g
Z;
output embedding of input symbol i, Z; Z(i, .), in attention layer
V.
value vector corresponding to input i, V. (1, +), in attention layer attention matrix
A
M
Transformer neural network
Wy. WoK
learnable weight matrices in the Transformer neural network
B Finding Possible Explanations from Causal Graph
This appendix includes the pseudo-code for the CLEANN (CausaL Explanations from Attention in Neural Networks) algorithm and explanatory figures. By abduction, we replace the masked symbol with predicted one, creating an imaginary sequence. We then feed-forward this sequence and extract the resulting attention matrix (Algorithm) [ lines 2-4). Using this attention matrix we learn a causal graph as discussed in Section 3.2 (Algorithm lines 5-7).
Given causal graph, various "why" questions be answered (26). We follow (38 explaining prediction using the symbols from the input sequence. That is, provide the minimal set of symbols that led specific prediction and provide an alternative prediction. To this end, define the following.
Definition 1 (PI-path). A potential influence path from A to Z in PAG G, is path II Z) (A...., Z), such that for every sub-path (U, V, W) of TT,, Z), where there are arrow heads into V and there is no edge between U and W in G.
Essentially, a PI-path ensures dependence between its two end points when conditioned on every node on the path (a single edge a Pl-path).
Definition 2 (PI-tree). A potential influence tree for A given causal PAG G, is a tree T rooted at A, such that there is path from A to Z in T, (A, Vi...., Vk, 4), if and only if there is an Pl-path (A, Vi...., VR, Z) in G.
Definition (PI-set), A set E is potentially inftuencing set (PI-set) on item A, with respect to if and only if:
1.
2. VE E there exists Pl-path, II(A, E) such that VV e E), e E
3. VE E E, E temporally precedes A.
An example for identifying Pl-sets is given in Figure 7 Although a Pl-set identify nodes that are conditionally dependent on the prediction, it may not be a minimal explanation for the specific prediction. That is, omitting certain symbols from the session alters the graph and may render other items independent of the prediction. Hence, we provide an iterative procedure ("FindExplanation" in Algorithm lines 10-18), motivated by the ICD algorithm (30), to create multiple candidate explanations by gradually increasing the value of r. See an example in Figure [ Note that from conditions (1) and (2) of Definition 3 effectively represents the maximal search radius from the prediction (in the extreme case, the items of the Pi-set lie on one PI-path starting at the prediction). The search terminates as soon as set qualifies as an explanation (Algorithm [
line 17). That is, as soon as different prediction is provided. If no explanation is found (Algorithm I line 19) higher value of should be considered for the algorithm.
14

---

Algorithm 1: CLEANN: CausaL Explanations from Attention in Neural Networks
Input:
8: a sequence 81; ... Sn, (mask))
M: a pre-trained attention-basee model
&n+1: prediction by M
a: significance level for CI-testing
Output:
E: an explanation set for the prediction
8 4 an alternative prediction (after masking the effect of the explanation)
G causal graph of : S2, 5n+1]
Function Main (8, M, 8n+1):
+ [s1..,.... Sn 5n+1]
replace mask with prediction
forward pass: _M)
get A, the attention matrix in the last layer of M(S)
define correlation matrix P3 Pi((, ()i C(i, (j..... where C ÄÄT
define (n) conditional independence test based on partial correlation CausalDiscovery (pn) a)
E, 1,1 FindExplanation (G, 5n41. M)
return E, 1.
10 Function FindExplanation 8,+1, M, s): create Т: Pl-tree for 8n+1 given g for r in {1,.... n 1} do create &, subsets of Nodes(T) 5n41 such that VE E, Pl-set 8,41 in with respect r. for E E do
12
13
15
+- 8 E 8,41 M(s')
16
b get an alternative prediction b return immediately ensuring smallest no explanation found at the current significance level
if R+1 # Sn+1 then return E, 2, explanation set
18
19
return 0, 9
C Proofs
In this section we provide proofs for the results described in Section 4.2
Let (In,, P.)) be SCM-consistent probability space, and M trained sequences from this probability space, complying with [
Theorem 1. s is sequence from an SCM-consistent probability space used to train M, Ga the causal graph of the SCM from which s was generated, and A the attention matrix for input sequence the deepest layer, then Ga is in the equivalence class learned by sound constraint-based algorithm that uses covariance matrix Cz = AAA for partial correlation Cl-testing.
Proof. From Assumption ii follows that output embedding, in the deepest attention layer. for each symbol is trained to predict the input symbol. It follows that the output embeddings of the deepest layer are one-to-one (injective) representations of the input symbols. Thus, the causal relations among the input symbols can be cast on their correponding output embeddings. That is, causal graph over the output embeddings is similar causal graph over the input symbols Gz G.. In Section 4.1 we showed that aiffatttetion in the Transformer architecture can be viewed as an SCM over the output embedding, matrix (Equation 9). A conditional independence test between output embeddings can then be defined calculate partial correlations from Cz. Finally, sound constraint baaee algorithm (35) that uses Cz for partial correlation Cl-testing, recovers equivalence class of Gz.
the covariance matrix, Cz, can be computed from the attention
15

---

H1
H1
2
H
Hz
explaining
confounding
(+)
(b)
(c)
Figure 7: An example for Pl-set on J5. Latent nodes are represented by squares. Nodes included in the PI-set are circled with a thick red line. Indirect dependency is marked with dashed red line. (a) The smallest PI-set for fully observed DAG is the Markov blanket {12, H2). However, H2 is latent. (b) Nodes I2, Is are included in the PI-set as they have direct effect on Is. where I3 depends on Is due the latent confounder Hz. However, I3 collider, henee including it in the Pl-set results in explaining-away and the Pl-set is not complete. (c) A complete PI-set { 12, Ia Including I3 creates an indirect dependence between Is and I (a Pl-path), which requires including as well. Note that the distance from I5 to /1 is two edges in the graph over observed variables.
soarch radis
64)
Figure 8: An example for (a) a causal graph (PAG), and (b) the nodes that influence the prediction Is having Pi-tree structure, depicted on coordinates where the radius axis is the distance from the prediction. In Algorithm line 12, the algorithm iterates over radius values, starting with In the first iteration, the sets {/2} and {/3} are tested if any of them qualifies as an explanation. If both sets fail, the search radius increases and the sets {I2, Is} and {I1, I3} are tested (set {I2, Is} is tested first members have smaller average distance from [s). both fail, r 3 and the last tested set (T,, J2, Iz).
Corollary 2. Nodes X; and X, are d-separated in Gz conditioned on X' C X if and only if symbols 8; and 8, are d-separated in Ge conditioned on s', a subset containing symbols corresponding to X'.
Proof. This result follows directly from Theorem 1 and faithfulness assumption (Definition) 2
16


Reviews:
[{"review_id": "TziCncgfxF", "reviewer": "Reviewer_wsNj", "summary": "The paper proposes Attention-Based Causal Discovery (ABCD), a causal interpretation for the attention matrix of transformers (e.g., LLMs like BERT). The authors suggest CLEANN, an algorithmic approach that first finds a causal structure of the input symbols (e.g., tokens (words) of a sentence as input to BERT). This causal structure is unique to each input (sentence). This can then be used to get causal explanation for different taks (in their experiments classification or recommendation) based on the representations of the input sentences (e.g., the word \"but\" and \"bad\" in the input sentence caused the negative sentiment classification).", "questions": "I have a few questions: \n\n* Section 2.1, ln. 50, Definition of an SCM. I have previously seen an SCM being defined often $M = (F  P(U))$, or $M = (X, U , F)$, but I have not seen both U or P(U). Is yours the proper formal definition? \n\n* Section 2.1, ln. 59, your understanding of a linear-Gaussian SCM is the same as an Additive Noise Model (ANM)? \n\n* Section 5, Fig. 6: I do not understand the x-axis. Could you explain this again? \n\n* Are there other applications of ABCD beyond providing causal explanations? \n", "limitations": "Section 6 Discussion:  ln. 305 mentions \"under certain assumptions\", but these assumptions are not re-stated here. It would be helpful to have them restated. In this regard, I am missing a discussion of the limitations regarding the (causal) assumptions. \n\nI am also wondering, how robust your causal explanations are? I believe, when providing causal explanations to individuals, it is important to discuss any limitations regarding the sensitivity to assumptions and to comment the potential negative societal impact, if assumptions are not met.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thank you for your submission. I enjoyed reading your paper. \n\nClarity & Presentation: I found the paper to be clearly structured and written. On clarity and presentation, I would like to positively highlight the introduction to each (sub) section gave an orientation that helped to clearly guide the reader, what the next chapter is about and how it fits in the bigger picture. The figures (e.g., Fig2a, Fig 4) helped a lot to understand the method/pipeline.\n\nNovelty: I found the causal interpretation and its application to causal explanation of predictions apealing and novel. Finding a causal explanation to downs-stream tasks of LLMs for example, can be very helpful. However, I am not an expert in the field of LLMs and I do not feel qualified to fully assess the novelty and hence the contribution of the work in the general field.\n\nEvaluation: I found the experimental section clearly structured and interesting to read.", "weaknesses": "I have a few comments for improvements. \n\n* Section 5, Table 1: What do the bold numbers indicate, what the +- (i.e., over how many runs did you do it, were these runs of CLEANN or different data samples)? This would be helpful to comment in the caption. Am I right in understanding tha CLEANN does yield lower average explanations, but the numbers suggest that it is not a significant? Over how many runs did you do it?  \n\n* Section 5, Figure 2 (b, c): I find it hard to understand and interpret Figure 2(b,c) despite your explanation in ln. 255ff. Also, the labels are very small and hard to read. \n\n* Section 5, Figure 3: I am missing an explanation/interpretation of Figure 3 in the text. Did I overlook it? \n\n* I am missing a related work section. \n\nMinor: \n* Section 1: While I enjoyed reading the empirical evaluation, I found the causal explaining application could be stronger highlighted in the introduction. \n\n* Section 2.1, line 67: While I appreciate the introduction to SCMs, I was a bit confused about the Causal Markov and Faithful properties at the end of the section. They seem very disconnected of the rest of the text. It would help, if there would be a short explanation, why they are stated / needed for the paper. \n\n* Section 2.2 is very brief. In the light of Section 3 (ln. 100 ff.), where the self-attention mechanisms is further detailed, it makes sense. However, it may be helpful to just point to that at the end of the section 2.2. \n\n* Section 5: Caption of Table 1 - by review length, you mean the length of the sentence, right? It would have helped me  to point that out. \n\n* Section 5: I find it weird that Fig. 6 is discussed before Fig. 5.  A natural way would be to discuss them the other way around.\n\n* I got a bit confused by the difference between ABCD and CLEANN. Could you kindly clarify?\n\nTypos:\n* ln 222 \"(1) In addition, it\" ->  \"(1) It\" \n* ln 262 \"influences on the \" -> \"influences the\"\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank you for the very detailed review and your positive feedback. Also, thank you for your comments for improvements that mitigate potential weaknesses. Addressing them as suggested strengthens the contribution and makes it clearer.\n\n**Re: Your comments for improvements (weaknesses).**\n* Re. Section 5, Table 1.: We will add the following clarification to the caption and in the text. The bold numbers indicate statistical significance with Wilcoxon signed-ranks test at significance level $\\alpha=0.01$ (see also sorted pairwise difference in Figure 3). They are also statistically significant with the single-sided $t$-test. In both tests the $p$-value is very small. Mean and standard deviation are calculated over 25000 samples. We will remove the $\\pm$ sign and place standard deviation values in parenthesis.\n* Re. Section 5 Figure 2 (b, c): We will increase the size of depicted matrices in Figure 2 (b ,c). In addition, we will clarify in the caption that these illustrate that high attention values do not necessarily translate to direct causal relation as in the case for CLS highest attention to the token 'was' (2.b) whereas it is disconnected from it in the causal graph (0 value in graph matrix in 2.c).\n* Section 5 Figure 3:  Thank you for spotting this. Line 257 should have been \"Table 1 and Figure 3 show …\", instead of \"Table 1 shows\". Figure 3 is given to demonstrate the significance of the results summarized in Table 1 (it is clear from Figure 3 that there is statistical significance when using the Wilcoxon signed-ranks test).\n* Related work is Section E in the supplementary material. We will move it to the main paper. \n\n**Re: Minor comments.**\n* As suggested we will add the causal explaining application at the end of the first introduction paragraph (line 24).\n* We will replace line 61, starting with ('In this paper we …'), with 'In this paper, we learn the causal graph, described by the non-zero elements of G, using constraint-based causal discovery approach. It requires the causal Markov and faithfulness assumptions.'. This will make a clearer connection to the assumptions. Thanks.\n* We will clarify at the end of section 2.2 that in Section 3 we will further detail self attention as required for establishing a link to SCM.\n* 'review length' means total number of tokens. we will improve Table 1 caption as suggested.\n* We will fix the ordering of Figures 5 and 6 as suggested.\n* ABCD is the approach that links SCM and self-attention, and provides means to learn the causal structure from pre-trained transformers. CLEANN is an algorithm that firstly employs ABCD to learn a causal structure. It then iteratively traverses the learned structure and searches for an explanation for the Transformer's output, which involves forward passes of the neural network to test the explanation.\n\n**Re: Typos.**\nThank you for identifying these typos. We will correct them as well as proofread the final version.\n\n**Answers to the question.**\n* Definition of SCM: Since we are interested in the covariance of endogenous variables, we follow the definition of probabilistic SCM by Pearl (Causality, 2009, definition 7.1.6). This is also the definition of SCM generally used by E. Bareinboim (e.g., Xia et al., 2021, Definition 1. Reference [10] in the supplementary material).\n* The noise in the Linear-Gaussian SCM we used is additive and is normally distributed.\n*  The X-axis in Figure 6: We ordered the sessions (data samples) on the X-axis in an order obtained by sorting the values of the Y-axis (e.g., explaining set size per session). This way, the plots are monotonically increasing. We will clarify the label. By sorting the values, we provide a description of the empirical cumulative distribution function (CDF). Figure 6-(a) describes how the explanation sizes are distributed. Figure 6-(b) describes how the difference between CLEANN and other baselines is distributed. \n* In essence, ABCD enables zero-shot causal discovery. Inferring the causal relations solely from observational data is important in many sciences (e.g., healthcare, economics, physics, chemistry). In this paper we demonstrated one practical use. It is readily practical for currently available Transformers. We expect that future applications, interested in recovering underlying causal structures in a zero-shot manner, may train attention-based models and use ABCD. For example, understanding the health condition and its direct causes for a specific user, after training on a large dataset of observations for humans. Another possible example is pretraining a BERT model on sequences of amino acids (proteins), and inferring various relations that may lead to insights on the specific protein's folding mechanism. These examples are not part of the paper and given here just for describing potential use of ABCD.\n\n**Re: Limitations.**\nWe discussed the limitation of ABCD in Section D in the supplementary material. We will move it to the main paper (using the extra 1 page available for the camera ready version). We will refer that the violation of the assumptions leads to errors in constraint-based causal discovery (as discussed in the third paragraph of Section D). Also, as suggested, we will restate them in the discussion.\n\nFinally, in light of your initial positive review and considering our answers we hope you will reconsider your initial overall score."}, {"review_id": "LzhJ5ZiVOR", "reviewer": "Reviewer_75HH", "summary": "In this work, authors, interpret self attention in transformers as linear gaussian SCM over output embeddings. \nSCM over input embeddings is same as SCM over output embeddings, since deepest attention layer is trained to predict input symbol.\nExisting conditional independence based testing algorithm is applied on the weights of the deepest attention layer of the pretrained transformer in order to estimate causal structure. \nFurthermore, the authors demonstrate the applicability of the proposed approach in deducing causal explanations for attention based recommendations.", "questions": "In the paper, the covariance matrix of value is assumed to be identity using central limit theorem, however, in practice, it wouldnot be identity as a result the graph learnt wouldnot be faithful. This will act as one more reason of inaccuracy. Is it possible for you to throw some light on this ?", "limitations": "1) The accuracy of the proposed method as explained by the authors is dependent on the prediction of the transformers and the conditional independence testing accuracy\n2) Authors assumed learnt graph will be faithful, since exogeneous variables V under central limit theorem will have identity covariance, however in practice the faithfullness assumption will be violated, this will act as one more reason of inaccuracy", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Authors proposed novel way of viewing self attention in transformers as weight matrix of linear gaussian SCM\nPretrained model is used to obtain causal structure, which would be of great use to gain causal perspectives for existing model\nEstimating causal structure from the attention layer and using it for causal explanantions is promising.", "weaknesses": "Theorem1, in main paper has few words missing, however, it's available in supplementart material", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we wish to thank you for your review, positive feedback, and questions that allowed us to further improve this contribution.\n\n**Re: Weaknesses.**\nThank you for spotting the cut-off sentence, and finding it complete in the supplementary material. We will correct that.\n\n**Re: Questions.**\nEach column of $\\mathbf{W}_V$ represents an axis on which the input is projected resulting an instance for the exogenous values of the SCM (dot-product between embedding of token $i$ from the previous layer with column $j$ of $\\mathbf{W}_V$ is the element $(i, j)$ of the values matrix). Thus, each column of the values-matrix, $\\mathbf{V}$, is treated independently, and the covariance is a diagonal matrix. Also, STD is 1 on the diagonal and scaled within $\\mathbf{A}$ as can be seen by parameter $\\Lambda$ that scales the STD in SCM in Equation 9. Thus, we consider the identity covariance matrix. From the SCM perspective, each column of $\\mathbf{V}$ is a unique context (instance of the exogenous variables) for which $\\boldsymbol{X}$ values are calculated. The $\\boldsymbol{X}$ values is a column in the self-attention output $\\mathbf{Z}$. Each column corresponds to a specific context, and the concatenated columns into matrix $\\mathbf{Z}$ is used for further processing by the Transformer (see Figure 1).\n\n**Re: Limitations.**\n1. As you indicated, we described the limitations in the supplementary material. For improving the clarity we will move Section D, 'Limitations of ABCD' to the main paper using the extra 1 page available for the camera-ready version.\n2. We will add to the limitations section that violation of faithfulness, which might happen in real-world cases, will cause errors in the constraint-based causal discovery algorithm. Nevertheless, this does not reflect a violation for the link we derived between self-attention and SCM.\n\nAddressing the points in your review improves this contribution, and we hope that in light of our clarifications and answers you will consider updating the overall score of this contribution."}, {"review_id": "TzPUKWqdTm", "reviewer": "Reviewer_qdMo", "summary": "The ABCD paper starts off by motivating the need for understanding the complex dependencies that underly current Transformer architectures, pointing to causality as the missing puzzle piece. Following brief introductions of both SCMs and Self-attention, the authors then provide covariance matrix derivations for both linear Gaussian SCM and attention such that they are able to equate them eventually. By introducing another assumption on how the attention was trained, they propose a theorem that provides a clear connection between causal discovery and attention in that one can make CI tests based on the attention matrix essentially. Finally, the authors propose a complete pipeline for going from e.g. a pre-trained Transformer model to causal explanations based on their ABCD approach computed from the last attention layer. As put by the authors, this can be considered as a \"one-shot\" causal discovery paradigm.", "questions": "1. Can you comment on Theorem 1 as mentioned in the weaknesses section?\n\n2. The assumption of linear Gaussian SCM can severely limit the method. What was the reasoning behind the assumption? \n\n3. What property is required for the deepest attention layer such that the equating to CB-CD works.\n\nOverall, a nice contribution but not devoid of weaknesses. I will be willing to raise my score based on the rebuttal.", "limitations": "No concerns here.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A unique perspective and valuable contribution to the causal discovery literature in that it expands the horizon on what kind of connections can be established between existing models to foster new understanding of said models. The derivation of the covariances and then equating them is both simple and effective, and then Thm.1 seems like a nice result for understanding the final correspondence between attention and CI-testing for constraint-based causal discovery.\n\n2. Presenting a complete pipeline is a great bonus (since it is a difficult task) and shows an immediate practical approach to the previously presented theory. The explanation approach is convincing and practically applicable since the theory justifies the use of the attention matrix.", "weaknesses": "1. While the causality assumptions Markov and faithfulness can be considered standard, as soon as moving to the realm of data that nowadays Transformers are confronted with we run the risk of violating them. Furthermore, the assumption of a linear Gaussian SCM is quite restrictive and should be highlighted more, especially under consideration of Transformers. Finally, Assumption 1 right before Theorem 1 is only vaguely comprehensible IMHO. I'd wish for the authors to be more specific about what property is required for the deepest attention layer such that the equating to CB-CD works. This could be done through formalization of that assumption or if not possible, then through an example.\n\n2. Thm.1 is incomplete writing-wise i.e., sentence cuts off at end of page 4. Starting from p.4 the paper generally lend the impression of being not as carefully written as the pages before. Please consider a careful readthrough post-submission.\n\n3. Missing discussion of explanation methods based on causal graphs. Two very different approaches include Causal Shapely Values by Heskes et al. 2020 and Causal Explanation of SCM by Zečević et al. 2021. The first is a rather classical approach, where numerical attributions are being computed and in Sec.5 they present an approach for causal chain graphs (a special case like your Linear-Gaussian case), and the second discusses a traversal algorithm for generating immediate textual representations based on more general SCM (although only showcased for a linear SCM setting).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank you for you detailed review, positive feedback, and identifying weakness that addressing them significantly improves this paper.\n\n**Re: Weaknesses.**\n1. **(a) Re: Causal Markov and faithful assumptions.** The causal Markov and faithfulness assumptions are required for constraint-based causal discovery. We will clarify this just before their definition in the preliminaries section by adding in line 61 'In this paper, we learn the causal graph, described by the non-zero elements of $\\mathbf{G}$, using constraint-based causal discovery approach. It requires the causal Markov and faithfulness assumptions.'. Nevertheless, they are not required for the link established between SCM and self-attention in Transformers. We will clarify that a violation of these assumptions will hinder the accuracy of the causal discovery algorithm (third paragraph of Section D: 'Limitations of ABCD' in the supplementary material). We will move Section D to the main paper using the 1 extra-page provided for the final camera-ready version.\n**(b) Re: linear Gaussian SCM.** We wish to clarify that this is not a restriction (assumption) but rather a result. We showed in section 3.1 that self attention can be viewed as a mechanism that estimates values of a linear-Gaussian SCM nodes. Specifically, Equation 4 describes the total effect one variable has on another one via all directed paths of a linear-Gaussian SCM. This is similar to the attention one token is given by another token. We then show in Equation 9 that the covariance over endogenous variables of the linear-Gaussian SCM can be computed using the attention matrix. The Transformer learns to 'disentangle' complex relations between input symbols into linear relations in the last attention layer (under Assumption 1).\n**(c) Re: Assumption 1.** In the supplementary material in Section C (Proofs), we provided a formal definition to the probability space (Definition 6) on which the assumption relies. In addition, since it is referred to in the proof of Theorem 1, a motivation for it is given. We will move the formal definition of the probability space and the accompanying explanations to the main paper using the extra 1 page available for the camera-ready version. We will also explain that common pre-training of self-attention (e.g., Mask Language Model, MLM in BERT) validate this assumption. In addition, we will explain that commonly in an unsupervised training of Transformers, the output embedding of a token alone (ignoring the output embeddings of other tokens) is used to predict the input token.\n2. We regret that the Theorem cuts off and sincerely apologize for that. The full Theorem 1 along with more a formal description is in Section C of the supplementary material. We will correct this along with other clarities you spotted for the camera-ready version, based on the supplementary material.\n3. We will discuss both papers as suggested with respect to the CLEANN algorithm.\n\n**Re: Questions.**\n1. Theorem 1, and other related details will be corrected based on Section C of the supplementary material (as addressed in our response to 'weaknesses 2'.\n2. The linear-Gaussian SCM is not an assumption. It is a result. We will clarify this in the paper as discussed in Section C in the supplementary material. In section 3.1 we showed that self-attention can be viewed as a mechanism that estimates the total effect one node has on another node in a Linear-Gaussian SCM, and used this to calculate the covariance (Equation 9). We addressed this question in more detail in our response to 'weaknesses 1.b'.\n3. The output embeddings of the deepest layer are one-to-one (injective) representations of the input symbols (Section C, proof of Theorem 1). That is, the input token should be able to be predicted using only its corresponding output embedding of the deepest layer. This commonly the optimization goal in Transformer pre-training (e. g., MLM for BERT). We provide more details in our response to 'weaknesses 1.c'.\n\nWe would like to sincerely thank you for the questions and suggestions that allowed us to improve the overall quality and clarity of this paper."}, {"review_id": "EgxF4R2def", "reviewer": "Reviewer_unWZ", "summary": "The paper establishes a formal link between a (pre-trained) self-attention layer and a causal graph underlying a sequence of symbols. Specifically, it shows that self-attention learns to represent pairwise associations for which there exists a linear-Gaussian Structural Causal Model that has the exact same pair-wise associations between its nodes. The paper then proceeds to derive an algorithm for recovering the causal structure of the SCM solely from the weights of a pre-trained attention layer", "questions": "1. I want to see a serious reconsideration of all the empirical evidence the paper provides. \n2. How can the causal explanations be used in an easy and/or automated setting?", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The Attention-Based Causal-Discovery (ABCD) method, which is developed for learning an equivalence class of the causal graph that underlies a given input sequence is reasonable. It relies on partial correlarion CI-testing, which is sound.\nThen the paper proceeds to apply ABCD to reasoning about predictions by generating causal explanations from pre-trained self-attention models, such as BERT (8). Specifically, the algorithm produces explanations through selecting a subset of the input symbols that are claimed to be solely responsible for BERT prediction. ABCD is shown empirically to be much stronger than the common approach of using the attention matrix to learn about input-output relations for providing explanations. ", "weaknesses": "The Attention-Based Causal-Discovery (ABCD) method relies on an unsupervised training assumption. This restricts applicability to only models of BERT type, you cannot deal e.g. with decoder-only models. \n\nIn addition, it is extremely troubling that the authors do not perform any comparative evaluation against standard methods in the field. They limit their examination to standard self-attention and a variant of it. Even worse, they have avoided use of any standard benchmark in the field. \n\nEventually, it is not clear how useable the generated causal explanations are, and how easy it is to exploit them in an application setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your review, and the important questions. We hope that our response below makes this paper's contribution clearer, and mitigates the concerns you raised.\n\n**Re: weaknesses.**\nWe wish to clarify that the main contribution of the paper is twofold. The first is a link between the unknown causal model governing the input symbols/token and a simple linear-Gaussian SCM over the embeddings calculated in the deepest self-attention layer. To the best of our knowledge, this is the first link made between pre-trained self-attention-based models and SCM (Figure 1). The second part of the contribution is a method that recovers a causal structure for a specific input sequence in a zero-shot manner. Generally in causal discovery, a large number of samples generated from a single causal models are used to recover this single causal model. In this contribution, each data sample is generated by a different causal model (Definition 6, Section C in the supplementary material). To the best of our knowledge, this is the first method for this setting and the first method that learns causal graphs from pre-trained self-attention. As you pointed out, this paper proposes a solution for self-attention and not for cross-attention (or decoder-only models). We note that this paper proposes a novel and non-trivial causal interpretation to self-attention, and since a similar treatment for cross-attention is not an incremental extension we leave it for future work.\n\nRegarding benchmarks, currently there is no known benchmark that contains a set of sequences (data samples) and their corresponding causal graphs; each sample has its own graph. Nevertheless, we quantitively evaluated explanations deduced from the learned causal graphs, and demonstrated their superiority over baselines that use the values of self-attention. This quantitively measures the usefulness of causal structures learned using the novel link established by ABCD. \n\nRegarding the usability of explanations and their ease of application in an automated system, here are three ways. 1) one example is improving training by penalizing/rewarding the explaining set, which can be traced-back to the attention weights, of an incorrect/correct prediction. This can be done for each sample in the training data independently. 2) In the paper we demonstrated the usefulness of ABCD for automatically creating causal explanations for the BERT4Rec [28] recommender system by proposing the CLEANN algorithm (Section B). An explanation to why a specific recommendation was given to a human in a tangible way can lead to greater trust in the recommendation, and greater human engagement with the system. 3) Finally, (in the recommender system setting) understanding the human decision process (causal graph over user-item interactions) can be used to enhance recommendations (beyond 'linear' recommendations). For example, by identifying independent causal pathways (some may pause in time, before others), and creating recommendations for each pathway, thereby enriching recommendations.\n\n**Re: Questions.**\n1. Please see the second paragraph in 'Re: weaknesses'.\n2. Please see the third paragraph in 'Re: weaknesses'.\n\nWe are confident that addressing the concerns you raised will improve the clarity of this paper and how it can promote future work in bridging Transformers and causality. We hope that our answers will allow reexamination of the contribution of this paper."}]

Author rebuttal:

Meta-review: The paper presents a causal interpretation for the attention matrix of transformers. While all reviews agree that this interesting, the authors have to highlight and discuss much more the assumptions made, in particular, the main assumption of causal Markov and faithfulness. Indeed, they are mentioned but it is not stressed that this simplified the whole story to independence testing. This is important, as there a known connection between transformers (self attention) and probability densities. There are even probabilistic attention layers. This discussion is very much required. Apart from this, I agree with the overall positive assessment of the reviewers.
Decision: Accept (poster)