Title: Causal Interpretation of Self-Attention in Pre-Trained Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-Attention

Causal Discovery

Reasoning

Explainability

Zero-shot

Transformer
Abstract: We propose a causal interpretation of self-attention in the Transformer neural network architecture. We interpret self-attention as a mechanism that estimates a structural equation model for a given input sequence of symbols (tokens). The structural equation model can be interpreted, in turn, as a causal structure over the input symbols under the specific context of the input sequence. Importantly, this interpretation remains valid in the presence of latent confounders. Following this interpretation, we estimate conditional independence relations between input symbols by calculating partial correlations between their corresponding representations in the deepest attention layer. This enables learning the causal structure over an input sequence using existing constraint-based algorithms. In this sense, existing pre-trained Transformers can be utilized for zero-shot causal-discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.

Causal Interpretation of Self-Attention in Pre-Trained Transformers

Abstract

We propose a causal interpretation of self-attention in the Transformer neural network architecture. We interpret self-attention as a mechanism that estimates a structural equation model for a given input sequence of symbols (tokens). The structural equation model can be interpreted, in turn, as a causal structure over the input symbols under the specific context of the input sequence. Importantly, this interpretation remains valid in the presence of latent confounders. Following this interpretation, we estimate conditional independence relations between input symbols by calculating partial correlations between their corresponding representations in the deepest attention layer. This enables learning the causal structure over an input sequence using existing constraint-based algorithms. In this sense, existing pre-trained Transformers can be utilized for zero-shot causal-discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.

1 Introduction

Causality plays an important role in many sciences such as epidemiology, social sciences, and finance (25; 34). Understanding the underlying causal mechanisms is crucial for tasks such as explaining a phenomenon, predicting, and decision making. An automated discovery of causal structures from observed data alone is an important problem in artificial intelligence. It is particularity challenging when latent confounders may exist. One family of algorithms (35; 7; 8; 46; 30), called constraintbased, recovers in the large sample limit an equivalence class of the true underlying graph. However, they require a statistical test to be provided. The statistical test is often sensitive to small sample size, and in some cases it is not clear which statistical test is suitable for the data. Moreover, these method assume that data samples are generated from a single causal graph, and are designed to learn a single equivalence class. There are cases where data samples may be generated by different causal mechanisms and it is not clear how to learn the correct causal graph for each data sample. For example, decision process may be different among humans. Data collected about their actions may not be represented by a single causal graph. This is the case for recommender systems that are required to provide a personalized recommendation for a user, based on her past actions. In addition, it is desirable that such automated systems will provide a tangible explanation to why the specific recommendation was given.

Recently, deep neural networks, based on the Transformer architecture (39) have been shown to achieve state-of-the-art accuracy in domains such as natural language processing (9; 3; 45), vision (10; 19), and recommender systems (36; 18). The Transformer is based on the attention mechanism, where it calculates context-dependent weights for a given input (32). Given an input consisting of a sequence of symbols (tokens), the Transformer is able to capture complex dependencies, but it is unclear how they are represented in the Transformer nor how to extract them.

In this paper we bridge between structural causal models and attention mechanism used in the Transformer architecture. We show that self-attention is a mechanism that estimates a linear structural equation model in the deepest layer for each input sequence independently, and show that it represents a causal structure over the symbols in the input sequence. In addition, we show that an equivalence class of the causal structure over the input can be learned solely from the Transformer's estimated attention matrix. This enables learning the causal structure over a single input sequence, using existing constraint-based algorithms, and utilizing existing pre-trained Transformer models for zero-shot causal discovery. We demonstrate this method by providing causal explanations for the outcomes of Transformers in two tasks: sentiment classification (NLP) and recommendation.

2 Related Work

In recent years, several advances in causal reasoning using deep neural networks were presented. One line of work is causal modeling with neural networks (42; 12). Here, a neural network architecture follows a given causal graph structure. This neural network architecture is constructed to model the joint distribution over observed variables (12) or to explicitly learn the deterministic functions of a structural causal model (42). It was recently shown that a neural network architecture that explicitly learns the deterministic functions of an SCM can be used for answering interventional and counterfactual queries (43). That is, inference in rungs 2 and 3 in the ladder of causation (26), also called layers 2 and 3 in the Pearl causal hierarchy (2). Nevertheless, this architecture requires knowing the causal graph structure before training, and supports a single causal graph.

In another line of work, an attention mechanism is added before training, and a causal graph is inferred from the attention values at inference (21; 17). Attention values are compared against a threshold, and high values are treated as indication for a potential causal relation. Nauta et al. (21) validate these potential causal relations by permuting the values of the potential cause and measuring the effect. However, this method is suitable only if a single causal model underlay the dataset.

3 Preliminaries

First, we provide preliminaries for structural causal models and the attention mechanism in the Transformer-based neural architecture. Throughout the paper we denote vectors and sets with bolditalic upper-case (e.g., V ), matrices with bold upper-case (e.g., A ), random variables with italic upper-case (e.g., X ), and models in calligraphic font (e.g., M ). Different sets of letters are used when referring to structural causal models and attention (see Appendix A-Table 1 for a list).

3.1 Structural Causal Models

A structural causal model (SCM) represents causal mechanisms in a domain (24; 35; 28). An SCM is a tuple { U , X , F , P ( U ) } , where U = { U 1 , . . . , U m } is a set of latent exogenous random variables, X = { X 1 , . . . , X n } is a set of endogenous random variables, F = { f 1 , . . . , f n } is a set of deterministic functions describing the values X given their direct causes, and P ( U ) is the distribution over U . Each endogenous variable X i has exactly one unique exogenous cause U i ( m = n ). The value of an endogenous variable X i , ∀ i ∈ [1 , . . . , n ] is

(the left-arrow indicates an assignment due to cause-effect relation), where Pa i is the set of direct causes (parents in the causal graph) of X i . A graph G corresponding to the SCM consists of a node for each variable, and a directed edge for each direct cause-and-effect relation, as evident from F . If the graph is directed and acyclic (DAG) then the SCM is called recursive .

In a special subclass of SCMs, referred in this paper linear-Gaussian SCM, a node value is a linear combination of its parents, and exogenous nodes are independently and normally distributed additive noise. Let G be a weight matrix, where G ( i, j ) is the weight of parent node X j determining the child node X i and G ( i, k ) = 0 if X k is not a parent of X i , and Λ be a diagonal matrix, where Λ ( i, i ) is the weight of exogenous node U i . Then, ∀ i ∈ [1 , . . . , n ]

and in matrix form X = G X + Λ U . In the case of a recursive SCM, with nodes sorted topologically (ancestors before their descendants), the weight matrix G is strictly lower triangular (zeros on the diagonal). In this paper we learn the causal graph, described by the non-zero elements of G , using constraint-based causal discovery approach, which generally requires assuming the following.

Definition 1 (Causal Markov) . In a causally Markov graph, a variable is independent of all other variables, except its effects, conditional on all its direct causes.

Definition 2 (Faithfulness) . A distribution is faithful to a graph if and only if every independence relation true in the distribution is entailed by the graph.

3.2 Self-Attention Mechanism

Attention is a mechanism that predicts context dependent weights (32) for the input sequence. In this paper we analyse the attention mechanism used by Vaswani et al. (39). Here, the input is a matrix Y where row vector Y ( i, · ) is the embedding of symbol s i in an input sequence s = [ s 1 , . . . , s n ] . The attention matrix is calculated by A = softmax ( YW QK Y ⊤ ) , such that the rows of A sum to 1 1 . Values matrix is computed by V = YW V , where row V ( i, · ) is the value vector of input symbol s i . We treat V as a set of projection of Y on column vectors W V . Finally, the output embeddings are Z = AV , where the i -th output embedding Z i is the i -th row of Z .

4 A Link between Pre-trained Self-Attention and a Causal Graph underlying an Input Sequence of Symbols

In this section we first describe self-attention as a mechanism that encodes correlations between symbols in an unknown structural causal model (Section 4.1). After establishing this relation, we present a method for recovering an equivalence class of the causal graph underlying the input sequence (Section 4.2). We also extend the results to the multi-head and multi-layer architecture (Section 4.3).

4.1 Self-Attention as Correlations between Nodes in a Structural Causal Model

We now describe how self-attention can be viewed as a mechanism that estimates the values of observed nodes of an SCM. We first show that the covariance over the outputs of self-attention is similar to the covariance over observed nodes of an SCM. Specifically, we model relations between symbols in an input sequence using a linear-Gaussian SCM at the output of the attention layer. In an SCM, the values over endogenous nodes, in matrix form, are X = G X + Λ U , which means

Since G is a strictly lower-triangular weight matrix, ( I -G ) -1 is a lower unitriangular matrix, which is equal to the sum of a geometric series

Thus, the ( i, j ) element represents the sum of directed-paths' weights, for all directed paths from X j to X i having length up to n -1 . The weight of a directed path is the product of the weights of edges along the path. In case some of the nodes are latent confounders, then the matrix in Equation 4 may not be triangular, due to spurious associations. Equation 3 represents a system with input U , output X and weights ( I -G ) -1 Λ . The covariance matrix of the output is

where µ X = ( I -G ) -1 Λ µ U , and C U is the covariance matrix of exogenous variables U .

An attention layer in a Transformer architecture, estimates an attention matrix A and a values matrix V from embeddings Y of an input sequence of symbols s = [ s 1 , . . . , s n ] ⊤ . The output embeddings is calculated by Z = AV . It is important to note that during self-attention training, an inductive bias is used in the form of learning separate representations for the attention matrix A and values matrix V . It can be seen that the j -th element of the i -th row-vector, V ( i, j ) is the projection of the symbol s i input embedding Y ( i, · ) on column vector W V ( · , j ) . In other words, an embedding is projected on the direction of vector W V ( · , j ) . Thus, each column of V can be seen as a projection of the input embeddings. Moreover, the attention matrix is shared with all projections for calculating the output embeddings. That is, for the j -th projection the output embedding is

Denote Z j ≡ Z ( · , j ) and V j ≡ V ( · , j ) corresponding to projection j . Covariance of Z j is

where C V j is the covariance matrix of input embeddings projected on vector W V ( · , j ) . Since columns of V describe different projections corresponding to independent columns of W V , and from the central limit theorem it follows that C V → I , which leads to

How does attention matrix A and a values vector V j translate into a structural causal model? We interpret the attention matrix A and values vector V as components of an SCM, such that the covariance matrix of its endogenous variables is equal to the covariance matrix of the attention layer's output, C X = C Z . From Equation 5 and Equation 8,

The attention matrix A measures the total effect each variable has on another variable, for any projection V ( · , j ) of the input embeddings. Similarly, matrix ( I -G ) -1 Λ in an SCM measures the effect one variable has on another variable through all directed paths in the causal graph (Equation 4) for any value of the exogenous variables U . Thus, form Equation 9 matrix A is analogous to ( I -G ) -1 Λ and V ( · , j ) to value assignment for U , which serves as context. Both A and ( I -G ) -1 Λ are not required to be triangular as latent confounders may introduce spurious associations, such that variables might be required to estimate the values of variables earlier in the topological order. In addition, the symbols in the sequence S are not assumed to be topologically ordered.

4.2 Attention-based Causal-Discovery (ABCD)

In Section 4.1 we show that self-attention learns to represent pairwise associations between symbols of an input sequence, for which there exists a linear-Gaussian SCM that has the exact same pair-wise associations between its nodes. However, can we recover the causal structure G of the SCM solely from the weights of a pre-trained attention layer? We now present the Attention-Based CausalDiscovery (ABCD) method for learning an equivalence class of the causal graph that underlies a given input sequence.

Causal discovery (causal structure learning) from observed data alone requires placing certain assumptions. Here we assume the causal Markov (24) and faithfulness (35) assumptions. Under these assumptions, constraint-based methods use tests of conditional independence (CI-tests) to learn the causal structure (27; 35; 8; 7; 30; 31; 22). A statistical CI-test is used for deciding if two variables are statistically independent conditioned on a set of variables. Commonly, partial correlation is used for CI-testing between continuous, normally distributed variables with linear relations. This test requires only a pair-wise correlation matrix (marginal dependence) for evaluating partial correlations (conditional dependence). We evaluate the correlation matrix from the attention matrix. From Equation 8 the covariance matrix of output embeddings is C Z = AA ⊤ , and (pairwise) correlation coefficients are ρ i,j = C Z ( i, j ) / √ C Z ( i, i ) C Z ( j, j ) . Unlike kernel-based CI tests (1; 11; 13; 14; 37; 48), we do not need to explicitly define or estimate the kernel, as it is readily

available by a single forward-pass of the input sequence in the Transformer. This implies the following. Firstly, our CI-testing function is inherently learned during the training stage of a Transformer, by that enjoying the efficiency in learning complex models from large datasets. Secondly, since attention is computed for each input sequence uniquely, CI-testing is unique to that specific sequence. That is, conditional independence is tested under the specific context of the input sequence.

Finally, the learned causal graph represents an equivalence class in the form of a partial ancestral graph (PAG) (29; 47), which can also encode the presence of latent confounders. A PAG represents a set of causal graphs that cannot be refuted given the data. There are three types of edge-marks (at some node X ): an arrow-head '> X ', a tail '-X ', and circle '-o X ' which represent an edge-mark that cannot be determined given the data. Note that reasoning from a PAG is consistent with every member in the equivalence class it represents.

What is the relation between the equivalence class learned by ABCD and the causal graph underlying the input sequence s = [ s 1 , . . . , s n ] ?

Definition 3 (SCM-consistent probability-space) . Let Ω n = { [ s 1 , . . . , s n ] : s i ∈ S} be a sample space, where S is a countable set of symbols. Let F n be an event space, and P n be a probability measure. For a finite n ∈ N , the probability space (Ω n , F n , P n ) is called SCM-consistent if every sequence s ∈ Ω n , was generated by a corresponding SCM with graph G s .

Assumption 1. Attention mechanism M is trained on sequences sampled from an SCM-consistent probability space, such that each input symbol is uniquely represented by its output embedding.

Many training methods, such as MLM (9), satisfy Assumption 1 as they train the network to predict input symbols using their corresponding output embeddings in the deepest layer. This assumption is required for relating the causal graph learned from the deepest attention layer to the causal graph underlying the input sequence.

Theorem 1. If s is a sequence from an SCM-consistent probability space used to train M , G s is the causal graph of the SCM from which s was generated, and A the attention matrix for input sequence s in the deepest layer, then G s is in the equivalence class learned by a sound constraint-based algorithm that uses covariance matrix C Z = AA ⊤ for partial correlation CI-testing.

Corollary 1. Nodes X i and X j are d-separated in G Z conditioned on X ′ ⊂ X if and only if symbols s i and s j are d-separated in G s conditioned on s ′ , a subset containing symbols corresponding to X ′ .

Corollary 1 is useful for cases in which the graph over the input symbols G s , as well as the structural equation model described by G Z , does not have a causal meaning. In these cases, one can reason about conditional independence relations among input symbols.

4.3 Multiple Attention Layers and Multi-Head Attention as a Structural Causal Model

Commonly, the encoder module in the Transformer architecture (39) consists of multiple self attention layers executed sequentially. A non-linear transformation is applied to the the output embeddings of one attention layer before feeding it as input to the next attention layer. The non-linear transformation is applied to each symbol independently. In addition each attention layer consists of multiple selfattention heads, processing the input in parallel using head-specific attention matrix. The output embedding of each symbol is then linearly combined along the heads. It is important to note that embedding of each symbol is processed independently of embeddings of other symbols. The only part in which an embedding of one symbol is influenced by embeddings of other symbol is the multiplication by the attentnion matrix.

Thus, a multi-layer, multi-head, architecture can be viewed as a deep graphical model, where exogenous nodes of an SCM are estimated by a previous attention layer. The different heads in an attention layer learn different graphs for the same input, where their output embeddings are linearly combined. This can be viewed as a mixture model. We recover the causal graph from the last (deepest) attention layer, and treat earlier layers as context estimation, which is encoded in the values of the exogenous nodes (Figure 1).

𝑋

!

𝑋

"

𝑋

#

𝑋

$

𝑛𝑜𝑛𝑙𝑖𝑛𝑒𝑎𝑟

𝑛𝑜𝑛𝑙𝑖𝑛𝑒𝑎𝑟

deep	context	representation

𝑙𝑖𝑛𝑒𝑎𝑟

𝑙𝑖𝑛𝑒𝑎𝑟

𝑙𝑖𝑛𝑒𝑎𝑟

𝐜𝐚𝐮𝐬𝐚𝐥	𝐠𝐫𝐚𝐩𝐡

transformation	type

:

attention	layer	0

attention	layer	1

attention	layer	2

4.4 Limitations of ABCD

The presented method, ABCD, is susceptible to two sources of error affecting its accuracy. The first is prediction errors of the Transformer neural network, and the second is errors from the constraintbased causal discovery algorithm that uses the covariance matrix calculated in the Transformer. The presence of errors from the second source depends on the first. For a Transformer with perfect generalization accuracy, and when causal Markov and faithfulness assumptions are not violated, the second source of errors vanishes, and a perfect result is returned by the presented ABCD method.

From Assumption 1, the deepest attention layer is expected to produce embeddings for each input symbol, such that the input symbol can be correctly predicted from its corresponding embedding (one-to-one embedding). However, except for trivial cases, generalization accuracy is not perfect. Thus, the attention matrix of an input sequence, for which the Transformer fails to predict the desired outcome, might include errors. As a result, the values of correlation coefficients calculated from the attention matrix might include errors.

Constraint-based algorithms for causal discovery are generally proved to be sound and complete when a perfect CI-test is used. However, if a CI-test returns an erroneous result, the constraintbased algorithm might introduce additional errors. This notion is called stability (35), which is informally measured by the number of output errors as a function of the number of CI-tests errors. In fact, constraint-based algorithm differ from one another in their ability to handle CI-test errors and minimize their effect. Thus, inaccurate correlation coefficients used for CI-testing might lead to erroneous independence relations, which in turn may lead to errors in the output graph.

We expect that as Transformer models become more accurate with larger training datasets, the accuracy of ABCD will increase. In this paper's experiments we use existing pre-trained Transformers and common datasets, and use the ICD algorithm (30) that was shown to have state-of-the-art accuracy.

5 ABCD with Application To Explaining Predictions

One possible application of the proposed ABCD approach is to reason about predictions by generating causal explanations from pre-trained self-attention models such as BERT (9). Specifically, we seek an explaining set that consists of a subset of the input symbols that are claimed to be solely responsible for the prediction. That is, if the effect of the explaining set on the prediction is masked, a different (alternative) prediction is provided by the neural network.

It was previously claimed that attention cannot be used for explanation (16); however, in a contradicting paper (41), it was shown that explainability is task dependent. A common approach is to use the attention matrix to learn about input-output relations for providing explanations (33; 5; 4). These often rely on the assumption that inputs having high attention values with the class token influence the output (44; 6). We claim that this assumption considers only marginal statistical dependence and ignores conditional independence and explaining-away that may arise due to latent confounders.

To this end, we propose CLEANN ( C ausa L E xplanations from A ttention in N eural N etworks). A description of this algorithm is detailed in Appendix B, and an overview with application to explaining movie recommendation is given in Figure 4. See also Nisimov et al. (23) for more details.

6 Empirical Evaluation

In this section we demonstrate how a causal graph constructed from a self-attention matrix in a Transformer based model can be used to explain which specific symbols in an input sequence are the causes of the Transformer output. We experiment on the tasks of sentiment classification, which classifies an input sequence, and recommendation systems, which generates a candidate list of recommended symbols (topk ) for the next item. For both experiments in this section we compare our method against two baselines from (38): (1) Pure-Attention algorithm (Pure-Atten.), that uses the attention weights directly in a hill-climbing search to suggest an explaining set, (2) Smart-Attention algorithm (Smart-Attn.), that adapts Pure-Attention , where a symbol is added to the explanation only if it reduces the score gap between the original prediction's score and the second-ranked prediction's score. Implementation tools are in https://github.com/IntelLabs/causality-lab .

6.1 Evaluation metrics

We evaluate the quality of inferred explanations using two metrics. One measures minimality of the explanation, and the other measures how specific is the explanation to the prediction.

6.1.1 Minimal explaining set

An important requirement is having the explaining set minimal in the number of symbols (40). The reason for that is twofold. (1) It is more complicated and less interpretable for humans to grasp the interactions and interplay in a set that contains many explaining symbols. (2) In the spirit of occum's razor, the explaining set should not include symbols that do not contribute to explaining the prediction (when faced with a few possibles explanations, the simpler one is the one most likely to be true). Including redundant symbols in the explaining set might result in a wrong alternative predictions when the effect of this set is masked.

6.1.2 Influence of the explaining set on replacement prediction

The following metric is applicable only to tasks that produce multiple predictions, such as multiple recommendations by a recommender system (e.g., which movie to watch next), as opposed to binary classification (e.g., sentiment) of an input. Given an input sequence consisting of symbols, s = { s 1 , . . . , s n } , a neural network suggests ˜ s n +1 , the 1 st symbol in the topk candidate list. CLEANN finds the smallest explaining set within s that influenced the selection of ˜ s n +1 . As a consequence, discarding this explaining set from that sequence should prevent that ˜ s n +1 from being selected again, and instead a new symbol should be selected in replacement (replacement symbol). Optimally, the explaining set should influence the rank of only that 1 st symbol (should be downgraded), but not the ranks of the other candidates in the topk list. This requirement implies that the explaining set is unique and important for the isolation and counterfactual explanation of only that 1 st symbol, whereas the other symbols in the original topk list remain unaffected, for the most part. It is therefore desirable that after discarding the explaining set from the sequence, the new replacement symbol would be one of the original (i.e. before discarding the explaining set) topk ranked symbols, optimally the 2 nd .

6.2 CLEANN for Sentiment classification

We exemplify the explainability provided by our method for the task of sentiment classification of movie reviews from IMDB (20) using a pre-trained BERT model (9) that was fine-tuned for the task. The goal is to propose an explanation to the classification of a review by finding the smallest explaining set of word tokens (symbols) within the review. Figure 2 exemplifies the explanation of an input review that was classified as negative by the sentiment classification model. We compare between our method and two other baselines on finding a minimal explaining set for the classification. We see that both baselines found the word 'bad' as a plausible explanation for the negative sentiment classification, however in addition, each of them finds an additional explaining word ('pizza', 'cinema') that clearly has no influence on the sentiment in the context it appears in the review. Contrary to that, our method finds the words 'bad' and 'but' as explaining the negative-review classification. Indeed, the word 'but' may serve as a plausible explanation for the negative review at the 2 nd part of the sentence, since it negates the positive 1 st part of the sentence (about the pizza). Additionally, Figure 2(b) shows

the corresponding attention map at the last attention layer of the model for the given input review. In addition, in Figure 2(c) we present the corresponding matrix representation of the learned graph (PAG). We can see that despite high attention values of the class token [cls] with some of the word tokens in the review (in the first row of the attention map), some of these words are found not to be directly connected to the class token in the causal graph produced by our method, and therefore may not be associated as influencing the outcome of the model.

Table 1 and Figure 3 show how the length of a review influences the explaining set size. CLEANN produces the smallest explaining sets on average, where statistical significance was tested using Wilcoxon signed-ranks test at significance level α = 0 . 01 . For an increasing review length it is evident that the two baselines increase their explaining set, correspondingly. Even if the additional explaining tokens are of negative/positive context, they are not the most essential and prominent word token of this kind. Contrary to that, the explaining set size produced by our method is relatively stable, with a moderate increase in the explaining set size, meaning that it keeps finding the most influential words in a sentence that dominates the decision put forward by the model.

I liked the pizza in this cinema but the movie I saw was bad.

𝐶𝐿𝑆

I liked the pizza in this cinema but the movie I saw was bad.

I liked the pizza in this cinema but the movie I saw was bad.

CLEANN

Pure

Attention

Smart

Attention

(b)

(a)

(c)

0

5

10

15

20

explanation set size

review IDs sorted by their explanation set size

Pure-Atten.

Smart-Atten.

CLEANN

0

5

10

15

20

25

30

explanation set size

review IDs sorted by their explanation set size

Pure-Atten.

0

5

10

15

20

25

30

35

40

explanation set size

review IDs sorted by their explanation set size

Pure-Atten.

0

10

20

30

40

50

60

explanation set size

review IDs sorted by their explanation set size

Pure-Atten.

(a)

(b)

(c)

(d)

𝑠

!

𝑠

"

𝑠

#

𝑠

$

𝑠̃

%

H

1

H

2

Causal	Discovery

ℳ

𝑠

!

𝑠

#

𝑠

&

𝑠̃

&'!

Abductive	Inference

attention

(b)

[mask]

Recommendation

ℳ

𝑠

!

𝑠

#

𝑠

&

𝑠̃

&'!

(a)

⋮

⋮

⋮

⋮

⋮

⋮

⋮

⋮

(c)

Star Wars:

Empire Strikes

Back

Jurassic

Park (1993)

Star Wars:

Episode IV - A

New Hope (1977)

Predator

(1987)

E.T. the Extra-

Terrestrial

(1982)

I recommended:

Star Wars: Episode V - The Empire

Strikes Back

Because you watched:

Star Wars: Episode IV - A New

Hope (1977)

If you would not have watched it, I

would have recommended:

Indiana Jones and the Last

Crusade (1989)

Because you watched:

Jurassic Park (1993)

(d)

Explain

6.3 CLEANN for recommendation system

We assume that the human decision process, for selecting which items to interact with, consists of multiple decision pathways that may diverge and merge over time. Moreover, they may be influenced by latent confounders along this process. Formally, we assume that the decision process can be modeled by a causal DAG consisting of observed and latent variables. Here, the observed variables are user-item interactions { s 1 , . . . , s n } in a session S , and latent variables { H 1 , H 2 , . . . } represent unmeasured influences on the user's decision to interact with a specific item. Examples for such unmeasured influences are user intent and previous recommendation slates presented to the user.

We exemplify the explainability provided by our method for the task of a recommendation system, and suggest it as a means for understanding of the complex human decision-making process when interacting with the recommender system. Using an imaginary session that includes the recommendation, we extract an explanation set from the causal graph. To validate this set, we remove it from the original session and feed the modified session into the recommender, resulting in an alternative recommendation that, in turn, can also be explained in a similar manner. Figure 4 provides an overview of our approach.

For empirical evaluation, we use the BERT4Rec recommender (36), pre-trained on the MovieLens 1M dataset (15) and estimate several measures to evaluate the quality of reasoned explanations.

Minimal explaining set: Figure 5(a) compares the explaining set size for the various sessions produced by CLEANN and the baseline methods. It is evident that the set sizes found by CLEANN are smaller. Figure 5(b) shows the difference between the explaining set sizes found by the baseline method and CLEANN, as calculated for each session, individually. Approximately 25% of the sessions are with positive values, indicating smaller set sizes for CLEANN, zero values shows equality between the two, and only 10% of the sessions are with negative values, indicating smaller set sizes for Pure-Attention.

Influence of the explaining set on replacement recommendation: Figure 6(a) compares the distribution for positions of replacement recommendations, produced by CLEANN and the baseline methods, in the original top-5 recommendations list. It is evident that compared to the baseline methods, CLEANN recommends replacements that were ranked higher (lower position) in the original top-5 recommendations list. In a different view, Figure 6(b) shows the relative gain in the number of sessions for each position, achieved by CLEANN compared to the baseline methods. There is a trend line indicating higher gains for CLEANN at lower positions. That is, the replacements are more aligned with the original top-5 recommendations. CLEANN is able to isolate a minimal explaining set that influences mainly the 1 st item from the original recommendations list.

(a)

0

5

10

15

20

25

explanation set size

session IDs sorted by their explanation size

Pure-Atten.

Smart-Atten.

CLEANN

(b)

-5

0

5

10

15

20

25

explanation size difference:

session IDs sorted by explanation size difference

Pure-Atten.

Smart-Atten.

(a)

0

8

16

24

32

2

3

4

5

percentage of sessions (%)

position of replacement recommendation

Pure-Atten.

Smart-Atten.

CLEANN

(b)

0

0.1

0.2

0.3

0.4

0.5

0.6

2

3

4

5

change percentage in counts

position of replacement recommendation

Pure-Atten.

Smart-atten.

7 Discussion

We presented a relation between the self-attention mechanism used in the Transformer neural architecture and the structural causal model often used to describe the data-generating mechanism.

One result from this relation is that, under certain assumptions, a causal graph can be learned for a single input sequence (ABCD). This can be viewed as utilizing pre-trained models for zero-shot causal-discovery . An interesting insight is that the only source of errors while learning the causal structure is in the estimation of the attention matrix. Estimation is learned during pre-training the Transformer. Since in recent years it was shown that Transformer models scale well with model and training data sizes, we expect the estimation of the attention matrix to be more accurate as the pre-training data increases, thereby improving the accuracy of causal discovery.

Another result is that the causal structure learned for an input sequence can be employed to reason about the prediction for that sequence by providing causal explanations (CLEANN). We expect learned causal graphs to be able to answer a myriad of causal queries (24), among these are personalized queries that can allow a richer set of predictions. For example, in recommender systems, assuming that the human decision process consists of multiple pathways that merge and split, by identifying independent causal pathways, recommendations can be provided for each one, independently.

The results of this paper contribute to the fields of causal inference and representation learning by providing a relation that bridges concepts from these two domains. For example, they may lead to 1) causal reasoning in applications for which large-scale pre-trained models or unlabeled data exist, and 2) architectural modifications to the Transformer to alleviate causal inference.

References

Francis R Bach and Michael I Jordan. Kernel independent component analysis. Journal of machine learning research , 3(Jul):1-48, 2002.

Elias Bareinboim, Juan D Correa, Duligur Ibeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl , pages 507-556. 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. Neural attentional rating regression with review-level explanations. In Proceedings of the 2018 World Wide Web Conference , pages 1583-1592, 2018.

Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 765-774, 2019.

Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems , 29, 2016.

Tom Claassen, Joris M Mooij, and Tom Heskes. Learning sparse causal models is not NP-hard. In Uncertainty in Artificial Intelligence , page 172. Citeseer, 2013.

Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics , pages 294-321, 2012.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Kenji Fukumizu, Francis R Bach, and Michael I Jordan. Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research , 5(Jan): 73-99, 2004.

Olivier Goudet, Diviyan Kalainathan, Philippe Caillou, Isabelle Guyon, David Lopez-Paz, and Michele Sebag. Learning functional causal models with generative neural networks. Explainable and interpretable models in computer vision and machine learning , pages 39-80, 2018.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77. Springer, 2005.

Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, and Bernhard Schölkopf. Kernel methods for measuring independence. Journal of Machine Learning Research , 6: 2075-2129, 2005.

F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS) , 5(4), dec 2015. ISSN 2160-6455. doi: 10.1145/2827872. URL https://doi.org/10.1145/2827872 .

Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 3543-3556, 2019.

Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Structural agnostic modeling: Adversarial learning of causal graphs. Journal of Machine Learning Research , 23(219):1-62, 2022.

Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM) , pages 197-206. IEEE, 2018.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .

Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention-based convolutional neural networks. Machine Learning and Knowledge Extraction , 1(1):312-340, 2019.

Shami Nisimov, Yaniv Gurwicz, Raanan Yehezkel Rohekar, and Gal Novik. Improving efficiency and accuracy of causal discovery using a hierarchical wrapper. In Uncertainty in Artificial Intelligence (UAI 2021), workshop on Tractable Probabilistic Modeling , 2021.

Shami Nisimov, Raanan Y Rohekar, Yaniv Gurwicz, Guy Koren, and Gal Novik. CLEAR: Causal explanations from attention in neural recommenders. Conference on Recommender Systems (RecSys), workshop on Causality, Counterfactuals and Sequential Decision-Making (CONSEQUENCES) , 2022.

Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge university press, second edition, 2009.

Judea Pearl. An introduction to causal inference. The international journal of biostatistics , 6(2), 2010.

Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect . Basic books, 2018.

Judea Pearl and Thomas Verma. A theory of inferred causation. In International Conference on Principles of Knowledge Representation and Reasoning , pages 441-452, 1991.

J. Peters, D. Janzing, and B. Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms . MIT Press, Cambridge, MA, USA, 2017.

Thomas Richardson and Peter Spirtes. Ancestral graph markov models. The Annals of Statistics , 30(4):962-1030, 2002.

Raanan Y Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik. Iterative causal discovery in the possible presence of latent confounders and selection bias. Advances in Neural Information Processing Systems , 34:2454-2465, 2021.

Raanan Yehezkel Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik. From temporal to contemporaneous iterative causal discovery in the presence of latent confounders. In International Conference on Machine Learning , pages 39939-39950. PMLR, 2023.

Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4(1):131-139, 1992.

Sungyong Seo, Jing Huang, Hao Yang, and Yan Liu. Interpretable convolutional neural networks with dual local and global attention for review rating prediction. In Proceedings of the eleventh ACM conference on recommender systems , pages 297-305, 2017.

Peter Spirtes. Introduction to causal inference. Journal of Machine Learning Research , 11 (May):1643-1662, 2010.

Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction and Search . MIT Press, 2nd edition, 2000.

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management , pages 1441-1450, 2019.

Xiaohai Sun, Dominik Janzing, Bernhard Schölkopf, and Kenji Fukumizu. A kernel-based causal learning algorithm. In Proceedings of the 24th international conference on Machine learning , pages 855-862, 2007.

Khanh Hiep Tran, Azin Ghazimatin, and Rishiraj Saha Roy. Counterfactual explanations for neural recommenders. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1627-1631, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. 2017. doi: 10.48550/ARXIV.1711. 00399. URL https://arxiv.org/abs/1711.00399 .

Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 11-20, 2019.

Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems , 34:10823-10836, 2021.

Kevin Xia, Yushu Pan, and Elias Bareinboim. Causal models for counterfactual identification and estimation. In Proceedings of the 11th Eleventh International Conference on Learning Representations, forthcoming , 2023.

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning , pages 2048-2057. PMLR, 2015.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , 32, 2019.

Raanan Yehezkel and Boaz Lerner. Bayesian network structure learning by recursive autonomy identification. Journal of Machine Learning Research , 10(7), 2009.

Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence , 172(16-17):1873-1896, 2008.

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , pages 804-813, 2011.

A Notations

In this section we provide Table 2, which describes the main symbols used in the paper.

B Finding Possible Explanations from a Causal Graph

This appendix includes the pseudo-code for the CLEANN ( C ausa L E xplanations from A ttention in N eural N etworks) algorithm and explanatory figures. By abduction, we replace the masked symbol with a predicted one, creating an imaginary sequence. We then feed-forward this sequence and extract the resulting attention matrix (Algorithm 1-lines 2-4). Using this attention matrix we learn a causal graph as discussed in Section 3.2 (Algorithm 1-lines 5-7).

Given a causal graph, various ' why ' questions can be answered (26). We follow (38) explaining a prediction using the symbols from the input sequence. That is, provide the minimal set of symbols that led to a specific prediction and provide an alternative prediction. To this end, we define the following.

Definition 1 (PI-path) . A potential influence path from A to Z in PAG G , is a path Π( A,Z ) = ⟨ A,. . . , Z ⟩ , such that for every sub-path ⟨ U, V, W ⟩ of Π( A,Z ) , where there are arrow heads into V and there is no edge between U and W in G .

Essentially, a PI-path ensures dependence between its two end points when conditioned on every node on the path (a single edge is a PI-path).

Definition 2 (PI-tree) . A potential influence tree for A given causal PAG G , is a tree T rooted at A , such that there is a path from A to Z in T , ⟨ A,V 1 , . . . , V k , Z ⟩ , if and only if there is an PI-path ⟨ A,V 1 , . . . , V k , Z ⟩ in G .

Definition 3 (PI-set) . A set E is a potentially influencing set (PI-set) on item A , with respect to r , if and only if:

| E | = r

∀ E ∈ E there exists a PI-path, Π( A,E ) such that ∀ V ∈ Π( A,E ) , V ∈ E

∀ E ∈ E , E temporally precedes A .

An example for identifying PI-sets is given in Figure 7. Although a PI-set identify nodes that are conditionally dependent on the prediction, it may not be a minimal explanation for the specific prediction. That is, omitting certain symbols from the session alters the graph and may render other items independent of the prediction. Hence, we provide an iterative procedure ('FindExplanation' in Algorithm 1-lines 10-18), motivated by the ICD algorithm (30), to create multiple candidate explanations by gradually increasing the value of r . See an example in Figure 8. Note that from conditions (1) and (2) of Definition 3, r effectively represents the maximal search radius from the prediction (in the extreme case, the items of the PI-set lie on one PI-path starting at the prediction). The search terminates as soon as a set qualifies as an explanation (Algorithm 1-line 17). That is, as soon as a different prediction is provided. If no explanation is found (Algorithm 1-line 19) a higher value of α should be considered for the algorithm.

Input:

C Proofs

In this section we provide proofs for the results described in Section 4.2. Let (Ω n , F n , P n ) be an SCM-consistent probability space, and M trained on sequences from this probability space, complying with Assumption 1.

Theorem 1. If s is a sequence from an SCM-consistent probability space used to train M , G s is the causal graph of the SCM from which s was generated, and A the attention matrix for input sequence s in the deepest layer, then G s is in the equivalence class learned by a sound constraint-based algorithm that uses covariance matrix C Z = AA ⊤ for partial correlation CI-testing.

Proof. From Assumption 1 it follows that an output embedding, in the deepest attention layer, for each symbol is trained to predict the input symbol. It follows that the output embeddings of the deepest layer are one-to-one (injective) representations of the input symbols. Thus, the causal relations among the input symbols can be cast on their correponding output embeddings. That is, a causal graph over the output embeddings is similar to causal graph over the input symbols G Z = G s . In Section 4.1 we showed that self-attention in the Transformer architecture can be viewed as an SCM over the output embedding, and the covariance matrix, C Z , can be computed from the attention matrix (Equation 9). A conditional independence test between output embeddings can then be defined to calculate partial correlations from C Z . Finally, a sound constraint-based algorithm (35) that uses C Z for partial correlation CI-testing, recovers an equivalence class of G Z .

̸

𝐼

!

𝐼

"

𝐼

#

𝐼

$

𝐼 "

%

H

1

H

2

𝐼

!

𝐼

"

𝐼

#

𝐼

$

𝐼 "

%

H

1

H

2

𝐼

!

𝐼

"

𝐼

#

𝐼

$

𝐼 "

%

H

1

H

2

(b)

(c)

(a)

explaining

away

confounding

time

𝐼

!

𝐼

"

𝐼

#

𝐼

$

𝐼 "

%

(a)

𝐼 "

%

𝐼

!

𝐼

#

𝐼

"

1

1

2

2

3

3

search	radius

1

2

3

(b)

Corollary 2. Nodes X i and X j are d-separated in G Z conditioned on X ′ ⊂ X if and only if symbols s i and s j are d-separated in G s conditioned on s ′ , a subset containing symbols corresponding to X ′ .

Proof. This result follows directly from Theorem 1 and faithfulness assumption (Definition 2).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes a formal link between a (pre-trained) self-attention layer and a causal graph underlying a sequence of symbols. Specifically, it shows that self-attention learns to represent pairwise associations for which there exists a linear-Gaussian Structural Causal Model that has the exact same pair-wise associations between its nodes. The paper then proceeds to derive an algorithm for recovering the causal structure of the SCM solely from the weights of a pre-trained attention layer\n\nSTRENGTHS:\nThe Attention-Based Causal-Discovery (ABCD) method, which is developed for learning an equivalence class of the causal graph that underlies a given input sequence is reasonable. It relies on partial correlarion CI-testing, which is sound.\nThen the paper proceeds to apply ABCD to reasoning about predictions by generating causal explanations from pre-trained self-attention models, such as BERT (8). Specifically, the algorithm produces explanations through selecting a subset of the input symbols that are claimed to be solely responsible for BERT prediction. ABCD is shown empirically to be much stronger than the common approach of using the attention matrix to learn about input-output relations for providing explanations.\n\nWEAKNESSES:\nThe Attention-Based Causal-Discovery (ABCD) method relies on an unsupervised training assumption. This restricts applicability to only models of BERT type, you cannot deal e.g. with decoder-only models. \n\nIn addition, it is extremely troubling that the authors do not perform any comparative evaluation against standard methods in the field. They limit their examination to standard self-attention and a variant of it. Even worse, they have avoided use of any standard benchmark in the field. \n\nEventually, it is not clear how useable the generated causal explanations are, and how easy it is to exploit them in an application setting.\n\nQUESTIONS:\n1. I want to see a serious reconsideration of all the empirical evidence the paper provides. \n2. How can the causal explanations be used in an easy and/or automated setting?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe ABCD paper starts off by motivating the need for understanding the complex dependencies that underly current Transformer architectures, pointing to causality as the missing puzzle piece. Following brief introductions of both SCMs and Self-attention, the authors then provide covariance matrix derivations for both linear Gaussian SCM and attention such that they are able to equate them eventually. By introducing another assumption on how the attention was trained, they propose a theorem that provides a clear connection between causal discovery and attention in that one can make CI tests based on the attention matrix essentially. Finally, the authors propose a complete pipeline for going from e.g. a pre-trained Transformer model to causal explanations based on their ABCD approach computed from the last attention layer. As put by the authors, this can be considered as a \"one-shot\" causal discovery paradigm.\n\nSTRENGTHS:\n1. A unique perspective and valuable contribution to the causal discovery literature in that it expands the horizon on what kind of connections can be established between existing models to foster new understanding of said models. The derivation of the covariances and then equating them is both simple and effective, and then Thm.1 seems like a nice result for understanding the final correspondence between attention and CI-testing for constraint-based causal discovery.\n\n2. Presenting a complete pipeline is a great bonus (since it is a difficult task) and shows an immediate practical approach to the previously presented theory. The explanation approach is convincing and practically applicable since the theory justifies the use of the attention matrix.\n\nWEAKNESSES:\n1. While the causality assumptions Markov and faithfulness can be considered standard, as soon as moving to the realm of data that nowadays Transformers are confronted with we run the risk of violating them. Furthermore, the assumption of a linear Gaussian SCM is quite restrictive and should be highlighted more, especially under consideration of Transformers. Finally, Assumption 1 right before Theorem 1 is only vaguely comprehensible IMHO. I'd wish for the authors to be more specific about what property is required for the deepest attention layer such that the equating to CB-CD works. This could be done through formalization of that assumption or if not possible, then through an example.\n\n2. Thm.1 is incomplete writing-wise i.e., sentence cuts off at end of page 4. Starting from p.4 the paper generally lend the impression of being not as carefully written as the pages before. Please consider a careful readthrough post-submission.\n\n3. Missing discussion of explanation methods based on causal graphs. Two very different approaches include Causal Shapely Values by Heskes et al. 2020 and Causal Explanation of SCM by Zečević et al. 2021. The first is a rather classical approach, where numerical attributions are being computed and in Sec.5 they present an approach for causal chain graphs (a special case like your Linear-Gaussian case), and the second discusses a traversal algorithm for generating immediate textual representations based on more general SCM (although only showcased for a linear SCM setting).\n\nQUESTIONS:\n1. Can you comment on Theorem 1 as mentioned in the weaknesses section?\n\n2. The assumption of linear Gaussian SCM can severely limit the method. What was the reasoning behind the assumption? \n\n3. What property is required for the deepest attention layer such that the equating to CB-CD works.\n\nOverall, a nice contribution but not devoid of weaknesses. I will be willing to raise my score based on the rebuttal.\n\nLIMITATIONS:\nNo concerns here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In this work, authors, interpret self attention in transformers as linear gaussian SCM over output embeddings. 
SCM over input embeddings is same as SCM over output embeddings, since deepest attention layer is trained to predict input symbol.
Existing conditional independence based testing algorithm is applied on the weights of the deepest attention layer of the pretrained transformer in order to estimate causal structure. 
Furthermore, the authors demonstrate the applicability of the proposed approach in deducing causal explanations for attention based recommendations.

STRENGTHS:
Authors proposed novel way of viewing self attention in transformers as weight matrix of linear gaussian SCM
Pretrained model is used to obtain causal structure, which would be of great use to gain causal perspectives for existing model
Estimating causal structure from the attention layer and using it for causal explanantions is promising.

WEAKNESSES:
Theorem1, in main paper has few words missing, however, it's available in supplementart material

QUESTIONS:
In the paper, the covariance matrix of value is assumed to be identity using central limit theorem, however, in practice, it wouldnot be identity as a result the graph learnt wouldnot be faithful. This will act as one more reason of inaccuracy. Is it possible for you to throw some light on this ?

LIMITATIONS:
1) The accuracy of the proposed method as explained by the authors is dependent on the prediction of the transformers and the conditional independence testing accuracy
2) Authors assumed learnt graph will be faithful, since exogeneous variables V under central limit theorem will have identity covariance, however in practice the faithfullness assumption will be violated, this will act as one more reason of inaccuracy
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Attention-Based Causal Discovery (ABCD), a causal interpretation for the attention matrix of transformers (e.g., LLMs like BERT). The authors suggest CLEANN, an algorithmic approach that first finds a causal structure of the input symbols (e.g., tokens (words) of a sentence as input to BERT). This causal structure is unique to each input (sentence). This can then be used to get causal explanation for different taks (in their experiments classification or recommendation) based on the representations of the input sentences (e.g., the word \"but\" and \"bad\" in the input sentence caused the negative sentiment classification).\n\nSTRENGTHS:\nThank you for your submission. I enjoyed reading your paper. \n\nClarity & Presentation: I found the paper to be clearly structured and written. On clarity and presentation, I would like to positively highlight the introduction to each (sub) section gave an orientation that helped to clearly guide the reader, what the next chapter is about and how it fits in the bigger picture. The figures (e.g., Fig2a, Fig 4) helped a lot to understand the method/pipeline.\n\nNovelty: I found the causal interpretation and its application to causal explanation of predictions apealing and novel. Finding a causal explanation to downs-stream tasks of LLMs for example, can be very helpful. However, I am not an expert in the field of LLMs and I do not feel qualified to fully assess the novelty and hence the contribution of the work in the general field.\n\nEvaluation: I found the experimental section clearly structured and interesting to read.\n\nWEAKNESSES:\nI have a few comments for improvements. \n\n* Section 5, Table 1: What do the bold numbers indicate, what the +- (i.e., over how many runs did you do it, were these runs of CLEANN or different data samples)? This would be helpful to comment in the caption. Am I right in understanding tha CLEANN does yield lower average explanations, but the numbers suggest that it is not a significant? Over how many runs did you do it?  \n\n* Section 5, Figure 2 (b, c): I find it hard to understand and interpret Figure 2(b,c) despite your explanation in ln. 255ff. Also, the labels are very small and hard to read. \n\n* Section 5, Figure 3: I am missing an explanation/interpretation of Figure 3 in the text. Did I overlook it? \n\n* I am missing a related work section. \n\nMinor: \n* Section 1: While I enjoyed reading the empirical evaluation, I found the causal explaining application could be stronger highlighted in the introduction. \n\n* Section 2.1, line 67: While I appreciate the introduction to SCMs, I was a bit confused about the Causal Markov and Faithful properties at the end of the section. They seem very disconnected of the rest of the text. It would help, if there would be a short explanation, why they are stated / needed for the paper. \n\n* Section 2.2 is very brief. In the light of Section 3 (ln. 100 ff.), where the self-attention mechanisms is further detailed, it makes sense. However, it may be helpful to just point to that at the end of the section 2.2. \n\n* Section 5: Caption of Table 1 - by review length, you mean the length of the sentence, right? It would have helped me  to point that out. \n\n* Section 5: I find it weird that Fig. 6 is discussed before Fig. 5.  A natural way would be to discuss them the other way around.\n\n* I got a bit confused by the difference between ABCD and CLEANN. Could you kindly clarify?\n\nTypos:\n* ln 222 \"(1) In addition, it\" ->  \"(1) It\" \n* ln 262 \"influences on the \" -> \"influences the\"\n\nQUESTIONS:\nI have a few questions: \n\n* Section 2.1, ln. 50, Definition of an SCM. I have previously seen an SCM being defined often $M = (F  P(U))$, or $M = (X, U , F)$, but I have not seen both U or P(U). Is yours the proper formal definition? \n\n* Section 2.1, ln. 59, your understanding of a linear-Gaussian SCM is the same as an Additive Noise Model (ANM)? \n\n* Section 5, Fig. 6: I do not understand the x-axis. Could you explain this again? \n\n* Are there other applications of ABCD beyond providing causal explanations?\n\nLIMITATIONS:\nSection 6 Discussion:  ln. 305 mentions \"under certain assumptions\", but these assumptions are not re-stated here. It would be helpful to have them restated. In this regard, I am missing a discussion of the limitations regarding the (causal) assumptions. \n\nI am also wondering, how robust your causal explanations are? I believe, when providing causal explanations to individuals, it is important to discuss any limitations regarding the sensitivity to assumptions and to comment the potential negative societal impact, if assumptions are not met.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}