Causal Abstractions of Neural Networks
Authors: Atticus Geiger
Hanson Lu
Thomas F Icard
Christopher Potts
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural network analysis
interpretability
explainability
causality

Abstract: Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model’s causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that neural representations encode the compositional structure of MQNLI examples.

Causal Abstractions of Neural Networks
Atticus Geiger, Hanson Lu," Thomas Icard, and Christopher Potts
Stanford Stanford, CA 94305-2150
(atticusg, hansonlu, icard, @epotts)@stanfor..ed
Abstract
Structural analysis methods (e-g.. probing and feature attribution) are increasingly important tools for neural network analysis. We propose sis method grounded formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. this method, neural representations are aligned with variables interpretable causal models, and then interchange interventions mentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that constructed with tree-structured natural logic causal model. We discover that BERT-basee model with state-of-the-art performance successfully realizes parts of the natural logic model's causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.
structural analy-
used to experi-
Introduction
Explainability and interpretability have long been central issues for neural networks, and they have taken on renewed importance as such models are now ubiquitous in research and technology. Recent structural evaluation methods seek to reveal the internal structure of these "black box" models. Structural methods include probes, attributions (feature importance methods), and interventions (manipulations of model-internal states). These methods can complement standard behavioral techniques (e.g., performance on gold evaluation sets), and they can yield insights into how and why models make the predictions they do. However, these tools have their limitations, and it has often been assumed that more ambitious and systematic causal analysis of such models is beyond reach.
Although there is sense in which neural networks are "black boxes", they have the virtue of being completely closed and controlled systems. This means that standard empirical challenges of causal inference due to lack of observability simply do not arise. The challenge is rather identify high-level causal regularities that abstract away from irrelevant (but arbitrarily observable and manipulable) low-level details. Our contribution in this paper is to show that this challenge be met. Drawing on recent innovations in the formal theory of causal abstraction 2 22], we offer a methodology for meaningful causal explanations of neural network behavior.
Our methodology causal abstraction a consists of three stages. (1) Formulate a hypothesis by defining a causal model that might explain network behavior. Candidate causal models can be naturally adapted from theoretical and empirical modeling work in linguistics and cognitive sciences. (2) Search for an alignment between neural representations in the network and variables in the
"cqual contribution
We provide tools for causal abstraction analysis at http://thww..cco//caaao/h/aaaaaa and the code base for this paper at h/gp
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

high-level causal model. (3) Verify experimentally that the neural representations have the same causal properties as their aligned high-level variables using the interchange intervention method of Geiger et al. [I]]
As study, we apply this methodology to LSTM-based and BERT-based natural language inference (NLI) models trained on the logically complex Multiply Quantified NLI (MQNLI) dataset of Geiger et [TO] This challenging dataset was constructed with tree-structured natural logic causal model 17 29 14.. Our BERT-based model has the structure of a standard NLI classifier, and yet is able to perform well on MQNLI (8%%) a result Geiger et al. achieved only with highly customized task-specific models. By contrast, our LSTM-based model is much less successful (46%).
The obvious scientific question in this case study is what drives the success of the BERT-based model on this challenging task. To answer this we employ our methodology. (1) We formulate hypotheses by defining simplified variants of the natural logic causal model. (2) We search potential alignments between neural representations BERT and variables in our high-level causal models. (3) We perform interchange interventions on the BERT model for each alignment. We find that our BERT model partially realizes the causal structure of the natural logic causal model; crucially, the LSTM model does not. High-level causal explanation for system behavior is often considered gold standard for interpretability, one that may be thought quixotic for complex neural models [16]. The point of our case study is show that this high standard can be achieved.
We conclude by comparing our methodology to probing and the attribution method of integrated gradients [27]. We argue probing is unable provide causal characterization of models. We show formally that attribution methods do measure causal properties, and that way they are similar to the tool of interchange interventions. However, our methodology of causal abstraction analysis provides a framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure.
2 Related Work
Probes Probes are generally supervised models trained on the internal representations of networks with the goal of determining what those internal representations encode [3] 20 28 Probes are
fundamentally unable to directly measure causal properties of neural representations, and Ravichander al. (21. Elazar al. [9], and Geiger et al. [ have argued that probes are limited in their ability to provide even indirect evidence of causal properties.
We now present analytic example in which probing identifies seemingly crucial information in representations that have no causal impact on behavior. We assume the structure of the simple addition network N4 in Figure [] For our embedding, we simply map every integer in Ng to the 1 I-dimensional vector [i]. The weight matrices are
W2 (1
0 ( 0 ) w
)
i 0
W1
W3
1 0
The output for an input sequence x = (i,j, k) is given by
w.
In this network, xW, perfectly encodes i + j. and xWa perfectly encodes A. Thus, the identity model probe will be perfect in probing those representations for this information. However, neither representation plays causal role in the network behavior; only xW2 contributes to the output.
Attribution Methods Attribution methods aim to quantify the degree to which a network represen- tation contributes to the output prediction of the model, for specific example or set of examples 26 27 (3). In contrast to probing, the well known integrated gradients method (IG) can be 3 24
given unambiguous causal interpretation. Following [27] we define the vector IG(x), for input x relative to baseline b, to have ith component (x) given by the expression on the left:
(x; - be) FF(ax + (1-a)b) do = (x;: - bi)
F(x F(x") do
F(x
F
lim Ja=0==0 500
dx:
Abbreviating the weighted average ax + (1 - ab by xa, letting x be the vector that differs from x2 in that the ith coordinate is increased by e, and then expanding the definition of partial derivative, this can be written in the form given the right. The difference F(x*.*) - F(x") is known in the

---

S
W
X
The causal model C+ (right) that first computes S X and W Z. before computing the final output S2 W S1 aligned with the neural network N+ (left) with L1 highlighted as the hypothesized location encoding S X + Y and L2 as the location encoding
12
15
L
2
D1
D2
D2
D2
D2
(b) Low-level neural network interchange intervention. network processes two different input sequences. The neural representation created at L, for input se- quence (1, 2, 3) is replaced by the corresponding rep- resentation created for input sequence (4, 5, 6).
High-level symbolic computation interchange in- tervention. The computation processes two different input sequences. The sum at S₁ for input sequence (1, 2, 3) is replaced by the corresponding representa- tion created for input sequence (4, 6).
Figure I: Our motivating example where we hypothesis that a symbolic computation C++ is a causal abstraction of a neural network N under particular aligmment (top). We can experimentally confirm this hypothesis by conducting an interchange intervention on both the network and the computation with every pair of inputs and evaluating whether the intervened network and intervened computation have the same counterfactual output behavior. We schematically depict interchange intervention on the network N4 (bottom left) and the computation C+ (bottom right) with the base input (1, 2, and the source input (4, 5, 6). Observe that the output of the intervened neural network matches the output of the intervened symbolic computation, so we have success for this pair of inputs.
causal literature as the (individual) causal effect on the output (e.g., [5] of increasing neuron i by relative the fixed input x So, essentially, IG,(x) measuring the average "limiting" causal effect of increasing neuron along the straight line from the baseline vector to the input vector x, weighted by the difference at between input and baseline. More recently, Chattopadhyay al. [6] develop an attribution method that explicitly treats neural models as structured causal models and directly computes the individual causal effect of a feature determine its attribution.
Attribution methods can measure causal properties, and, in that way, they are similar to the tool of interchange interventions. However, framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure.
methodology of causal abstraction analysis provides
Causal Abstraction Our goal is to evaluate whether the internal structure of a neural network realizes an abstract causal process. To concretize this, turn to formal, broadly interventionist theories of causality [25] 19] in which causal processes are characterized by effects of interventions, where relationships between two causal processes
and theories of abstraction determined by the presence of systematic correspondences between the effects of interventions.
5 22
The notion of abstraction that we employ here is a relatively simple one called constructive abstraction [] Informally, high-level model a constructive abstraction of low-level model if there is a way to partition the variables in the low-level model where each high-level variable can be assigned to low-level partition cell, such that there is systematic correspondence between interventions on the low-level partition cells and interventions on the high-level variables.

---

There are two properties of constructive abstraction that make it ideal for neural network analysis. First, the information content of partition cells of low-level variables can be determined by the high-level variables that they correspond to. For neural networks, the partition cells of low-level variables are sets of neurons, and our method supports reasoning at the level of vector representations (sets of neurons). Second, the causal dependencies between partitions of low-level variables are not necessarily preserved as causal dependencies between the high-level variables corresponding to these partitions. For example, the low-level model might be fully connected neural network, whereas the high-level model might have much sparser connections. For neural network analysis, this means we find causal abstractions that have far simpler causal structures than the underlying neural networks. We provide an example in the next section.
3 Causal Abstraction Analysis of Neural Networks
We now deseribe our methodology in more detail, illustrating the relevant concepts with example of neural network performing basic arithmetic. Specifically, suppose that we have a neural network N_ that takes in three vector representations Dz: Dy: D= representing the integers 2, V, and 2, and outputs the sum of the three inputs: (Dz. Dy, Dz) = x + y + 2. We seek an informative causal explanation of this network's behavior.
Formulating Hypothesis A human performing this task might follow an algorithm in which they add together the first two numbers and then add that sum to the third number. We can hypothesize that the behavior of N+ is explained by this symbolic computation. Specifically, the network combines D2 and Dy to create an internal representation at some location Lı encoding x + y: it encodes at some location L2; and Ln and L2 are composed encode representation. This hypothesis is given schematically in Figure Ta
the location of the output
Following methodology, we first define the causal model C4 in Figure la
Our informal
hypothesis that neural network's ' behavior is explained by simple algorithm can then be restated more formally: C+ is constructive abstraction of the neural network N+.
Alignment Search Now that we have hypothesized that the causal model C+ is causal abstraction of the network NN, the next step is to align the neural representations in N+ with the variables in C4. The input embeddings Dz. Dy. and D2 must be aligned with the input variables X, Y, and Z and the output neuron O must be aligned with the output variable S2. That leaves the intermediate variables S1 and W to be aligned with neural representations some undetermined locations L1 and L2. If this were an actual experiment (see below), we would perform an alignment search to consider many possible values for Ln and L2. Each alignment is hypothesis about where the network N stores and uses the values of S₁ and W. For the example, we assume the alignment in Figure Ta
Interchange Interventions Finally, for a given alignment, we experimentally determine whether the neural representations Ln and L2 have the same causal properties S₁ and W. The basic experimental technique is an interchange intervention, in which a neural representation created during prediction on a "base" input is interchanged with the representation created for "source" input [II]. We now show informally that this method can be used to prove that the causal model C+ is constructive abstraction of the neural network N4 (Appendix G has formal details).
We first intervene on the causal model. Consider two inputs a, a' E (Ng)3 where Ng is the set of integers 0-9. Let a (I, u, 2) be the base input and a' (x', y 2') be the source input. Define
Si (a) = y
to be the output provided by C+2 when Si, the variable representing the intermediate sum, is intervened on and set to the value x' + y/. Thus, for example, if the base input is C++ C+(1, 21, = 6, and the source input is a' = (4, 6). then C. C (1,2,3) 3) = 4 + 5 + 3 = 12. This process is depicted in Figure Ic
Next, we intervene on the neural network N4. Let D be an embedding space that provides unique representations for Ng, and consider two inputs D (Dz, Dy. D.) and D' (Dz, Dy, D where all D; and Dr are drawn from D. In parallel with (I). define
+ (D)
(2)
4

---

to be the output provided by N+ processing the input D when the representation at location L1 is replaced with the representation location L, created when N_ processing the input D This process is depicted in Figure lb
With these two definitions, we can define what it means test the hypothesis that N+ computes I + y at position L1. Where Da is an embedding for a and Da is an embedding for a', we test:
C+
C. t- (a) = N+ - D2' (Da)
(3)
If this equality holds for all source and base inputs a and a'. then we conclude that, for every intervention on St, there is an equivalent intervention on L1_ If we can establish a corresponding claim for W and L2, then we have shown that C+ is constructive abstraction of N+, since the inputs' relationships are established by our embedding and there are no other interventions on C+ to test.
Analysis Suppose that all of our intervention experiments verify our hypothesis that C+ is constructive abstraction of N with variables S₁ and W aligned to neural representations at L1 and Lg. This explains network behavior by resolving two crucial questions.
First, we learn what information is encoded in the representations L1 and Lg. Neural representations encode the values of the high-level variables they aligned with. The location L1 encodes the variable S and the location L2 encodes the variable W. This is similar to what probing achieves. However, our method is crucially different from probing. In probing, information content is established through purely correlational properties, meaning a neural representation with no causal role in network behavior can be successfully probed, as showed in Section [ In causal abstraction analysis, information content is established through purely causal properties, ensuring that the neural representation actually implicated in model behavior.
Second, we learn what causal role L1 and L2 play in network behavior. Neural representations play a parallel causal role to their aligned high-level variables. At the location L1. D. and Dy are composed to form a neural representation with content x + that is then composed with L2 to create an output. tells us that while L1 depends on D3 and representations at L1
The fact that S₁ doesn't 't depend may even correlate with 2, the information about 2 is not causally represented at I.,. At the location T2, the value of is simply repeated and then composed with I, to create final output.
Our method assigns causally impactful information content, but also identifies the abstract causal structure along which representations are composed. It thus encompasses and improves on both correlational (probing) and attribution methods.
4 The Natural Language Inference Task and Models
The Multiply Quantified NLI (MQNLI) dataset of Geiger et al. [00 contains templatically generated English-language NLI examples that involve very complex interactions between quantifiers, negation, and modifiers. We provide few examples in Figure 2b the empty-string symbol e ensures perfect alignments at the token level both between premises and hypotheses and across all examples.
Multiply Quantified NLI Dataset
The MQNLI examples are labeled using an algorithmic implementation of the natural logic of MacCartney and Manning [18] over tree structures, and MQNLI has train/dev/test splits that vary in their difficulty. In the hardest setting, the train set provably the minimal set of examples required to ensure that the dev and test sets can be perfectly solved by simple symbolic model; in the easier settings, the train set redundantly encodes necessary information, which might allow a model to perform perfectly in assessment by memorization despite not having found a truly general solution. For a fuller review of the dataset, see Appendix
MQNLI is fitting benchmark given our goals for few reasons. First, we can focus on the hardest splits that can be generated, which will stress-test our NLI architectures in a standard behavioral way. Second, the MQNLI labeling algorithm itself suggests an appropriate causal model of the data-generating process. Figure 2a summarizes this model tree form, and it is presented in full detail in Geiger et al. [0O]. This allows to rigorously assess whether neural network has learned to implement variants of this causal model. The complexity of the MQNLI examples creates many opportunities to do this in linguistically interesting ways.
5

---

OPee
Outy
OPte
Maa
Nia
Od
Adde
Adc
V
Addo
(a) The causal structure of the high-level natural logic causal model Chantag that performs inference on MQNLI. The superscripts P and I. stand for 'premise' and 'hypothesis' and the subscripts 'Subj' and 'Obj' stand for 'Subject' and 'Ohject'. The node labels are used to explain the experimental results Section
every baker eats no c bread contradiction angry baker
Model
Train
Dev
Test
bread every silly professor e e sells not every & book neutral
CBoW
88.04
54.18
53.99
TreeNN
67.01
54.01
53.73
CompTreeNN
99.65
80.17
80.21
every silly professor e E sells not every chair
BiLSTM
99.42
46.41
46.32
not every sud baker fairly admits not every odd idea entailment some baker does admits idea
BERT
99.99
88.25
88.50
(c) MQNII results. The first three models are from Geiger al. [0 where the CompTreeNN specific model not suitable for general NL.I and functions as an idealized upperbound. Our results show that BERT-based models can surpass this without such alignments.
task-
(b) MQNLI examples. The token serves padding (but still attended by the model) and ensures perfect align- ment between both premises and hypotheses and across all examples. It is semantically an identity element.
Figure 2: The natural logic causal model (top), MQNLI examples (left) and MQNLI results (right).
Models We evaluated two models MQNLI: randomly initialized multilayered Bidirectional LSTM (BiLSTM; (23)) and a BERT-based classifier model which the English bert-base param- eters 8 are fine-tuned on the MQNLI train set. Output predictions are computed using the final representation above the [CLS] token. Models are trained to predict the relation of every pair of aligned phrases in Figure 2a Additional model and training details are given in Appendix
Results Figure 2e summarizes the results of our BERT and BiLSTM models on the hardest fair generalization task Geiger et al. [10] creates with MQNLI. We find that our BiLSTM model is not able to learn this task, and that our BERT model is able to achieve high accuracy. The only models in Geiger et al. [10] able achieve above 50% accuracy were task-specific tree-structured models with the structure of the tree in Figure 2a Thus, our BERT-based model is the first general-purpor model able to achieve good performance this hard generalization task. Without pretraining, the BERT-based model achieves 2=4.. %, confirming that pretraining is essential, as expected.
A natural hypothesis is that the BERT-based model achieves this high performance because it has in effect induced some approximation to the tree-like structure the data- lata-generating process in its own internal layers. With causal abstraction analysis, we are actually in position to test this hypothesis.
5
A Case Study in Structural Neural Network Analysis
5.1 Causal Abstractions of Neural NLI models
Formulating Our Hypotheses We proceed just we did for the simple motivating example in Section except that we are now seeking to assess the extent which the natural logic algebra in Figure 2a is a causal abstraction of the trained neural models in the above section.
The hallmark of Figure 2a is that it defines alignment between premise and hypothesis both lexical and phrasal levels. This permits us to run interchange interventions in a naturally compositional
way. For given non-leaf node N in Figure 2al let CN Nattin be submodel of Cootog that computes
the relation between the aligned phrases under N and uses them to compute the final output relation
NPos
between premise and hypothesis. For example, let Nattog be the submodel of C that computes

---

Tabel
NPaii
Oayy Adllcs New Ney" Aav" v Gis
A4loy Noe
(CLS)
Adg...
NS
Adting
NSu
2g Aillly Nfa, Neg" Aev" qill Qu4
Adii' No
Figure 3: A BERT-based NLI model (left) aligned with the natural logic causal model N (right),
where the fourth vector representation above the Adj token in the network is aligned with NPoij- the variable representing the relation between the object noun phrases. When analyzing sample of 1000 examples, we found subset of 383 where Nadlog is an abstraction of NNt under this alignment.
Nas
the relation between the two aligned object noun phrases and then uses that relation in computing the final output relation between premise and hypothesis (see Figure FigureBright). We would like to ask whether our trained neural models also compute this relation between object noun phrases and use it make a final prediction. We can pose this same question for other nodes which correspond a pair of aligned subphrases.
Alignment Search For each N, we search for an alignment between neural representation in NNLI and the variable in C. Matto In principle, any location in the network could the right one for any causal model. Testing every hypothesis in this space would be intractable. Thus, for each Natlog: we consider a restricted set of hidden representations based on the identity of N. The BERT model we use has 12 Transformer layers [30], meaning that there are 12 hidden representations for each input token. Each alignment search considers aligning the intermediate high-level variable with dozens of possible locations in the grid of BERT representations. Specifically, the following locations were considered for each N:
CNN
Subj. Adjsubj+ Subjj- Neg. Adv, V, Qoej Adjoyj. Now: hidden representations above the two descendant leaf tokens.
QP hidden representations above Q and Qōkj-
NegP: same but above Neg and Neg".
NPsuhj. VP, and NPosj: same but above the four descendant leaf tokens.
All nodes (for BERT): same but above [CLS] and [SEP].
For each alignment considered, we performed a full causal abstraction analysis. We report the results from the best alignments in Table I and summarize the results from all alignments in Appendix
Interchange Interventions We first focus our high-level causal models. Consider non-leaf node N from Figure 2a and two input token sequences e and from MQNLI. Define
Ne' (e)
(4)
to be the output provided by the causal model Nod when processing input e where the relation between the aligned subphrases under the node N is changed to the relation between those subphrases in e'. For example, simplifying for the sake of exposition, suppose e is (some happy baker, no baker), which has output label contradiction, and suppose e' (every happy person, some happy baker), which has output label entailment. We wish to intervene on the noun phrase, so N = NP. In C'NP-e'
the noun phrase relation is entailment; in e', it is reverse entailment. Thus, Caatog (e) changes the object noun phrase relation in entailment while holding everything else about constant. This results in the output label for the example (some happy person, no c baker), which is neutral.
Next, consider interventions in neural model NNU. Define
Le-e NNJ
(e)
(5)
7

---

Table Largest subsets examples on which specific models C Natlag are abstractions of an LSTM and BERT model trained MQNLI. We record the size of such subsets percentage of the total 1000 examples. On this subset, we know that the neural models compute representation of the relation between the aligned subphrases under N and use this information make final prediction.
Causal Model
LSTM
BERT
Nodes added
BERT
Qsubj
0.7
13.1
Adjsun
30.5
Qoii
0.9
7.3
Nodes removed
BERT
Nautj.
37.2
Neg
0.7
21.4
Nobi
31.9
Neg'
14.9
Adjsan)
2.5
6.7
Agg
15.7
Adv
26.9
Nsubj
1.2
5.5
Nobj
33.8
V
35.6
Adjanj
0.9
Agjj
15.8
Qosj
16.2
Nobj
0.7
8.8
Nonj Aguj
31.9
Adjsuty
13.4
V
0.4
11.4
NPsabj
Adv
1.4
1.0
7.9
6.7
No Nobi
Nbjs Agui
32.2
14.1
Neg H
Suty
34.4
12.0
NPos
0.7
38.3
Novj Ag
31.6
Adv
16.2
VP
0.4
11.4
Aobi Agoi
8.8
V
13.4
NegP
0.9
11.8
Non Age
32.1
Q
12.0
(a) Main results (clique sizes) for non- leaf nodes of the tree in Figure 2a The hypothesis have most evidence for that the BERT model computes a rep- resentation the NP node with the alignment shown in Figure 3 Remark- ably, with 1000 examples sampled, found subset of 383 examples where NPoH Kantag is abstraction of BERT.
(b) Detailed results (clique sizes) for Alternative causal models & in "ncighborhood" around the model NNutagg which has a single intermediate variable composed of four lexical items (See Figure At left, have alternative causal models where two those lexical items are removed from composition. At right, we have alternatives obtained by adding one lexical item to the composition. We observe that no alternative hypothesis about causal structure considered has more evidence.
to be the output provided by NuI processing the input when the representation location Z, is replaced with the representation at location L created when NNu is processing e'. This is exactly the process depicted in Figure except now the networks are the complex trained networks of Section 4
Our hypothesis linking Figure 2a with model NNu takes the same form 3 The causal model Nontog is a constructive abstraction of Nuu when, for some representation location L, it is the case that, for all MQNLI examples and e', we have
No Nailog = N---e' (e)
(N)=NN
(6) the representations at L in network NNu
This asserts correspondence between interventions and interventions on the variable N in the causal model C. Nat If holds, then Nnw computes the relation between the aligned phrases under the node N and uses this information to compute the relation between the premise and hypothesis.
We call pair of examples ee,e') successful if satisfies equation ( i.c., interventions in both the target causal model and neural model produce equal results. addition, to isolate the causal impact of our interventions, specifically focus on pairs e', for which performing the intervention produces a different output value than without the intervention. We call a pair (e, e') impactfil if:
N NotLog (e) o NanLog (e)
Quantifying Partial Success Equation 66) universally quantifies over all examples. We do not expect this kind of perfect correspondence to emerge in practice for real problems: neural network training is often approximate and variable in nature, and even our best model does not achieve perfect performance. However, we can still ask how widely (6) holds for a given model. To do this, we seek find the largest subset MQNLI on which CN non-leaf node N in ONaatog
NauLog is an abstraction of our neural models, for each

---

More specifically, considering each example in MQNLI as vertex in a graph, we add an undirected edge between two examples e; and e, if and only both the ordered pairs (e,, e₃) and (e,,e, satisfy (6). In other words, C (antog is an abstraction of neural model on a subset of examples S of MQNLI if and only if all examples in S form a clique.
need to run scales quadratically with the number of inputs The number of interventions consider, so we sample 1000 MQNLI examples, producing a total of 1000² = 1M ordered pairs. We only consider examples for which the neural network outputs correct label. For each node N and each of its corresponding neural network locations L, we perform interventions on all of these pairs.
We choose to measure the largest clique with at least one impactful edge, because (1) the causal abstraction relation holds with full force on that clique, but other measures such as the total number of connections lack this theoretical grounding, and (2) if a clique has at least one impactful edge, that guarantees the high-level variable is being used.
Results and Analysis For each target causal model node N and neural network representation location L. we construct a graph as described above with 1000 examples as vertices and add an edge between two examples e, and ej if and only if both (e,, e;) and (e,, ei) are successful. We then find the largest clique in this graph with at least one impactful edge and record its size.
Table Ia shows, for each causal model node N, the maximum size of cliques found among all neural locations. With this stricter impactful criterion (as opposed to simply using intervention success), our results show that, for almost all nodes N, our target causal model C Nattoy is indeed causal abstraction of BERT on significant number of examples in our dataset. These subsets are much smaller for the BiL-TM model.
We also investigated alternative high-level causal structures that are not variants of CNimoog from Figure 2a Specifically, we consider alternative models in a "neighborhood" around the model Na that can be obtained by adding one leaf, or by removing one or two leaves to the composition. These results are in Table Tb Remarkably, all of these alternative models result in smaller clique sizes, significantly so for many of them. This further supports the significance of our results.
This analysis is similar to the analysis of our hypothetical addition example in Section 3 except for two crucial differences. First, for each variable N, we are hypothesizing that the causal model C.N is an abstraction of Nwu, whereas in the addition example there was only one model. To investigate this difference, we take N = NPos as a paradigm case, as it is the model with the strongest results. (The results for other nodes success, whereas in the addition example we assumed complete success. Crucially, this means that the following analysis will be valid only on subsets of the input space on which the abstraction relation holds between NNL and NPas
Natlog
in Appendix D Second, we only achieved partial experimental
Nag
We visualize the results of our intervention experiments for the node NPobj in Figure The alignment with the largest subset of inputs aligns the NPosj variable in N with the neural representation on the fourth layer of BERT above the Adjōs token (see Figure 3). Because neural representations encode the value of their aligned variables and play a parallel causal role to their high-level variables, we know that, on this subset of input examples, at the fourth neural representation above the Adi token, the four input embeddings for the object nouns and adjectives in the premise and hypothesis are composed to form a neural representation with information content of the relation between the object noun phrases in the premise and hypothesis. Then this representation is composed with the other input-embeddings to create an output representing the relation between the premise and hypothesis.
5.2 Comparison with Other Structural Analysis Methods
Probes We probed neural representation locations for the relation between aligned subexpressions on a subset of 12,800 randomly selected MQNLI examples. For a pair of aligned subexpressions below node N in Figure 2a we probe the columns above the same set of restricted class of tokens as described in Section
To evaluate these probes, we report accuracy as well as selectivity as defined by Hewitt and Liang 12] probe accuracy minus control accuracy, where control accurucy is the train set accuracy of a probe with the same architecture but trained on control task to factor out probe success that can be

---

01234567889 9
012345678
0.6
0.3
-0.
-D.4
-0.2
-08
0.0
(CLS]
sajosy"
1989)
waby"
(STP)
(CLS)
Aijoby
NObf
[SFP]
Ad
NOO/
[e
Mijojj
NCa/
[SEP)
Mgooor wosj"
(SEP)
(cus)
Ad|Cif
NGAr
[SEP)
Asjoor"
NOw"
(sen)
(a) Clique size.
(b) Interchange success.
(c) Probe selectivity.
(d) Probe Accuracy.
Figure 4: Interchange intervention and probing results for the NPosj position. Vertical axes denote layers of BERT and horizontal axes denote the token position of hidden representations. The intervention success rates reported here are calculated based on intervention experiments with change in the output label. Clique sizes are reported as % of 1000 examples.
attributed to the probe model itself. Our control task is to learn a random mapping from node types to semantic relations; see Appendix C for full details how this task was constructed.
Figure summarizes our probing results for N = NPosj. along with corresponding interchange intervention results for comparison. Probes tell that information about the relation between the aligned noun phrases is encoded in nearly all of the locations we considered, and using the selectivity metric does not result any qualitative change. In contrast, our intervention heatmaps indicate only small number of locations store this information in a causally relevant way. Clearly, our intervention experiments far more discriminating than probes. Appendix D provides examples involving other variables along with the intervention experiments, where the general trend of interchange interventions being more discriminating holds.
Integrated Gradients Attribution methods that estimate feature importance measure causal properties of neural representations, but single feature importance method is an impoverished characterization of representation's 's role in network behavior. Whereas our interchange interventions gave us high-level information about how neural representation is composed and what it is composed into, attribution methods simply tell "how much" a representation contributes to the network output on give input. Moreover, intervention interchanges provide rich, high-level characterization of causal structure on a space of inputs.
We use integrated gradients on our models to verify the intuitive hypothesis that if a premise and hypothesis differ by single token, then the neural representations above that token should be more causally responsible for the network output than other representations. For example, given premise 'Every sleepy cat meows' and hypothesis 'Some hungry cat meows', the attributive modifier position is different and the rest are matched. The neural representations above the adjective tokens sleepy and hungry should be more important for the network output than others, because if those adjectives were the same, the example label would change from neutral to entailment. We summarize the results of our integrated gradient experiments in Appendix E. where we confirm our intuitive hypothesis.
6 Conclusion
We have introduced methodology for deriving interpretable causal explanations of neural network behaviors, grounded in formal theory of causal abstraction. The methodology involves first formulating hypothesis in the form of high-level, interpretable causal model, then searching for an alignment between the neural network and the causal model, and finally verifying experimentally that the neural representations encode the same causal properties and information content as the corresponding components of the high-level causal model. As case study demonstrating the feasibility of the approach, we analyzed neural models trained on the semantically formidable MQNLI dataset. Guided by the intuition that success on this challenging task may call for way of recapitulating the causal structure of the natural logic model that generates the MQNLI data, we were able to verify the hypothesis that state-of-the-art BERT-based model partially realizes this structure, whereas baseline models that do not perform as well fail to do so. This suggestive case study demonstrates that our theoretically grounded methodology work in practice.
10

---

Acknowledgments and Disclosure of Funding
Our thanks to Amir Feder, Noah Goodman, Elisa Kreiss, Josh Rozner, Zhengxuan Wu, and our anonymous reviewers. This research is supported in part by grants from Facebook and Google.
References
[1] S. Beckers and J. Halpem. Abstracting causal models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):268-26655 Jul. 2019. doi: 10.1609/aaai, .33011..3.. httph:t/p/:/o//// aaai index.ndexxxpe http://www..pp//A///rrttcceeeeee
[2] S. Beckers, F. Eberhardt, and J. Y. Halpem. Approximate causal abstractions. In R. P. Adams and V. Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 606-615 Tel Aviv, Israel, 22-25 Jul 2020. PMLR. URL httphh::///prcceeinngg..o.. wttpigssmm.....rree. w.l..prres//111//bckkrr223
http::/ppoceeedss....rrssssssssp html
[3] A. Binder, G. Montavon, S. Bach, K. Müller, and W. Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. CoRR, abs/1604.00825, 2016. URL 00825
http://arxiv.org//abs..
[4] S. Bongers, P. Forré J. Peters, B. Schölkopf, and J. M. Mooij. Foundations of structural causal models with cycles and latent variables. arXiv.rr preprint, arXiv:1611.062211v [stat.ME], Oct. 2020. URL http:::tttxt....ttoo org abs /161 06221v4
[5] K. Chalupka, F. Eberhardt, and P. Perona. Multi-level cause-effect systems. In A. Gretton and C. C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 361-369, Cadiz, Spain, 09-11 May 2016. PMLR. URL http:://proceedingg.. mlr ptt.peee///////haalppala html
[6] A. Chattopadhyay, Manupriya, A. Sarkar, and V. N. Balasubramanian. Neural network attributions: A causal perspective. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 981-990, Long Beach, California, USA, 09-15 Jun 2019. 9a html
PMLR. URL BtLlhttp:/pprceeeeie
mlr htt../ppreeee//e/cttttttaaaeeee
[7] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does BERT look at? an analysis of BERT's s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10. 10.1853//1/W11-4580 URL https //www.aclwww. orgorggg/g... wwwhhoooggowiit 4828
[8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NI 1423. URL https: //ts:///www.. http://wwwwwww....c//// htrggantoooogy//ae 1423
[9] Y. Elazar, S. Ravfogel, A. Jacovi, and Y. Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. In Proceedings of the 2020 EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NLP. Association for Computational Linguisties, Nov. 2020, doi: 10. 18653/v1/WW httpw///////// 18-5426.
[10] A. Geiger, Cases, L. Karttunen, and C. Potts. Posing fair generalization tasks for natural language inference. In Proceedings of the 2019 Conference on Empirical Meth- ods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)), pages 4475 4485, Stroudsburg, PA, Novem- ber 2019. Association for Computational Linguistics. doi: 10.1 10.18665/1//11114456 URL. https://wwet aclweb. httpr///rr/aaatooooo////------/

---

[11] A. Geiger, K. Richardson, and C. Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxVLP Workshop on Analyzing and Interpreting Neural Networks for NLP., pages 163-173, Online, Nov. 2020. Association for Computational Linguistics. doi: 10. 1111116611//2222... URL /www://wwwwwwww awwweb. org/ antho onnthooggy/2en 2020 blackboxnlp-1..1
blackboxnlp-111666
[12] J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Prucessing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-IJCNLP)) pages 9733-2743, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: N2.16633/1/DD19-227 URL URI.https: https: / wwtp:wwww.clccccc. awlweb..ww.................... or httpg/aahoooooo/1111120
[13] D. Hupkes, S. Bouwmeester, and R. Fernández. Analysing the potential of seq-to-seg models for incremental interpretation in task-oriented dialogue. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pages 165-174, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics. doi: 10. 18633v13/v/ W18-5419. URL htt.st /www://wwwwwwo/o www.ww.ww.b...................ww org/ oottholog/w1185444 8-19
[14] T. F. Icard and L. S. Moss. Recent progress on monotonicity. Linguistic Issues in Language Technology, 9(7):1-31, January 2013.
[15] G. W. Imbens and D. B. Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[16] T. P. Lillicrap and K. P. Kording, What does it mean to understand neural network?, 2019.
[17] B. MacCartney and C. D. Manning. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE '07, pages 193-200, Stroudsburg, PA, USA, 2007. Association for Computational Linguistics. URL http://dl. wwt.ddl..cm.orr.oo..o.o http://dd..ac.org/citaattoooooot
1654575
[18] B. MacCartney and C. D. Manning. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140-156, Tilburg, The Netherlands, Jan. 2009. Association for Computational Linguistics. URL https: aclweb.
[19] J. Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, OAI'11 page 411 111-420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
[20] M. Peters, M. Neumann, L. Zettlemoyer, and W.-t. Yih. Dissecting contextual word embed- dings: Architecture and representation. In Proceedings of the 2018 Conference on Émpir- ical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium, Oct.- Nov. 2018. Association for Computational Linguistics. doi: 10. 10.1885//1//18-1111 1179. URL
https://www. http://www..llww.. www.wb..wwbbbb.... http:ee.....g.tbbhgeeeeee http.////aaaooooy/////////
[21] A. Ravichander, Belinkov, and E. Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance?, 2020.
[22] P. K. Rubenstein, S. Weichwald, S. Bongers, J. M. Mooij. D. Janzing, M. Grosse-Was Wentrup. and B. Schölkopf. Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI). Association for Uncertainty in Artificial Intelligence (AUAI), Aug. 2017. papers/11. pdf * *equal contribution.
[23] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE transactions Signal Processing, (5(11))2663- 268822, 1997.
[24] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kundaje. Not just a black box: Learning important features through propagating activation differences. CoRR, abs/ 1o00//605.,,13,, 2016. URL httt::ttt//riii http:/v..orgabb///eeeeeeee
abs/ 1605. 01713
[25] Spirtes, C. N. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT Press, 2nd edition, 2001. ISBN 9780262194402.
12

---

[26] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, 12 2014.
[27] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328, 3328, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:/ pproced/proosss.ngg mlr.pt Aur.press/vTT v70/ wttp://unddaaaaaaaaaaaaaaa an 17. html
[28] I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4593- 4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10 P19-1452. URL httph:h/pwwwttt http:://www.a...wbbboo aclweb. org hatp/ooogo///111145555555
[29] van Benthem. brief history natural logic. In M. Chakraborty, B. Löwe, M. Nath Mitra, and S. Sarukki, editors, Logic, Navya-Nyay and Applications: Homage to Bimal Matilal, 2008.
[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. I. Guyon, U. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc., 2017. URL httt:://papers.iigg 1ttee//188l-ttttentoo--s---ll--
need. pdf
[31] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, hts/1990.03771, 2019.
[32] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision ECCV 20/4, pages 818-833, Cham, 2014. Springer International Publishing. ISBN 100---3111-199----
13


Reviews:
[{"review_id": "vjw0oy48TTH", "reviewer": "Reviewer_Wd32", "summary": "see main", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to formalize and generalize the \"interchange\" approach to\ninterpreting computation in deep networks first proposed by Geiger et al.\n(BlackBoxNLP 2020). In this experimental paradigm, an abstract causal model f of\nthe computation performed by a deep network model g is first hypothesized. Next,\nalignments between individual variables in this causal model and components of\nthe deep network are guessed. Finally, the causal model + alignment combination\nis validated by testing whether (in roughly Pearlian terms) f(y | x, do(z_f')) =\ng(y | x, do(z_g')) for model outputs y and aligned latent / representation\npairs (z_f', z_g') extracted the application of f and g to alternative inputs x'. \n\nThe paper has three contributions:\n\n(1) formalizing this analyis procedure in the language of constructive causal\n    abstractions (Beckers & Halpern 2019).\n\n(2) extending it to include a (brute-force) search over alignments rather than\n    requiring researchers to fix these up-front\n\n(2) applying it to the MQNLI dataset rather than the MONLI dataset originally\n    used by Geiger et al '20\n\nSTRENGTHS\n\n- Nice contextualization / formalization of the experimental paradigm of Geiger\n  in terms of existing work on causal inference\n\n- Useful example constructions for clarifying the expressive power of the\n  probing paradigm in NLP\n\n- Convincing evidence that the MacCartney-style inference is actually\n  implemented (in a causal sense) by BERT models trained on MQNLI.\n\nWEAKNESSES\n\n- Approach itself is not new, and empirical findings largely overlap with the\n  earlier Geiger paper.\n\n- Most of the interesting content is in the appendix!\n\nI'm quite conflicted about this paper. On one hand, drawing a link between\ninterchange interventions and the rest of the causal inference literature is\nsuper useful. So are the example failure modes for probing and IG---while these\nare simple constructions, I haven't seen them written down before and it would\nbe nice to have something to cite! But instead of focusing on this content, the\npaper is mostly about analyzing the behavior of a single model trained on a\nsynthetic NLI dataset trained with an exotic form of supervision. I don't think\nthis finding is generalizable enough to be of interest to a general machine\nlearning audience. Meanwhile, the other methodological innovation on top of the\noriginal Geiger paper is the addition of a search over candidate encodings,\nwhich this paper largely sidesteps by manually pruning the search space by exploiting the regular structure of the\nsynthetic data. I think a stronger version of the paper would either (1) promote\nappendices A and G to the main paper (and talk a little more about how we can\nuse other insights from causal inference to guide analysis of deep nets), (2) do\nan empirical analysis of more than one model (e.g. a model trained on ordinary\nNLI data), or (3) say a little bit more about how to automate the search (e.g.\nby first building the matrix of correlations between causal variables and model\nrepresentations, and using these to prune the search space explored by\ninterchange interventions).\n\nHowever, I'm absolutely willing to be persuaded that the paper is substantial enough to\naccept in its current form, and would be interested in hearing more from the\nauthors about what they view as the delta (formally or empirically) relative to\nGeiger '20 and the significance of the MQNLI results.\n\nSome other thoughts:\n\nSECTION 3\n\nWhile the example in Fig 1 is useful, it's a little odd for a NeurIPS paper to\ncontain *only* a worked example, and leave the general framework that is being\nexemplified out of the paper entirely. Building on the general comment above, is\nit possible to interleave the discussion in section 3 with any concrete\ndiscussion of constructive abstractions / interchange interventions in general?\nTo get space back, Fig 3 could be shrunk, and the nearly 2 pages of related work\nspread across sections 2 and 5 could probably be condensed a bit.\n\nSince interchange success and clique size tell different stories\nabout where causally relevant information is located (Fig 4), can we say\nanything meaningful about the relationship between the two, or to formal\ncriteria for constituting a causal abstraction?\n\nSECTION 5\n\nIs max clique size really the right measurement here? I'd obviously rather have\none giant cluster and one medium sized cluster vs one giant cluster and 500 tiny\nones, and you could look at something like the entropy of the induced cluster or\nthe probability that any two nodes are connected by an edge.\n\nWhat's the motivation for requiring cliques to have *at least one* impactful\nedge as opposed to something stricter? (e.g. start with the graph in which\n*every* edge corresponds to a label-changing substitutions in the true\ncausal model, then just measure the fraction of these edges where interchange\ncauses a corresponding label change in the neural net).\n\nMISC\n\nExtremely fussy: use \\citet in expressions like \"the dataset of [CITE]\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see main", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "N6lRfxxsOwB", "reviewer": "Reviewer_iJ4z", "summary": "The paper presents a structural analysis tool grounded in the causal abstraction theory, through which it attempts to identify the causal properties governing the inference of the given task. This method is an alternative to probing, which involves training a supervised classifier to identify the causal relationships in many NLP tasks. Using this causal abstraction analysis tool, the authors study MQNLI corpus, which is a specifically constructed tree-structured corpus for natural language inference, grounded in natural logic. Through various experiments, the authors observe that BERT-based model approximately realizes the causal properties of the data generating process better than the baseline LSTM model.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents novel, original method of identifying the causal mechanisms in play in neural networks for reasoning. This paper provides clear distinction of its methods from the prior work of probing literature and attribution methods using integrated gradients. The paper also provides good coverage of the related work which helps the reader to understand the necessary background.\n\nThe paper begins with a formal description of the hypothesis-alignment-intervention process. Certain parts of this section is not clear. For example, it is not clear (in L158 to L160) why z is not causally represented at L1 even if L1 depends on z in the MLP. The comparison with attribution based methods following the above argument is not clear, as a scalar value of 0 assigned with L1 with z would represent the same non-causal relationship the author is alluding to.\n\nThe paper is slightly hard to follow as it lacks clear examples. While the paper provides a good toy example describing the hypothesis formulation, alignment search and intervention process, those examples are not aligned with MQNLI pairs. In Figure 2, the causal structure is provided but it is not clear which pair it aligns with. Hence, following the arguments presented in Section 5.1 is hard. In Section 5.1, the interchange intervention possibly converts subsequence $e$ (\"every happy person\", \"some happy baker\") to (\"every happy baker\", \"some \\epsilon baker\"). It is unclear how changing this subsequence renders the output label to neutral. A better example (including the entire sentence, not only the subsequence) would be helpful for the reader to understand the intervention process.\n\nThe analysis presented in the paper is also restricted to MQNLI - possibly due to the fact such interventional analysis would require practitioners to have access to the data generating process of the underlying dataset. This reduces the significance of the work a bit, although the sound theoretical motivation provided in this paper balances out the lack of downstream usability from an academic point of view. However, the paper only investigates two models - LSTM and a pre-trained BERT base - which reduces the significance of the results even academically. Out of these two models, LSTM model is much weaker compared to the BERT model - 46.32% compared to 88.50% - which makes the baseline of the causal abstraction argument much weaker anyway. The success of the high score of the BERT-based model is mostly due to pre-training, which the paper does not identify in the discussion. A better comparison would have been to isolate the non-pretrained models (including a BERT trained from scratch on MQNLI) and the pre-trained models (BERT and its family).\n\nThe alignment search phase raises a lot of questions as well. It is unclear in case of BERT where the representations are taken from. Due to multiple cascaded attentions in Transformer models, it is hard to pin down the representation of a word being the same location after several layers of self-attention. Thus, it is unclear how the hidden representations for tree nodes are extracted from BERT. Also, it is unclear how the word phrase representations (from nodes higher up in the tree) are used from BERT in the experiments.\n\nThe results and discussion section needs polish as well. While it is understandable that the causal intervention tool provides better signal than probing tasks, less discussion is provided on what exactly do we learn from the analysis of the alignment - i.e how important it is on the final prediction.\n\nOverall, having lack of model exploration and restriction to MQNLI dataset does not exhibit condfidence that the proposed methodology can work in practice on other tasks. However, the theoretical experimental setup are still valuable for future practitioners to develop interventional analysis tools.\n\nEdit: I have increased my score to 6 following the discussion with the authors, where they provide additional experiments and commit to clarify the examples and results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have not indicated any limitations of their study in the paper.\n- In my opinion, the work should not result in any negative societal impact or ethical concerns. Thus, it does not require any further ethics review.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "7xMsFGpjwj3", "reviewer": "Reviewer_5z8f", "summary": "1. The authors introduce a \"causal abstraction\" method for attributing high-level descriptions (combining representations and computations) to task-optimized neural networks.\n2. They operationalize their method by performing causal abstraction analysis on a BERT model fine-tuned for MQNLI. They find substantial subsets of an MQNLI dataset for which their interchange intervention predictions are successful between any two pairs of examples.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a very interesting interpretability method which goes beyond correlational and weaker causal analyses in the literature. However, I am concerned about a few practical issues that might affect the generalizability of the method. These are addressed in the \"quality\" section below.\n\nI would be happy to see work like this published, and would recommend this paper for acceptance if the authors can adequately address these concerns. I may have misunderstood some parts of the analysis and would happily accept corrections as well. \n\n## Quality\n\n1. My main issue is with the lack of a comparison / baseline in the MQNLI evaluation. It's hard for me to know how to distinguish between a successful and unsuccessful analysis here. Concretely: at what level (and on what measure) would we confidently say that a proposed causal model is / is not a causal abstraction of the NN?My first thought on seeing the numbers in figure 3 was that this was not a success: for some syntactic constituents there are moderately-sized cliques of examples on which intervention is successful. But this doesn't hold more generally for the dataset, and the clique metric seems to degrade quickly as we move to the higher constituents covering larger spans of the input sentence.I would like the authors to address whether and how we can understand when a causal abstraction analysis has succeeded. (Are there reasonable a priori points on the evaluation metric scale which signal success? Are there statistical tests, e.g. a permutation test involving alternative forms of the causal model, which can give us a firm basis for judging success?)\n2. A substantial practical issue with this method is inferring the alignment between variables in the causal model and \"locations\" in the neural model (I believe, following lines 89--90, this means subsets of neurons). As far as I can understand it, the authors did not actually run an alignment search in the MQNLI experiments, but rather stipulated that particular subsets of neurons should correspond to causal model variables (lines 218--231).I believe the generalizability of this method is severely limited by this alignment search method, as I understand it. There is very little information in the main text suggesting how this was carried out in their experiments, or gestures at how this would transfer to the general case. I enumerate my specific concerns below.I would like the authors to address these questions in the paper, and ideally demonstrate through their experiments that a full alignment search process can be tractable in a large-scale model.\n  1. Such an alignment search involving models with complex causal structures would be expensive in terms of computation and data. For any node in the causal model, we would need to search among all subsets of neurons for correlations between causal model variables and neural representations. (I'm assuming correlation would be the measure, though I don't believe this is stated in the paper.) This would naturally require an extra development dataset separate from the one used for interchange experiments.\n  2. The success criterion for alignment search is also an important question not addressed in the paper. At what degree of correlation do we conclude that an NN location is related to a causal model variable?\n\n## Originality\n\nI believe this principled approach to causal intervention is a novel contribution to the literature. However, it might be interesting to explore the relationship between this general method and related specific causal attribution analyses in NLP with a more implicit causal model -- for example, \\[1\\] run intervention analyses on language models to test how plurality and syntactic depth are represented.\n\n## Minor comments and questions\n\n1. 222--224: Why are there \"two descendant tokens\" for constituents such as Adj, Adv, V? As I understand from the example figure, you only consider inputs with one token for these constituents.\n2. Fig. 3: which layer of the BERT model is used for these experiments?\n3. More generally, I don't understand the relationship between fig. 3 and the \"interchange intervention\" experiments shown in fig. 4\\. Fig. 3 shows the largest subset of examples on which C is a causal abstraction of the BERT model -- is this not the same as the size of the largest clique in the example graph with edges denoting interchange intervention success (i.e. the values shown in fig. 4a)?\n4. I like the idea of using graph metrics for broad quantitative evaluation of the metric. But why is clique size the right metric? This seems to set a very high standard. Why not also incorporate a more permissive metric such as average degree, or visualize the full distribution over node degrees?\n\n## References\n\n1. Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., & Baroni, M. (2019). The emergence of number and syntax units in LSTM language models. arXiv:1903.07435 \\[cs\\].[http://arxiv.org/abs/1903.07435](http://arxiv.org/abs/1903.07435)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0_ZGBxyA1Lu", "reviewer": "Reviewer_C38X", "summary": "This paper proposes a method for identifying causal structures captured within neural networks. The proposed method does so by aligning a causal model with the neural networks’ prediction using interchange interventions. The authors demonstrate the efficacy of this method using the MQNLI corpus, where they show that a BERT-based model is compatible with a natural logic causal model such as the one originally used to generate the data. \n\nThus, the main contribution of the paper is a methodology for testing hypotheses regarding underlying causal structures learned in neural networks, which works well on an automatically generated natural language inference dataset.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a complete process for generating hypotheses regarding possible causal structures governing the behavior of neural networks, and for testing their fit to the model's observed behavior. While it does require advanced prior task-related knowledge regarding possible causal structures which might be learned by the neural network, the proposed method is useful for tasks which could rely on such structure, such as many syntactic and semantic tasks in NLP.\n\nThe paper presents some originality in its attempt to provide causal abstractions and not only estimate causal effects, it is essentially a method for testing specific hypotheses regarding the interaction of model components. For researchers who are interested in estimating causal effects on neural networks and not in what specific model components (i.e. layers, neurons) are driving behavior, there are existing causal explanation methods that have already suggested possible solutions.  \n\nStill, this is an interesting paper that provides a clear and sound evaluation, and can surely contribute to researchers interested in uncovering where in the model a specific reasoning might occur (similarly to Causal Mediation Analysis). If rephrased as a contribution for reasoning about a specific model reasoning structure and not as a contribution for causal effect estimation, I think that this paper has a clear contribution to the literature. The current state of the paper, where it is presented as a tool for generating causal explanations (such as CausaLM), is somewhat misleading. Without comparing examples to counterfactuals, it is simply not true to state that this method truly uncovered causal effects.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper can potentially have a positive societal impact, as understanding the causal structure engrained in neural networks can uncover hidden biases and fairness issues which govern the model’s behavior. \n\nThere are certainly limitations to this work, such as the reliance on a user’s ability to design a causal graph which might explain the neural network, and the inability of the method to provide causal identification besides fitting some potential model to the data. However, the paper provides a way forward for potentially explaining where causal knowledge is captured in neural networks. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I don't think this paper requires a long metareview. They introduce a method for aligning the predictions of a network with an induced causal model of the task's "dynamics". They show it works on some NLI tasks which is respectable (although I would have liked to see some other domain too... maybe something to do in future work!). The reviewers were excited about the work, and several were ready to champion its acceptance after discussion with the authors. I recommend acceptance, and hope that ACL's loss here will be NeurIPS' gain.
Decision: Accept (Poster)