Causal Abstractions of Neural Networks

1 Introduction . Explainability and interpretability have long been central issues for neural networks , and they have taken on renewed importance as such models are now ubiquitous in research and technology . Recent structural evaluation methods seek to reveal the internal structure of these “ black box ” models . Structural methods include probes , attributions ( feature importance methods ) , and interventions ( manipulations of model-internal states ) . These methods can complement standard behavioral techniques ( e.g. , performance on gold evaluation sets ) , and they can yield insights into how and why models make the predictions they do . However , these tools have their limitations , and it has often been assumed that more ambitious and systematic causal analysis of such models is beyond reach . Although there is a sense in which neural networks are “ black boxes ” , they have the virtue of being completely closed and controlled systems . This means that standard empirical challenges of causal inference due to lack of observability simply do not arise . The challenge is rather to identify high-level causal regularities that abstract away from irrelevant ( but arbitrarily observable and manipulable ) low-level details . Our contribution in this paper is to show that this challenge can be met . Drawing on recent innovations in the formal theory of causal abstraction [ 1 , 2 , 5 , 22 ] , we offer a methodology for meaningful causal explanations of neural network behavior . Our methodology causal abstraction analysis2 consists of three stages . ( 1 ) Formulate a hypothesis by defining a causal model that might explain network behavior . Candidate causal models can be naturally adapted from theoretical and empirical modeling work in linguistics and cognitive sciences . ( 2 ) Search for an alignment between neural representations in the network and variables in the ∗equal contribution 2We provide tools for causal abstraction analysis at http : //github.com/hansonhl/antra and the code base for this paper at http : //github.com/atticusg/Interchange 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) high-level causal model . ( 3 ) Verify experimentally that the neural representations have the same causal properties as their aligned high-level variables using the interchange intervention method of Geiger et al . [ 11 ] . As a case study , we apply this methodology to LSTM-based and BERT-based natural language inference ( NLI ) models trained on the logically complex Multiply Quantified NLI ( MQNLI ) dataset of Geiger et al . [ 10 ] . This challenging dataset was constructed with a tree-structured natural logic causal model [ 17 , 29 , 14 ] . Our BERT-based model has the structure of a standard NLI classifier , and yet it is able to perform well on MQNLI ( 88 % ) , a result Geiger et al . achieved only with highly customized task-specific models . By contrast , our LSTM-based model is much less successful ( 46 % ) . The obvious scientific question in this case study is what drives the success of the BERT-based model on this challenging task . To answer this we employ our methodology . ( 1 ) We formulate hypotheses by defining simplified variants of the natural logic causal model . ( 2 ) We search over potential alignments between neural representations in BERT and variables in our high-level causal models . ( 3 ) We perform interchange interventions on the BERT model for each alignment . We find that our BERT model partially realizes the causal structure of the natural logic causal model ; crucially , the LSTM model does not . High-level causal explanation for system behavior is often considered a gold standard for interpretability , one that may be thought quixotic for complex neural models [ 16 ] . The point of our case study is to show that this high standard can be achieved . We conclude by comparing our methodology to probing and the attribution method of integrated gradients [ 27 ] . We argue probing is unable to provide a causal characterization of models . We show formally that attribution methods do measure causal properties , and in that way they are similar to the tool of interchange interventions . However , our methodology of causal abstraction analysis provides a framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure . 2 Related Work . Probes Probes are generally supervised models trained on the internal representations of networks with the goal of determining what those internal representations encode [ 7 , 13 , 20 , 28 ] . Probes are fundamentally unable to directly measure causal properties of neural representations , and Ravichander et al . [ 21 ] , Elazar et al . [ 9 ] , and Geiger et al . [ 11 ] have argued that probes are limited in their ability to provide even indirect evidence of causal properties . We now present an analytic example in which probing identifies seemingly crucial information in representations that have no causal impact on behavior . We assume the structure of the simple addition network N+ in Figure 1 . For our embedding , we simply map every integer i in N9 to the 1-dimensional vector [ i ] . The weight matrices are W1 = ( 1 1 0 ) W2 = ( 1 1 1 ) W3 = ( 0 0 1 ) w = ( 0 1 0 ) The output for an input sequence x = ( i , j , k ) is given by ( xW1 ; xW2 ; xW3 ) w. In this network , xW1 perfectly encodes i + j , and xW3 perfectly encodes k. Thus , the identity model probe will be perfect in probing those representations for this information . However , neither representation plays a causal role in the network behavior ; only xW2 contributes to the output . Attribution Methods Attribution methods aim to quantify the degree to which a network representation contributes to the output prediction of the model , for a specific example or set of examples [ 3 , 24 , 26 , 27 , 32 ] . In contrast to probing , the well known integrated gradients method ( IG ) can be given an unambiguous causal interpretation . Following [ 27 ] we define the vector IG ( x ) , for an input x relative to a baseline b , to have ith component IGi ( x ) given by the expression on the left : ( xi − bi ) · ∫ 1 α=0 ∂F ( αx+ ( 1− α ) b ) ∂xi dα = ( xi − bi ) · ∫ 1 α=0 lim →0 F ( xα , ) − F ( xα ) dα Abbreviating the weighted average αx+ ( 1− α ) b by xα , letting xα , be the vector that differs from xα in that the ith coordinate is increased by , and then expanding the definition of partial derivative , this can be written in the form given on the right . The difference F ( xα , ) − F ( xα ) is known in the x y z Dx Dy Dz L2 L1 O X Y Z WS1 S2 ( a ) The causal model C+ ( right ) that first computes S1 = X + Y and W = Z , before computing the final output S2 = W + S1 aligned with the neural network N+ ( left ) with L1 highlighted as the hypothesized location encoding S1 = X + Y and L2 as the location encoding W = Z . 1 2 3 D1 D2 D3 L1 12 4 5 6 D4 D5 D6 L1 15 ( b ) Low-level neural network interchange intervention . The network processes two different input sequences . The neural representation created at L1 for input sequence ( 1 , 2 , 3 ) is replaced by the corresponding representation created for input sequence ( 4 , 5 , 6 ) . 1 2 3 39 12 4 5 6 69 15 ( c ) High-level symbolic computation interchange intervention . The computation processes two different input sequences . The sum at S1 for input sequence ( 1 , 2 , 3 ) is replaced by the corresponding representation created for input sequence ( 4 , 5 , 6 ) . Figure 1 : Our motivating example where we hypothesis that a symbolic computation C+ is a causal abstraction of a neural networkN+ under a particular alignment ( top ) . We can experimentally confirm this hypothesis by conducting an interchange intervention on both the network and the computation with every pair of inputs and evaluating whether the intervened network and intervened computation have the same counterfactual output behavior . We schematically depict an interchange intervention on the network N+ ( bottom left ) and the computation C+ ( bottom right ) with the base input ( 1 , 2 , 3 ) and the source input ( 4 , 5 , 6 ) . Observe that the output of the intervened neural network matches the output of the intervened symbolic computation , so we have success for this pair of inputs . causal literature as the ( individual ) causal effect on the output ( e.g. , [ 15 ] ) of increasing neuron i by relative to the fixed input xα . So , essentially , IGi ( x ) is measuring the average “ limiting ” causal effect of increasing neuron i along the straight line from the baseline vector to the input vector x , weighted by the difference at i between input and baseline . More recently , Chattopadhyay et al . [ 6 ] develop an attribution method that explicitly treats neural models as structured causal models and directly computes the individual causal effect of a feature to determine its attribution . Attribution methods can measure causal properties , and , in that way , they are similar to the tool of interchange interventions . However , our methodology of causal abstraction analysis provides a framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure . Causal Abstraction Our goal is to evaluate whether the internal structure of a neural network realizes an abstract causal process . To concretize this , we turn to formal , broadly interventionist theories of causality [ 25 , 19 ] , in which causal processes are characterized by effects of interventions , and theories of abstraction [ 1 , 2 , 5 , 22 ] where relationships between two causal processes are determined by the presence of systematic correspondences between the effects of interventions . The notion of abstraction that we employ here is a relatively simple one called constructive abstraction [ 1 ] . Informally , a high-level model is a constructive abstraction of a low-level model if there is a way to partition the variables in the low-level model where each high-level variable can be assigned to a low-level partition cell , such that there is a systematic correspondence between interventions on the low-level partition cells and interventions on the high-level variables . There are two properties of constructive abstraction that make it ideal for neural network analysis . First , the information content of partition cells of low-level variables can be determined by the high-level variables that they correspond to . For neural networks , the partition cells of low-level variables are sets of neurons , and our method supports reasoning at the level of vector representations ( sets of neurons ) . Second , the causal dependencies between partitions of low-level variables are not necessarily preserved as causal dependencies between the high-level variables corresponding to these partitions . For example , the low-level model might be a fully connected neural network , whereas the high-level model might have much sparser connections . For neural network analysis , this means we can find causal abstractions that have far simpler causal structures than the underlying neural networks . We provide an example in the next section .

Summary: 1. The authors introduce a "causal abstraction" method for attributing high-level descriptions (combining representations and computations) to task-optimized neural networks. 2. They operationalize their method by performing causal abstraction analysis on a BERT model fine-tuned for MQNLI. They find substantial subsets of an MQNLI dataset for which their interchange intervention predictions are successful between any two pairs of examples.