Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: see main
main_review: This paper aims to formalize and generalize the "interchange" approach to
interpreting computation in deep networks first proposed by Geiger et al.
(BlackBoxNLP 2020). In this experimental paradigm, an abstract causal model f of
the computation performed by a deep network model g is first hypothesized. Next,
alignments between individual variables in this causal model and components of
the deep network are guessed. Finally, the causal model + alignment combination
is validated by testing whether (in roughly Pearlian terms) f(y | x, do(z_f')) =
g(y | x, do(z_g')) for model outputs y and aligned latent / representation
pairs (z_f', z_g') extracted the application of f and g to alternative inputs x'. 

The paper has three contributions:

(1) formalizing this analyis procedure in the language of constructive causal
    abstractions (Beckers & Halpern 2019).

(2) extending it to include a (brute-force) search over alignments rather than
    requiring researchers to fix these up-front

(2) applying it to the MQNLI dataset rather than the MONLI dataset originally
    used by Geiger et al '20

STRENGTHS

- Nice contextualization / formalization of the experimental paradigm of Geiger
  in terms of existing work on causal inference

- Useful example constructions for clarifying the expressive power of the
  probing paradigm in NLP

- Convincing evidence that the MacCartney-style inference is actually
  implemented (in a causal sense) by BERT models trained on MQNLI.

WEAKNESSES

- Approach itself is not new, and empirical findings largely overlap with the
  earlier Geiger paper.

- Most of the interesting content is in the appendix!

I'm quite conflicted about this paper. On one hand, drawing a link between
interchange interventions and the rest of the causal inference literature is
super useful. So are the example failure modes for probing and IG---while these
are simple constructions, I haven't seen them written down before and it would
be nice to have something to cite! But instead of focusing on this content, the
paper is mostly about analyzing the behavior of a single model trained on a
synthetic NLI dataset trained with an exotic form of supervision. I don't think
this finding is generalizable enough to be of interest to a general machine
learning audience. Meanwhile, the other methodological innovation on top of the
original Geiger paper is the addition of a search over candidate encodings,
which this paper largely sidesteps by manually pruning the search space by exploiting the regular structure of the
synthetic data. I think a stronger version of the paper would either (1) promote
appendices A and G to the main paper (and talk a little more about how we can
use other insights from causal inference to guide analysis of deep nets), (2) do
an empirical analysis of more than one model (e.g. a model trained on ordinary
NLI data), or (3) say a little bit more about how to automate the search (e.g.
by first building the matrix of correlations between causal variables and model
representations, and using these to prune the search space explored by
interchange interventions).

However, I'm absolutely willing to be persuaded that the paper is substantial enough to
accept in its current form, and would be interested in hearing more from the
authors about what they view as the delta (formally or empirically) relative to
Geiger '20 and the significance of the MQNLI results.

Some other thoughts:

SECTION 3

While the example in Fig 1 is useful, it's a little odd for a NeurIPS paper to
contain *only* a worked example, and leave the general framework that is being
exemplified out of the paper entirely. Building on the general comment above, is
it possible to interleave the discussion in section 3 with any concrete
discussion of constructive abstractions / interchange interventions in general?
To get space back, Fig 3 could be shrunk, and the nearly 2 pages of related work
spread across sections 2 and 5 could probably be condensed a bit.

Since interchange success and clique size tell different stories
about where causally relevant information is located (Fig 4), can we say
anything meaningful about the relationship between the two, or to formal
criteria for constituting a causal abstraction?

SECTION 5

Is max clique size really the right measurement here? I'd obviously rather have
one giant cluster and one medium sized cluster vs one giant cluster and 500 tiny
ones, and you could look at something like the entropy of the induced cluster or
the probability that any two nodes are connected by an edge.

What's the motivation for requiring cliques to have *at least one* impactful
edge as opposed to something stricter? (e.g. start with the graph in which
*every* edge corresponds to a label-changing substitutions in the true
causal model, then just measure the fraction of these edges where interchange
causes a corresponding label change in the neural net).

MISC

Extremely fussy: use \citet in expressions like "the dataset of [CITE]"

limitations_and_societal_impact: see main
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 1. The authors introduce a "causal abstraction" method for attributing high-level descriptions (combining representations and computations) to task-optimized neural networks.
2. They operationalize their method by performing causal abstraction analysis on a BERT model fine-tuned for MQNLI. They find substantial subsets of an MQNLI dataset for which their interchange intervention predictions are successful between any two pairs of examples.
main_review: The paper presents a very interesting interpretability method which goes beyond correlational and weaker causal analyses in the literature. However, I am concerned about a few practical issues that might affect the generalizability of the method. These are addressed in the "quality" section below.

I would be happy to see work like this published, and would recommend this paper for acceptance if the authors can adequately address these concerns. I may have misunderstood some parts of the analysis and would happily accept corrections as well. 

## Quality

1. My main issue is with the lack of a comparison / baseline in the MQNLI evaluation. It's hard for me to know how to distinguish between a successful and unsuccessful analysis here. Concretely: at what level (and on what measure) would we confidently say that a proposed causal model is / is not a causal abstraction of the NN?My first thought on seeing the numbers in figure 3 was that this was not a success: for some syntactic constituents there are moderately-sized cliques of examples on which intervention is successful. But this doesn't hold more generally for the dataset, and the clique metric seems to degrade quickly as we move to the higher constituents covering larger spans of the input sentence.I would like the authors to address whether and how we can understand when a causal abstraction analysis has succeeded. (Are there reasonable a priori points on the evaluation metric scale which signal success? Are there statistical tests, e.g. a permutation test involving alternative forms of the causal model, which can give us a firm basis for judging success?)
2. A substantial practical issue with this method is inferring the alignment between variables in the causal model and "locations" in the neural model (I believe, following lines 89--90, this means subsets of neurons). As far as I can understand it, the authors did not actually run an alignment search in the MQNLI experiments, but rather stipulated that particular subsets of neurons should correspond to causal model variables (lines 218--231).I believe the generalizability of this method is severely limited by this alignment search method, as I understand it. There is very little information in the main text suggesting how this was carried out in their experiments, or gestures at how this would transfer to the general case. I enumerate my specific concerns below.I would like the authors to address these questions in the paper, and ideally demonstrate through their experiments that a full alignment search process can be tractable in a large-scale model.
  1. Such an alignment search involving models with complex causal structures would be expensive in terms of computation and data. For any node in the causal model, we would need to search among all subsets of neurons for correlations between causal model variables and neural representations. (I'm assuming correlation would be the measure, though I don't believe this is stated in the paper.) This would naturally require an extra development dataset separate from the one used for interchange experiments.
  2. The success criterion for alignment search is also an important question not addressed in the paper. At what degree of correlation do we conclude that an NN location is related to a causal model variable?

## Originality

I believe this principled approach to causal intervention is a novel contribution to the literature. However, it might be interesting to explore the relationship between this general method and related specific causal attribution analyses in NLP with a more implicit causal model -- for example, \[1\] run intervention analyses on language models to test how plurality and syntactic depth are represented.

## Minor comments and questions

1. 222--224: Why are there "two descendant tokens" for constituents such as Adj, Adv, V? As I understand from the example figure, you only consider inputs with one token for these constituents.
2. Fig. 3: which layer of the BERT model is used for these experiments?
3. More generally, I don't understand the relationship between fig. 3 and the "interchange intervention" experiments shown in fig. 4\. Fig. 3 shows the largest subset of examples on which C is a causal abstraction of the BERT model -- is this not the same as the size of the largest clique in the example graph with edges denoting interchange intervention success (i.e. the values shown in fig. 4a)?
4. I like the idea of using graph metrics for broad quantitative evaluation of the metric. But why is clique size the right metric? This seems to set a very high standard. Why not also incorporate a more permissive metric such as average degree, or visualize the full distribution over node degrees?

## References

1. Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., & Baroni, M. (2019). The emergence of number and syntax units in LSTM language models. arXiv:1903.07435 \[cs\].[http://arxiv.org/abs/1903.07435](http://arxiv.org/abs/1903.07435)
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a structural analysis tool grounded in the causal abstraction theory, through which it attempts to identify the causal properties governing the inference of the given task. This method is an alternative to probing, which involves training a supervised classifier to identify the causal relationships in many NLP tasks. Using this causal abstraction analysis tool, the authors study MQNLI corpus, which is a specifically constructed tree-structured corpus for natural language inference, grounded in natural logic. Through various experiments, the authors observe that BERT-based model approximately realizes the causal properties of the data generating process better than the baseline LSTM model.


main_review: The paper presents novel, original method of identifying the causal mechanisms in play in neural networks for reasoning. This paper provides clear distinction of its methods from the prior work of probing literature and attribution methods using integrated gradients. The paper also provides good coverage of the related work which helps the reader to understand the necessary background.

The paper begins with a formal description of the hypothesis-alignment-intervention process. Certain parts of this section is not clear. For example, it is not clear (in L158 to L160) why z is not causally represented at L1 even if L1 depends on z in the MLP. The comparison with attribution based methods following the above argument is not clear, as a scalar value of 0 assigned with L1 with z would represent the same non-causal relationship the author is alluding to.

The paper is slightly hard to follow as it lacks clear examples. While the paper provides a good toy example describing the hypothesis formulation, alignment search and intervention process, those examples are not aligned with MQNLI pairs. In Figure 2, the causal structure is provided but it is not clear which pair it aligns with. Hence, following the arguments presented in Section 5.1 is hard. In Section 5.1, the interchange intervention possibly converts subsequence $e$ ("every happy person", "some happy baker") to ("every happy baker", "some \epsilon baker"). It is unclear how changing this subsequence renders the output label to neutral. A better example (including the entire sentence, not only the subsequence) would be helpful for the reader to understand the intervention process.

The analysis presented in the paper is also restricted to MQNLI - possibly due to the fact such interventional analysis would require practitioners to have access to the data generating process of the underlying dataset. This reduces the significance of the work a bit, although the sound theoretical motivation provided in this paper balances out the lack of downstream usability from an academic point of view. However, the paper only investigates two models - LSTM and a pre-trained BERT base - which reduces the significance of the results even academically. Out of these two models, LSTM model is much weaker compared to the BERT model - 46.32% compared to 88.50% - which makes the baseline of the causal abstraction argument much weaker anyway. The success of the high score of the BERT-based model is mostly due to pre-training, which the paper does not identify in the discussion. A better comparison would have been to isolate the non-pretrained models (including a BERT trained from scratch on MQNLI) and the pre-trained models (BERT and its family).

The alignment search phase raises a lot of questions as well. It is unclear in case of BERT where the representations are taken from. Due to multiple cascaded attentions in Transformer models, it is hard to pin down the representation of a word being the same location after several layers of self-attention. Thus, it is unclear how the hidden representations for tree nodes are extracted from BERT. Also, it is unclear how the word phrase representations (from nodes higher up in the tree) are used from BERT in the experiments.

The results and discussion section needs polish as well. While it is understandable that the causal intervention tool provides better signal than probing tasks, less discussion is provided on what exactly do we learn from the analysis of the alignment - i.e how important it is on the final prediction.

Overall, having lack of model exploration and restriction to MQNLI dataset does not exhibit condfidence that the proposed methodology can work in practice on other tasks. However, the theoretical experimental setup are still valuable for future practitioners to develop interventional analysis tools.

Edit: I have increased my score to 6 following the discussion with the authors, where they provide additional experiments and commit to clarify the examples and results.

limitations_and_societal_impact: - The authors have not indicated any limitations of their study in the paper.
- In my opinion, the work should not result in any negative societal impact or ethical concerns. Thus, it does not require any further ethics review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for identifying causal structures captured within neural networks. The proposed method does so by aligning a causal model with the neural networks’ prediction using interchange interventions. The authors demonstrate the efficacy of this method using the MQNLI corpus, where they show that a BERT-based model is compatible with a natural logic causal model such as the one originally used to generate the data. 

Thus, the main contribution of the paper is a methodology for testing hypotheses regarding underlying causal structures learned in neural networks, which works well on an automatically generated natural language inference dataset.


main_review: This paper presents a complete process for generating hypotheses regarding possible causal structures governing the behavior of neural networks, and for testing their fit to the model's observed behavior. While it does require advanced prior task-related knowledge regarding possible causal structures which might be learned by the neural network, the proposed method is useful for tasks which could rely on such structure, such as many syntactic and semantic tasks in NLP.

The paper presents some originality in its attempt to provide causal abstractions and not only estimate causal effects, it is essentially a method for testing specific hypotheses regarding the interaction of model components. For researchers who are interested in estimating causal effects on neural networks and not in what specific model components (i.e. layers, neurons) are driving behavior, there are existing causal explanation methods that have already suggested possible solutions.  

Still, this is an interesting paper that provides a clear and sound evaluation, and can surely contribute to researchers interested in uncovering where in the model a specific reasoning might occur (similarly to Causal Mediation Analysis). If rephrased as a contribution for reasoning about a specific model reasoning structure and not as a contribution for causal effect estimation, I think that this paper has a clear contribution to the literature. The current state of the paper, where it is presented as a tool for generating causal explanations (such as CausaLM), is somewhat misleading. Without comparing examples to counterfactuals, it is simply not true to state that this method truly uncovered causal effects.


limitations_and_societal_impact: This paper can potentially have a positive societal impact, as understanding the causal structure engrained in neural networks can uncover hidden biases and fairness issues which govern the model’s behavior. 

There are certainly limitations to this work, such as the reliance on a user’s ability to design a causal graph which might explain the neural network, and the inability of the method to provide causal identification besides fitting some potential model to the data. However, the paper provides a way forward for potentially explaining where causal knowledge is captured in neural networks.
review_title: null

Meta-review: I don't think this paper requires a long metareview. They introduce a method for aligning the predictions of a network with an induced causal model of the task's "dynamics". They show it works on some NLI tasks which is respectable (although I would have liked to see some other domain too... maybe something to do in future work!). The reviewers were excited about the work, and several were ready to champion its acceptance after discussion with the authors. I recommend acceptance, and hope that ACL's loss here will be NeurIPS' gain.
Decision: Accept (Poster)