Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: Apologies for the delay in responding.  

I maintain that the paper is lacking precise claims and thorough evaluation of those claims.  Many of my concerns remain after the authors' response.  I believe more substantial changes would be needed for this paper to meet the standards of ICLR.  I understand that the revision period is over, but I believe the changes needed were already too large to address in a revision, and my opinion is that the paper should be substantially rewritten and experiments should be expanded before resubmitting.

I also don't believe the reviewers have properly addressed this concern of reviewer wPfU:
> The introduction poses three advantages of mechanistic interpretation, "better predict out-of-distribution behavior, identify and fix model errors , and understand emergent behavior". However, none of them is demonstrated in the experiments, and it is not clear to me how they are enabled.

The new OOD results are a step in the right direction, but I think much more would be needed to sufficiently support the authors claims:
- How does the model determine that an example belongs to the IOI "task"?  This seems like a critical part of explaining its behavior on these examples.  Does it perform this same behavior even when it shouldn't?  e.g. "When Maury and John knelt down to pray, John gave thanks to" should probably be completed with the name of some diety, not "Maury".  Does it?  What happens if there is additional context prepended to the example?  What happens if there is some other clauses inserted (that don't contain S or IO, or contain both of them some number of times, etc.)?
- I would also expect an analysis of cases where the model predicts something other than the IO (why does this happen?  Are these "errors" random, or if there some sense to them?)
- I believe there should be more work put into validating the methods used to evaluate faithfulness, completeness, and minimality.
- These are just three ideas; I think it is on the authors to devise and present a more convincing set of experiments.

On a related note, contra reviewer iR9D, the promise of mechanistic interpretability is quite important to discuss, since the paper claims in the abstract that: "Our work provides evidence that a mechanistic understanding of large ML models is feasible, opening opportunities to scale our understanding to both larger models and more complex tasks."  I am not convinced by this claim, and remain unconvinced by the central claims of the paper, which I think are overstated; in my mind, the evidence provided by this work is all still quite circumstancial.  I think this needs to be more clearly acknowledged in the writing, and also affects the framing of the contributions.

Given that we have substantial disagreement about the paper I suggest we have a video chat.


To the authors (RE response):
* Logit difference: my point was that you should care about the probabilities assigned to other tokens as well.  I also don't think the mean difference is a good measure, since errors in opposite directions should not cancel out.  If you want to say not only that C performs the IOI task to the same extent as M, but also *in the same way*, then you need to look at more fine-grained measures of behavior. 

* Name mover heads: I'm still not clear on this point; do they copy whatever the attend to, or not?
review_title: Still in favor of rejection -- we should do a video chat.

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Update after rebuttal** After reading the other reviews and the authors' responses I remain in favor of accepting the paper. I think wPfU, jy1a raise some very valid issues, but some of the issues raised concern the viability of the approach/research field of mechanistic interpretability in general, rather than the concrete results presented in the paper - which I think is valid to be discussed but to me is a less relevant criterion to judge one particular paper/analysis. jy1a raises additional relevant criticism specific to this particular paper, and in my opinion the authors have addressed that criticism sufficiently (I'd be keen to hear jy1a's opinion on this).

The paper is a case study analysis for identifying the mechanism / “algorithm” that GPT-2 small uses for solving a particular language task (Indirect object identification; indirect reference to one of two previously named persons in the same sentence to be precise). Backtracking from attention matrices - the paper essentially identifies groups of attention heads that each have an intuitive functionality, and, when combined, solve the task. To strengthen the identified circuit the paper defines metrics for faithfulness (circuit alone can explain good score on task), completeness (potentially duplicate parts of the circuit have been identified too), and minimality (no superfluous parts, that play no role in functionality and are not duplicates, have been identified) and confirms that the identified circuit fulfills all three notions. To demonstrate that the identified circuit is causally responsible for the functionality Knockouts (rendering parts of the network irrelevant) and Patching (splicing in activations from a different input into parts of the circuit, resembling a more fine-grained intervention than Knockouts) are used.
strength_and_weaknesses: **Main contributions, Impact**
1) Well executed case study for identifying a mechanistic understanding of the circuit that solves the analysis task in GPt-2 small. Impact: medium - the techniques introduced to identify the circuit are not entirely novel, but are certainly put to good use; the identified circuit is interesting, but it is unclear whether the identified components will play a significant role in other analyses or a general understanding of strategies used by transformers to solve tasks.

2) Use of interventional analysis to establish causal relevance of identified circuit. This is crucial for any serious attempt at *explaining* network functionality; an explanation must lead to falsifiable (causal) predictions. While Knockouts are a standard technique for crude interventional analysis, the paper also uses more fine-grained Patching, demonstrating its usefulness for analysis of complex circuits. Impact: medium - falsification (or establishing causal relevance) is crucial for hypotheses established via interpretability/explainability techniques. While Knockouts are certainly not novel, and I am not sure whether Patching has been used in this precise context before (but would be somewhat surprised if nothing similar has been reported before), the consistent and effective use of the techniques in the paper serves as a good example for future analysis.

3) Analysis of the identified circuit in terms of faithfulness, completeness, and minimality. I particularly like that the paper aims at making these notions formal and quantitative, and reports quantitative results. For instance, without a check for completeness, the backup heads would likely not have been identified - which is an interesting result, highlighting the potential degrees of (complex) redundancies in neural circuitry. Impact: medium to high - the measures seem sensible, though perhaps not maximally universal, and I think that there is a good chance that they will be picked up by the community.

**Strengths**
 *  Well executed analysis case-study, showing that it is possible to open the “black box” or neural circuitry
 * Establishing of causal relevance of explanation, as well as quantitative definition and evaluation of faithfulness, completeness, and minimality of explanation
 * Serious attempts at verifying the correctness of the explanation, and “poking holes” to make sure the results are reliable.

**Weaknesses**
 * The biggest weakness to me is that the results are specific to a particular network (and as the paper suggest do not seem to easily transfer even to GPT-2 medium, a very closely related architecture) and task-instance (which allows for a fairly simple solution strategy as discuss at the beginning of Section 3). There is not much that can be done here - and I actually appreciate focusing on one meticulous analysis rather than analyzing a handful of tasks in a shallow fashion - but the limited generalizability of the results is one weakness of the work - I am more optimistic in terms of generalizing the methodology presented, though not all of it is entirely novel.
 * The second largest weakness to me is that the paper currently does not analyze whether the identified circuit has additional functionality (e.g. is used or even crucial for other tasks). Given the current writing one is inclined to (probably wrongly) infer that the (complete and minimal) sub-circuit found might exclusively serve to solve the particular task presented and that the sub-components also have clear and exclusive functionality (suggested by nomenclature such as ‘Name Mover heads’). I find it quite plausible that the sub-circuit and individual parts of it serve quite different functionality in different tasks, and are also partly involved in other (maybe even quite unrelated) tasks - e.g. there is no ‘name moving’ in an arithmetic task even though the same heads might potentially be involved. This exclusivity is not explicitly claimed in the paper, and perhaps not intended by the authors, but it would be great to see this discussed, and perhaps empirically analyzed (see improvements). Essentially, this is a word of caution regarding the recurrence of the well-known “polysemanticity” problem coined in C. Olah’s earlier work - perhaps the term “polyfunctionality” is more appropriate for circuits instead of individual neurons.

**Improvements**
1) Please discuss the possibility that the identified circuit and sub-parts (groups of heads) might be involved in other tasks where they might play a quite different functional role. Ideally (though this might be a stretch for a rebuttal phase) analyze whether knocking out the (minimal) circuit leads to performance losses in other, unrelated tasks. (I do not consider empirical results here necessary for publication).

2) Please discuss limitations of the current work slightly more prominently (the discussion already raises some points, but it would be great to see a strong and critical discussion of the limitations).

3) Please make sure that all details to reproduce the analysis are given in the appendix.

**Minor comments**

A) Though section 4 is fairly well written, it did take me two passes to grasp the quantitative metrics introduced for faithfulness, completeness, and minimality. I am not sure if pseudo-algorithms or graphical intuitions would be more helpful, but readability (or perhaps simply notation) could be improved a bit in Section 4. Not a major issue though.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written, except section 4 which took me a second pass to fully grasp. The quality of the analysis is very high, hypotheses are derived sensibly and verified in multiple ways. As in many analysis case studies, there remains a certain amount of irreducible complexity, but e.g. the beginning of Section 3 is very helpful to set up the right intuitions. To the best of my knowledge the results are completely novel, and while some of the analysis techniques have been used before they have been combined in an exemplary fashion. Given the detailed description I think the analysis should be fairly easily reproducible, but I am not sure if all of the necessary details to do so are currently in the paper - maybe I missed if there is a planned code-release.
summary_of_the_review: The paper is a strong example of an analysis case study to identify a simple algorithm for solving the indirect object identification task learned by GPT-2. Such case studies are rare, but luckily increasing in number - I personally think it is important to show the community to which degree it is actually possible to understand the internal (mechanistic) function of neural networks. I do not have any major concerns other than expanding the discussion around limitations and specificity of the findings to this particular task (same circuit or parts of it might be involved in other tasks, potentially even with different qualitative functionality). I currently think the work is polished and ready for publication and of interest to a fairly large audience.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work authors discovered specific transformer circuits for a natural language task called the indirect object identification, This task requires the LM to identify the indirect object and needs simple logical reasoning. The significance of this paper is that authors identified a large circuit in GPT-2 small that performs a relatively natural, linguistically meaningful task . Furthermore, the authors present a set of criteria that aim to measure the structural correspondence between the circuit and the model. 
strength_and_weaknesses: Strength. 
Authors demonstrated interpretability techniques and criteria through example, making the abstract task of interpretability concrete. 
Clarity: the paper is well written and easy to read.
Quality: The authors have formulated a set of three criteria to measure interpretability and have showed empirical evidence that their circuit have significant improvements compared to a naïve circuit. 
Clear limitation: Authors are also being very clear on the limitations of their circuit, showing that their circuit does not do well with the completeness criterion. 

Weakness
Failing the completeness criterion could be seen as a weakness but I tend to agree with the authors that worst-case completeness is too high a bar and could be explored in future works. 

clarity,_quality,_novelty_and_reproducibility: Apart from what was discussed in the previous section, I have a question that the authors could perhaps clarify. I’m not too familiar with the literature, is the indirect object identification task an original contribution by the authors or is it from prior works? I didn’t see any reference but it was put under the background section, so I’m a bit confused. This task is a great task to study interpretability as authors rightly argued, as it is linguistically meaningful yet it is very clear what simple interpretable algorithm can perform well on this task. 

Another potential improvement to clarity is adding a summary or overview of all methods used to identify circuit components in the introduction section, or somewhere else, so that reader can have a holistic idea about what is the complete set of techniques used to identify all the circuit components. Maybe expand a bit more on figure 1 in the introduction, I think that could be helpful in terms of readability. Without this overview, readers might need to put together this summary themselves as currently the organization of the circuit discovery is by circuit components rather than techniques used. 

Nitpick: a typo in the last sentence of 4.3: “… worst-case completeness criteria” -> “… worst-case completeness criterion”

summary_of_the_review: I’d recommend acceptance, as I think this paper is a solid contribution to the interpretability of language models in natural tasks that are linguistically meaningful.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a method for mechanistic interpretation of neural networks, specifically transformer models. The high-level idea is to derive a small subnetwork that can simulate the functionality of the full neural network while satisfying faithfulness, completeness, and minimality criteria. The paper empirically analyze the subnetwork using the GPT-2-small model trained on an indirect object identification (IOI) task. The proposed method is shown to identify several steps needed for the IOI task. 
strength_and_weaknesses: Strengths: 
1. Mechanistic interpretation is a new direction of interpretability research. 
2. The analyzed model is GPT-2-small, of significant complexity. 
3. The writing is clear and easily understandable. 

Weaknesses: 

Even though the paper tackles a very new area, for which I am fine with some rough and underdeveloped ideas, the general method suffers from too many flaws for me to "see the light at the end of the tunnel". 

1. The method requires an _a priori_ known model working mechanism (e.g. the heuristic in Sec. 3), and basically maps different subgraphs (i.e. circuits) to various parts of this mechanism. This seems to be a very strong and possibly unjustified assumption: for all but the very synthetic tasks such as IOI, explicit rule-based classifiers (from the 80s and 90s) perform very poorly, suggesting that human-designed heuristics are most likely not how high-performing models work in general. Second, even if humans have a good grasp on one valid heuristic, it is likely that neural networks can use something totally different, due to accidental spurious correlations in the dataset, a problem known as shortcut learning [1]. 

2. The IOI task has three simple heuristic steps, mapping to the three major circuit classes. For more complex tasks, there may be tens or even hundreds of heuristics, especially for multi-class classification with a large number of classes (e.g. ImageNet). Identifying and interpreting them could be very cumbersome. 

3. The introduction poses three advantages of mechanistic interpretation, "better predict out-of-distribution behavior, identify and fix model errors , and understand emergent behavior". However, none of them is demonstrated in the experiments, and it is not clear to me how they are enabled. 

4. The paper assumes that attentions work "as expected", in that, e.g. "Name Mover Heads, by default, attend to previous names in the sentence, but due to the S-Inhibition Heads attend less to the S1 and S2 tokens". There have been numerous evidences suggesting that attentions may not be faithful interpretations [e.g. 2, 3, 4]. How would they be reconciled with the focus on attention in this paper? 

[1] https://arxiv.org/abs/2004.07780

[2] https://arxiv.org/abs/1902.10186

[3] https://arxiv.org/abs/2004.03685

[4] https://arxiv.org/abs/2104.14403
clarity,_quality,_novelty_and_reproducibility: * Clarity: the writing is highly clear
* Novelty: the method is novel, as far as I know. 
* Reproducibility: although code is not included, I believe that this paper and the method should not be hard to reproduce
* Impact: quite limited, which is my major concern as discussed in the weakness section. 
summary_of_the_review: Overall, currently I am not convinced that mechanistic interpretation, as demonstrated in this paper, is a promising direction for understanding black-box models. Details are discussed in the weakness section. Therefore, I vote for rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper finds a "circuit" (in this case, a subset of attention heads), C, that is meant to explain how GPT-2 small solves a particular synthetic task named Indirect Object Identification (IOI).
This task composes 15 templates of forms like "Then, [B] and [A] went to the [PLACE]. [B] gave a [OBJECT] to [A]".
The goal is to predict the name [A] at the end of the sentence, given all of rest of the sentence as context.
The model's behavior on other types of inputs was not evaluated.

The  claim is supported by experiments which aim to establish that C meets the authors' criteria of being faithful, complete, and minimal.
Some of their experimental results challenge this interpretation, however (see Sections 3.4 and 4.3).

The authors also describe how they go about identifying the proposed circuit, describing the techniques of knock-outs (i.e. looking at a subgraph of M), and patching (substituting different activations at some nodes of the computational graph while computing a forward pass).

The novelty of the proposed criteria and techniques is not clearly stated (except completeness, which they state they introduce).

strength_and_weaknesses: 
Strengths:
- The notions of faithfulness, completeness, and minimality seem like a promising approach to rigorously evaluating mechanistic interpretability claims.
- The paper covers a lot of ground; besides (apparently?) introducing these notions, they also propose methods for looking for circuits, and an interpretation of a particular model on a particular task.
- The interpretability efforts themselves are potentially useful as a case study that could guide or inform future work.
- The topic is neglected and potentially significant: mechanistic interpretability is worthy goal and not much has been published on it.

Weaknesses:
- The three claimed contributions are not sufficiently well supported.  
1) RE "(1) We identify a large circuit in GPT-2 small that performs indirect-object identification (Figure 2 and Section 3)":
The discovery of the Backup Name Mover Heads suggests that other important bits of circuitry might've been missed, and that the model's behavior on this task might not be very well captured by the circuit.  Perhaps there are also Backup Backup Name Mover Heads, etc.?  Furthermore, as noted, computing the completeness score is intractable, the approximations produce somewhat inconsistent results, and no effort is made to determine which ones are reliable, excepting Section 4.3, where it is indirectly suggested both that the greedy method is both both better at estimating the completeness score, and that the 
This is methodologically backwards: it assumes the interpretation is correct, and uses this as a basis to challenge the definition of the completeness score.
2) RE "(2) Through example, we identify useful techniques for understanding models, as well as surprising pitfalls":
This is a vague contribution and does not make a clear statement of novelty... which techniques are novel?  Or is it their application to "understanding models" that is novel?  What are the "surprising pitfalls"?
3) RE "(3) We present criteria that ensure structural correspondence between the circuit and the model, and check experimentally whether our
circuit meets this standard (Section 4)":
It has not been established that these criteria ensures structural correspondence; indeed, the notion of structural correspondence is not defined.  The criteria don't seem quite conceptually correct (see below).
The experimental results are not conclusive, and their interpretation remains subjective.  The checks that are performed seem ad-hoc and incomplete.
- There are a few limitations/concerns that deserve more discussion:
1) There seems to be an unstated assumption that circuits are they a meaningful way of understanding a model's behavior.  This assumption doesn't seem to be evaluated or supported by this work. 
2) The paper seems optimistic about applying a similar approach to larger models, but doesn't discuss the challenges of scaling this approach (models may have a very high level of inherent complexity, mechanistic interpretability may require a prohibitive amount of human labor needed, even if models are simple enough to be understood mechanistically).  
3) It seems the interpretation provided in this work can (at most) only tell us about how this model handles this very specific synthetic task.  How does this lead to a meaningful or practical understanding of how a model works more generally?  Or is the idea to study every behavior of interest in isolation? 
- Overall, the writing is not careful in its claims (e.g. Section 3.2 starts with "Given that the Name Mover Heads are primarily responsible for constructing the output", but this hasn't been established).
- The definitions of faithfulness, completeness, and minimality do not seem fine-grained enough to fully capture mechanistic similarity between the circuit and the model.  I would expect the definition of F to measure the average difference in logits instead of the difference in average logits, and measure the difference in the vector of logits or probabilities instead of just the difference between IO and S tokens.  The work does not justify or discuss this chosen definition of F or alternatives.
- The paper should provide more motivation for this line of research.  What are the pros and cons versus other approaches to interpretability?  What is the value of understanding how the model behaves on this very specific set of inputs?  Can we draw more general conclusions about the model's behavior?  Or what is the end game?  What are the prospects for automating mechanistic interpretability?
- The definition of the task here is extremely narrow.  The model should be evaluated on OOD examples that similarly require outputting the correct name.  For instance, what if you add distractor text with other names in it to the context?  This work does not address how the model determines whether it should perform this task / engage this circuit.  Is there circuitry for recognizing what the task is?


Questions:
- Is the goal only to determine how the model solves this one task?  If some of the components of C played a completely different role in different tasks or on OOD examples, how much would that challenge the interpretation?
- The approach here seems based on finding attention heads that have large individual contributions, but couldn't there be a long-tail of other circuitry contributing to the model's behavior (e.g. in figure 6: what if you include more stuff?  why is .06 / 3% the magic cut-off)?  Section 3.4 suggests this might be the case.  Does this represent a fundamental challenge to the project of finding a circuit?  Consider an analogy with "typicality", where most of the probability mass of a distribution (such as a high-dimensional Gaussian) does not come from the most likely samples.
- For the "name mover heads", what does it mean that the OV matrix is a "name copying matrix"?  If they "copy whatever they attend to" (and not just names), is the claim that they are just approximating the identity matrix?  This could be investigated more.
- The role of the ABC distribution wasn't clear enough to me; can you clarify why it is used?

Minor comments:
- The names of the scores are counter-intuitive; e.g. the completeness score measures *in*completeness! 
- The writing sometimes pretty unclear, e.g. "v can significantly recover F"; last 2 paragraphs of 3.3.
- "serial circuits" appears undefined.

clarity,_quality,_novelty_and_reproducibility: Clarity:
- The writing was mostly clear, but the organization and style were not.  It lacks rigor in places.
- The work doesn't clearly outline its methodology and methodological assumptions.  
- Poor organization: Section 3 doesn't clearly distinguish between experimental proceedures, results, and interpretation. Technical definitions and techniques are presented in a somewhat adhoc fashion throughout the paper. 

Quality: 
- Setting aside the issues with the work mentioned elsewhere in the review, the quality of the work is fairly high.  There are many interesting experiments and results.
- There are, however, insufficient ablation studies / sanity checks, and several of those performed question the validity of the interpretation (Section 3.4) or methods (Section 4.3).

Novelty:
- The novelty of the proposed criteria and techniques is not clearly stated.
- The paper lacks sufficient discussion of related work.  It doesn't discuss alternative approaches to interpretability, how their approaches and aims differ from mechanistic interpretability, or the pros/cons of mechanistic interpretability vs. other approaches.  It also doesn't make sufficiently clear claims about the novelty of this work vs. previous work on mechanistic interpretability. 


Reproducibility:
- It wasn't clear how the model was trained.  Was it pretrained on natural language?  Fine-tuned on this task?  Trained only on this task?
- Identifying C seems to involve various judgment calls.  This is a fundamental barrier to reproducibility, which should not disqualify the work.  But the authors should be clear in identifying such judgment calls and stating the basis for making them.  How easily could the approach here be generalized to other tasks or models?  Can the authors provide a recipe that a reader could follow?  Or otherwise outline how this work could facilitate future work on mechanistic interpretability?
summary_of_the_review: 
The paper doesn't provide sufficient support for its claimed contributions.
In particular, I'm not convinced that the proposed circuit is an accurate representation of the model's behavior.
While it makes some meaningful progress in terms of techniques and methodology, more work is needed to justify and solidify these approaches; and the presentation also needs work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper presents an analysis for mechanisms that GPT-2 small uses for solving a particular language task, indirect object identification. 

Strength: 
1. The paper is a well executed analysis case-study in a relatively new/small mechanistic interpretability area . 
2. The analysis of the formulated criteria (faithfulness, completeness, and minimality) in the identified circuit makes these notions formal and quantitative. 
3. The methodology could be potentially interesting to the community to which degree it is possible to understand the internal function of neural networks. 

Weakness: 
1. As reviewers pointed out, some claims are overstated or not well supported. 
2. Limitation of the work could be expanded to reflect all reviewers' concerns. 
3. The analysis is only done for a particular network and a task-instance.

The major concerns from the reviewers are the current limited limitation section and a few not-well supported/overstated claims in the paper.
Request to the authors: Please update the paper to have a stronger and more critical limitation discussion, as well as substantially change the writing to justify all claims/assumptions (or not to overstate claims) in order to reflect reviewers’ comments. 
justification_for_why_not_higher_score: There are a few critical issues identified by reviewers:
1. Substantial changes needed for a stronger and critical limitation discussion.
2. There are a few not-well supported/overstated claims in the paper. 
justification_for_why_not_lower_score: Despite the fact that there are some critical issues raised by reviewers (also distinct scores), this paper is still an interesting and solid case study in the relatively new/small area – mechanistic interpretability. The methodology could be potentially interesting to the community to which degree it is possible to understand the internal function of neural networks.
Decision: Accept: poster