Title: EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

explainable AI

model interpretability
Abstract: Understanding the internal mechanisms of transformer-based language models remains challenging. Mechanistic interpretability based on circuit discovery aims to reverse engineer neural networks by analyzing their internal processes at the level of computational subgraphs. In this paper, we revisit existing gradient-based circuit identification methods and find that their performance is either affected by the zero-gradient problem or saturation effects, where edge attribution scores become insensitive to input changes, resulting in noisy and unreliable attribution evaluations for circuit components. To address the saturation effect, we propose Edge Attribution Patching with GradPath (EAP-GP), EAP-GP introduces an integration path, starting from the input and adaptively following the direction of the difference between the gradients of corrupted and clean inputs to avoid the saturated region. This approach enhances attribution reliability and improves the faithfulness of circuit identification. We evaluate EAP-GP on 6 datasets using GPT-2 Small, GPT-2 Medium, and GPT-2 XL. Experimental results demonstrate that EAP-GP outperforms existing methods in circuit faithfulness, achieving improvements up to 17.7\%. Comparisons with manually annotated ground-truth circuits demonstrate that EAP-GP achieves precision and recall comparable to or better than previous approaches, highlighting its effectiveness in identifying accurate circuits.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on improving gradient-based circuit discovery techniques that aim to reverse engineer neural networks by analyzing their computational graph. In particular, the authors propose Edge Attribution Patching with GradPath (EAP-GP), which improves over existing techniques such as EAP and EAP-IG by avoiding zones of gradient saturation. The paper reports improvements of up to 17% in circuit faithfulness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The work seems very relevant for the field of Mechanistic Interpretability: Circuit Discovery techniques are constrained by the quality and cost of the metrics for measuring the importance of each edge/node in the neural network computational graph.\n- EAP-GP seems to be a sound proposal that improves over the state of the art (EAP and EAP-IG)\n- The work includes a solid experimental setup measuring EAP-GP vs EAP and EAP-IG on 6 different tasks and 3 different variants of GPT-2 (small, medium, and XL)\n- The authors also compare the computation time for EAP-GP vs EAP and EAP-IG, which is important to understand the tradeoffs between these techniques\n- Writing is very clear and concise for the most part\n\nWeaknesses:\n- No mention of limitations or future work. Are there any edge cases that might be problematic for EAP-GP? Can we make EAP-GP cheaper to run (since it seems to be ~4x more expensive than EAP-IG)? Etc.\n- Unclear if the ground-truth circuits used for comparison (Syed et al) provide a good signal or not on the quality of the circuits found automatically. I.e., if these ground-truth circuits are not very faithful, then EAP-GP having less precision/recall than EAP-IG might be good.\n- A qualitative analysis of some of the circuits found (e.g., for IOI) would be nice. Does EAP-GP find attention heads or MLPs that are interesting or relevant for the circuit and haven't been analysed before?\n- Unclear what the practical applications of having more faithful circuits for AI Safety are. Also, circuit discovery might not scale to bigger SOTA models.\n\nMinor issues and typos:\n- The main body should reference the compute analysis in Table 3, which seems to suggest that EAP-GP is ~4x more expensive to run than EAP-IG.\n- Maybe rename Section 6 to \"Evaluation\" instead of \"Experiments\"\n- Missing an explicit mention of the AUCs for all methods in the precision-recall curves of Fig. 4. \n- Paper goes back and forth with these terms: baseline input, clean input, corrupted input, original input, . Please choose two and go with them, otherwise is quite confusing for the reader.\n- Faithfulness of circuits is talked about in intro (line 51), way before it's defined in the paper, which is slightly confusing if you don't know before hand what it means.\n- Typo in line 125, should be \"graph\" instead of \"subgraph\". I.e., \"Given a model G, which can be represented as a computational **graph**\"\n- IOI dataset is mentioned for the first time in line 165, without any cite. Please add the cite or reference to the appropriate section where this is explained.\n- Typo in line 211: \"iteratively\" twice in the same sentence.\n\nQUESTIONS:\nImprove motivation of the work: What is the theory of impact for this work? What is the utter goal of having more faithful circuits?\n\nCan we make EAP-GP faster so that it's more competitive against EAP-IG?\n\nAre there any limitations on using manually found ground-truth circuits as a part of a quantitative experiment?\n\nDoes it make sense to do a qualitative analysis of the circuits found by these techniques?\n\nLIMITATIONS:\nAuthors have addressed some limitations in the Experiments section, but I think the paper would benefit from having a dedicated section. As mentioned above in the review, authors should also make explicit that EAP-GP is way more costly than EAP-IG (i.e., there's a tradeoff compute/accuracy between techniques).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an improvement to the gradient-based automatic circuit discovery method EAP and its follow-up, EAP-IG. These methods identify limitations in prior gradient-based approaches to circuit discovery. In particular, such methods compute an edge's importance by multiplying the gradient with the change in activation, and, therefore, EAP suffers from the zero-gradient problem: when the gradient approaches zero, the importance score also approaches zero, regardless of the actual significance of the edge. EAP-IG was proposed to mitigate this issue by using integrated gradients and achieved some improvements. However, it remains affected by gradient saturation: if the corrupted input enters a \"saturation zone,\" the gradient again becomes nearly zero.\n\nThe authors proposed GradPath to mitigate this issue. EAP-IG can be seen as applying a \"big jump\" along a straight line, which may lead the method into a saturation region, resulting in a small gradient. Instead of integrating in one step, the authors' method iteratively follows the steepest steps one at a time, producing a \"GradPath\" that mitigates the saturation effect. As a result, EAP-GP achieves better performance than both EAP and EAP-IG.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this paper presents a novel and effective improvement to gradient-based automatic circuit discovery methods, offering a useful mitigation strategy for the saturation effect. The main limitation is that it does not provide any new insights into the interpretation of LMs, which should be the primary goal of circuit discovery. This is concerning, as such work risks turning automatic circuit discovery into a leaderboard benchmarking task, drifting away from its original purpose: understanding the internal workings of LLMs. Nonetheless, this work provides a new tool to the community that shows promise as a generalisable approach. I believe it can support further research and help others develop more effective interpretations in the future.\n\nStrength\n- This paper is very well written. The presentation of the paper is clear and easy to follow. \n- The authors identify that the saturation effect for EAP-IG, which suggests that a less severe version of the zero gradient problem also exists for EAP-IG.\n- This work presents solid performance-based evaluation and showcased the state-of-the-art performance.\nWeakness\n- Although this work concerns interpretability, the paper does not offer new insights into the interpretation of the LM. This is acceptable for the this work, but it would still be beneficial to include at least a discussion of the interpretability aspect. The paper would be significantly stronger if the authors could use EAP-GP's new, improved result to refute previous misconceptions or reveal previously overlooked points caused by the limitation of EAP or EAP-IG.\n\nQUESTIONS:\n- What do you think the better performance of EAP-GP can bring for the interpretability community?\n\n- Can any misconceptions be refuted, or previously overlooked points revealed, based on the improved performance? You do not need to do new experiments, but please pointing out some possible directions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Edge Attribution Patching with GradPath (EAP-GP), an improvement to an existing circuit discovery algorithm, EAP-IG. This work identifies a problem with EAP-IG (i.e., that integrated gradients pushes the model along a path that may traverse through a region of near-zero gradient) which hinders its attributions to connections between model components, and then introduces a simple solution to that problem. Across several evaluations, EAP-GP outperforms existing methods in both faithfulness and consistency with manually annotated circuits.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem that authors identify with EAP-IG is reasonable, as is their solution. Though the method empirically outperforms across a variety of tasks, I am left wondering how ubiquitous the saturation effects are. Section 4 plots only one edge in the IOI task, and could be improved by including similar plots aggregated over all tasks, or more comprehensive discussions of e.g., how many edges face these saturation effects.\n\nAdditionally, further discussion of the greater-than and gender-bias results is warranted. Why does EAP-GP not still produce performance gains at very high sparsities if these tasks are basically solved by simple feature extraction?\n\nWriting: \nThe \"zero-gradients problem\" is introduced early on, but not explicitly defined at that point. This will confuse some readers, so please define.\n\nFinally, limitations are discussed at the end of the results section, but should be put in a limitations section, with a disclaimer that the method is sensitive to hyperparameters\n\nQUESTIONS:\nWhy does EAP-GP not still produce performance gains at very high sparsities if these tasks are basically solved by simple feature extraction?\n\nHow ubiquitous is the saturation effect within a model, across models, across tasks?\n\nLIMITATIONS:\nLimitations are discussed at the end of the results section, but should be put in a limitations section, with a disclaimer that the method is sensitive to hyperparameters", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns methods for circuit discovery, that is, finding the minimal, complete and faithful subgraph of the NN's computational graph which is sufficient to perform the task. It identifies a shortcoming of the existing most popular method: EAP-IG. EAP-IG computes the gradient with respect to computational graph components, at various points in the straight line between activations at the clean input x and at corrupted input x', and uses the gradient's average as the \"attribution score\". Components with the highest attribution score are taken as the relevant subgraph, that is, the circuit.\n\nThe problem is that at many of these intermediate straight-line points, in LLMs, the gradient is saturated to zero. As a result, the attribution scores for many of the subgraph components is too low. The authors propose to instead take gradient descent steps from the clean towards the corrupted input, which has fewer saturation problems. As a result, they are able to improve the faithfulness scores (\"loss recovered\") by 10-17% (depending on the task)\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\nThe paper addresses an important problem (circuit discovery), makes a correct observation about where the most popular method falls short (saturation of gradients) and makes a nice algorithmic contribution (why bother following the straight line?). It also shows an improvement in the benchmarks. It's well written and easy to understand, and overall a pretty good paper.\n\nEven though the following \"weaknesses\" section is very long, I think this paper is a good idea with a fair execution, so overall it's a decent paper. It's just my job as a reviewer to point out mistakes! Together we can make it a good paper.\n\n# Weaknesses:\n\nThe paper has several weaknesses, which I will now point out. I think they're all addressable, though some of them might require more substantial changes than are doable in one round of rebuttals.\n\n## Figure 1 (right) is misleading: this method does not end up at corrupted x'\n\nOne core thread that has unified all circuit discovery algorithms until now is: we want to check what happens when we move edges from their value on clean activations at original input x, to corrupted activations at x'.\n\nThis paper instead optimizes \\gamma (belonging to NN inputs) using gradient descent on ||Model(\\gamma) - Model(x_corrupted)||_2^2 (eq. 8), and uses the iterates of this optimization to evaluate the gradient of outputs withe respect to circuits. Figure 1 (right) says that this method, which starts at x, ends at x_corrupted: the arrow goes from the original to the corrupted point. The text also says so. However, making Model(\\gamma) = Model(x_corrupted) does NOT guarantee that \\gamma = x_corrupted.\n\nIs it actually sufficient for attribution that we are at an input which is as bad at the behavior as x_corrupted (x')? Perhaps, but this discrepancy should be made explicit and discussed. Figure 1(left) clearly shows that the gradients are *not* the same for x_corrupted (x') and for the found point via EAP-GP: the right-most points have gradient ~2.00 for EAP-GP, and ~0.00 for EAP-IG (which is guaranteed to end up at x').\n\nThis discrepancy needs to be explicitly written down or I cannot in good conscience accept the paper.\n\n## No explicit exploration of whether the method is better due to dealing with saturation, or some other reason\n\nThere's anecdotal evidence in Fig. 1, but nothing systematic. Showing something like the average norm of gradients during the path integration, and comparing between EAP-GP and EAP-IG, would go some way to show that.\n\nYou cannot claim that your method is better because it reduces gradient saturation, without actually showing it empirically. This also needs addressing\n\n## Syed et al.'s benchmark is not representative of modern models.\n\nIn 2023, when people were working on circuit discovery for the first time, it was understandable to use the circuits in GPT-2 small as the targets of the benchmark. Nowadays, however, the models used have changed a lot in size and a little bit in architecture: using rotary embeddings (RoPE) instead of purely learned embeddings, and RMSNorm instead of LayerNorm. There have also been published benchmarks such as the [Mechanistic Interpretability Benchmark, which has a circuit discovery track](https://arxiv.org/abs/2504.13151). Why not use those, and instead stick to the relatively unreliable evaluations from Syed et al?\n\nThis is should ideally be addressed too, but I'm willing to budge on this point.\n\n## Prior art on saturation: AtP* from early 2024\n\nThere is key prior art on dealing with saturated activations, specifically caused because of the softmax on the QK: AtP*, from early 2024 (https://arxiv.org/pdf/2403.00745). See especially Figure 3 in that paper. This present paper does not compare with AtP* at all, even though they also present a method to deal with the saturation.\n\nIn comparison with AtP*, the present paper is more generally applicable because it makes no assumptions about the architecture of the model. But, being applied to transformers, the saturation is almost certainly also because of the softmaxes, so all of the experiments in the paper are in the domain of applicability of AtP*. A comparison between the two methods on how well they deal with saturated inputs is important. Which one should I use? AtP* is fairly computationally cheap, even if you apply it for each intermediate part of the gradient integration. Actually, both methods could be combined because they make incremental improvements.\n\nThis paper needs to compare with AtP*.\n\nQUESTIONS:\nThe paper should compare with AtP*, which also deals with saturation; explain why it's OK for the line to not be between x and x' and instead be something else, and empirically show that the method works because of why they say it works.\n\n\nAlso:\n- What is the variability between runs in Figures 3 and 4? It is impossible to say if the current method is an improvement if there is a lot of variation based on e.g. random seeds and the exact selection of training points.\n- Are the quantities in Figure 3 evaluated on test datasets? They should be.\n\n\nI'm willing to accept this paper if it satisfactorily addresses all the 5 things above.\n\n## Minor questions\n- Where exactly does the 17.7% figure for the \"improvement up to 17.7%\" in the abstract come from? Is it the largest gap between EAP-GP and the next method in Figure 3?\n- How did you choose the circuit edge for Fig. 1? In L165 it just says \"we analyze a circuit edge\".\n- Presentation tips:\n  - Showing the sparsity in %s close to 100% in Fig. 2 and 3 was confusing to me. I'm used to low % (so the % is the amount of nonzero things) or straight up counting. That way you can also space your datapoints in a log-space.\n  - It would be nice if the caption of Fig. 1 linked to Section 4, which discusses it, since it appears 3 pages later. I was looking for an explanation of the figure in the same and next page and could not find it, leaving me a bit confused.\n  - use \\left( and \\right) in equations like eq. 7 to make the brackets as high as the thing they're bracketing (the fraction)\n\nLIMITATIONS:\nNot for factual limitations, see my point about a misleading Fig. 1 (and text, which is also misleading) above.\n\nTheir work should be generally of societal positive impact, though it's fairly basic research so the impacts are not near.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}