IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Authors: Tian Bian
Yifan Niu
Chaohao Yuan
Chengzhi Piao
Bingzhe Wu
Long-Kai Huang
Yu Rong
Tingyang Xu
Hong Cheng
Jia Li
Venue: ICML 2025 poster (2025)
Keywords: Circuit Discovery
Primary area: deep_learning->large_language_models

Abstract: Circuit discovery has recently attracted attention as a potential research direction to explain the non-trivial behaviors of language models. It aims to find the computational subgraphs, also known as circuits, within the model that are responsible for solving specific tasks. However, most existing studies overlook the holistic nature of these circuits and require designing specific corrupted activations for different tasks, which is inaccurate and inefficient. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, to holistically identify informative circuits. In contrast to traditional causal interventions, IBCircuit is an optimization framework for holistic circuit discovery and can be applied to any given task without tediously corrupted activation design. In both the Indirect Object Identification (IOI) and Greater-Than tasks, IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work.

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Tian Bian
12
Yifan Niu
Chaohao Yuan' Chengzhi Piao Bingzhe Wu Long-Kai Huang Yu Rong Tingyang Xu 27 Hong Cheng' Jia Li
Abstract
heads and MLPs in Transformers (Vaswani et al., 2017)) that either partially or fully explain the model's behaviors on tasks like Indirect Object Identification (IOI), modular arithmetic, and forecasting subsequent dates (Wang et al., 2022; Nanda et al., 2023; Hanna et al., 2024). The challenge of eircuit discovery lies in the fact that eircuits are hidden in the complex black boxes of language models which involve complex non-linear interactions in and embed in high-dimensional space (Wang et al., 2022; Li et al., 2025a).
Circuit discovery has recently attracted attention potential research direction to explain the non-trivial behaviors of language models. It aims to find the computational subgraphs, also known as circuits, within the model that are responsible for solving specific tasks. However, most exist- ing studies overlook the holistic nature of these circuits and require designing specific corrupted activations for different tasks, which is inaccu- rate and inefficient. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, to identify informative circuits holistically. IBCir- cuit an optimization framework for holistic eir- cuit discovery and can be applied any given task without tediously corrupted activation design. In both the Indirect Object Identification (IOI) and Greater-Thaal tasks, IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work. The code is available at https://ithub..omm/// wwtp://ww/////////ccoeeoo
layers
Recent circuit discovery works seek to decode these models by reverse engineering (Räuker et al., 2023), such as acti- vation patching (Meng et al., 2022) and attribution patch- ing (Nanda, 2023). The activation patching (Geiger et al., 2021; Goldowsky- Wang et al., 2022) performs causal interventions indepen- dently on each component, ignoring that the circuit is a holistic system rather than an independent combination. Additionally, they do not scale well with the model size. An- other research line has proposed attribution patching (Nanda, 2023) and its variants (Chintam et al., 2023; Hanna et al., 2022; Marks et al., 2024) to efficiently estimate the impor- tance of each component in the computational graph based on gradient methods. However, they still need to redesign corrupted activations for different tasks, which is inconve- nient and complicated.
et al., 2023; Conmy et al., 2023;
1. Introduction
Circuit discovery in language models usually involves iden- tifying the subgraphs (circuits) within the model that are responsible for solving specific tasks (Olah et al., 2020). Previous efforts to identify circuits within language mod- els have led to the discovery of components (e.g., attention
An ideal minimal circuit should (i) contain only the es- sential components for specific tasks, and (ii) exclude any irrelevant redundant components. Intuitively, identify high-quality circuits, we should maximize the circuit's 's rele- vance to the tasks while minimizing irrelevant components. Interestingly, the objective of circuit discovery aligns with the Information Bottleneck (IB) principle. The IB leverages Shannon mutual information to distill the compressed yet informative data distributions (Tishby al.. 2000). From the IB perspective, circuit discovery is to (i) find circuits that are the most informative to the specific tasks and (ii) compress them from the overall language models.
"Equal contribution; alphabetical order by last name. 'Department of Systems Engineering and Engineering Manage- ment, The Chinese University of Hong Kong, Hong Kong, China DDAMO Academy, Alihaba Group, Hangzhou, China 'Hon Kong University Science and Technology (Guangzhou), Guangzhou, China 'Department of Computer Science, Hong Kong Baptist University, Hong Kong, China School of Artificial Intelligence, Shenzhen University, Shenzhen, China TTencent, Shenzhen, China 'Hupan Lab, Hangzhou, China. Correspondence to: Tingyang Xu <xuty_007e @ hom>> Jia Li <jialce@ust.hk.
In this work, we propose the Information Bottleneck Circuit (IBCireuit), novel IB framework designed to identify the informative components within Transformer-batt language models. IBCireuit an optimization framework for holistic
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
circuit discovery and can be applied any given task with- tediously corrupted activation design. To parameterize circuit, IBCircuit injects controllable Gaussian noise with learnable IB weights into various model components (e.g., the activations of attention heads and MI.Ps). It modulates the flow of clean information from the original pretrained model to its distorted version. The IBCircuit objective en- courages the distorted flow retain its informativeness. Therefore, IBCircuit preserves the most informative compo- nents with minimal noise injection. This process simulates information compression, ensuring that the most informa- tive components are preserved while irrelevant components are filtered out. Finally, the circuit is formed by discretiz- ing the continuous weights to select the most informative components. The contributions of this work are as follows:
2024) employ gradient-based approaches estimate com- ponent importance across the computational graph. While these techniques improve efficiency through parallelizable gradient computations, they introduce new practical chal- lenges. Current implementations require task-specific en- gineering of corrupted activation patterns (Nanda, 2023), creating implementation complexity and limiting generaliz- ability across different behaviors. This fundamental tension between intervention fidelity and computational efficiency remains unresolved in existing literature.
Information Bottleneck The principle of the information bottleneck (IB) aims to extract a compressed yet predic- tive code from the input signal (Tishby et al., 2000; Yu et al., 2024). Alemi et al. (2017) initially introduced the variational information bottleneck (VIB) to deep learning interpretability research. Currently, IB and VIB primarily focus on informative representation learning and feature se- lection. In representation learning, researchers aim to learn compressed representation with the IB principle (Luo et al., 2019; Qian et al., 2020; Wu et al., 2020; Liu et al., 2024), For feature selection, IB is used to select a subset of input features, such as image pixels or vector dimensions (Achille & Soatto, 201 1b; Kim et al., 2021; Schulz et al., 2020). For instance, Yu al. (2020) et al. present a graph information bottleneck to identify important subgraphs. Wu & Deng (2023) et al. introduce the neck (TIB) method, which uses a standard IB and a Reverse Information Bottleneck (RIB) to detect unknown objects.
We propose a circuit discovery framework, IBCireuit, which utilizes information bottleneck to holistically identify the most informative and compressed circuit within the language models.
We introduce a circuit parameterization strategy that incorporates noise injection with learnable IB weights, providing task-agnostic alternative to the task- specific corrupted activation construction.
We conducted experiments in the Indirect Object Iden- tification (IOI) and Greater-Thar tasks, verifying that IBCircuit can identify more faithful and minimal cir- cuits in terms of selecting critical node and edge com- ponents compared to baseline methods.
Information Bottle-
3. Preliminaries
2. Related Work
Neural Circuits A Neural Circuit for a given task is the minimal computation subgraph C C G. where C and G denote the set of components in the circuit and complete model, respectively (Olah et al.. 2020). In the computational graph of transformer-basee language models G, the nodes are defined as MLPs and attention heads, and the edges are defined as the dependencies between nodes (Li et al., 2025b; Chang et al., 2024). The objective of circuit discovery is to identify sparse subgraph that encapsulates the behavior of the complete model for specific task. We denote the output of the circuit, given the original and distorted information flows x, ., as Pc Py xr .), while the output of the complete model is represented as po r). Formally, the goal of circuit discovery can be expressed as follows:
Circuit Analysis Recent advances in circuit analysis fo- cus reverse-engineerring neural networks through two primary methodologies: activation patching and attribu- tion patching. The activation patching paradigm (Vig et al., 2020; Finlayson et al., 2021; Geiger et al., 2021; Goldowsky- Dill et al., 2023; Conmy et al., 2023; Wang et al., 2022) conducts causal interventions by modifying individual com- ponents' activations. Common implementations include overwriting activations with zeros (Cammarata et al., 2021; Olsson et al., 2022), substituting with dataset mean values (Wang et al., 2022; Hanna et al., 2024), or applying inter- change interventions (Geiger et al., 2021; Wu et al., 2024). However, these approaches face two fundamental limita- tions: (1) They treat circuit components as independent entities, disrupting the model's holistic computation flow (Räuker et al., 2023); (2) The substituted activations often deviate from plausible activation distributions (Chan et al., 2022), and their sequential intervention process becomes computationally expensive for large models. In contrast, attribution patching methods (Nanda, 2023) and their vari- ants (Chintam al., 2023; Hanna et al., 2022; Marks et al.,
arg min C
E(G,))TT (D(po() (x) ll pel x. x,)))),
(1)
s.t. 1 - 1C/1g/ 2c
The constraint c is designed to ensure a desired sparsity for the circuit. The set T represents the relevant task distribu- tion. The objective function D captures the discrepancy between the outputs of the complete model and the eircuit.
2

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Information Bottleneck The Information Bottleneck (IB) (Tishby et al., 2000) is to optimize the representation Z to capture the minimal sufficient information within the input data D to predict the target Y. Its objective can be formulated as follows:
Proposition 4.1 (Variational lower bound of I(;;)). For the output Y of the original transformer language model g, the output Ye of the given circuit C, the variational lower bound can be written as:
1 N N
min (D; - T(;; 2)+ P(Z)D)E
(2)
I(Y;)) 2
E
DRI H(y
(4)
where Q defines the search space of the optimal model, and 1(3)) denotes the mutual information (Cover, 1999). The first term - Z) encourages Z to be informative about the target Y, while the second term [D;; Z) ensures that Z does not receive irrelevant information from D. The IB provides a critical principle for representation learning: an optimal representation should contain the minimal sufficient information for the downstream prediction task.
where D D-I(-11-)) is the Kullback-Leibler Divergence, (.)) is the entropy.
In general, given a language model on a specific task, the Hy((i) is a constant. Therefore, we can omit this term, and the lower bound of Eq. (4) can be reformulated as - E (x ). This inequality is highly intuitive, demonstrating that maximizing /(Y;C) can be achieved by minimizing the KL Divergence of the output between the whole computation graph and the circuit. In other words, the circuit contains the most informative com- ponents for specific tasks.
4. IBCircuit
4.1. Intuition: Informative Circuit
In language model, let consider the specific task X (200) Jimi consisting of N i.i.d. samples and the target Y == (yl J Ji=1- We assume that the data are generated by some random process, involving an unobserved random circuit (c(9) N Ji=1- An ideal minimal circuit (i) contains the most informative components for specific tasks, and (ii) does not include irrelevant redundant components. Intuitively, the goal of circuit discovery in Eq. (1) aligns with the concept of the Information Bottleneck (IB) principle. We denote G = (0) J as the whole computation graph of the transformer-based model, Y as the output of the model on specific tasks X, and C as the circuit composed of critical components. We reformulate Eq. (2) to obtain the objective of IBCireuit follows:
Minimizing I(G:C). For the second term I(G: C) in Eq. (3), which ensures C contains minimal irrelevant or redundant information about G. Its variational upper bound is as follows:
Proposition 4.2 (Variational upper bound of (G; For the complete transformer language model G, and the circuit C, the variational upper bound can be written as:
I(G;C) DKL(((C1))1I(())
(5)
where (C1[)) and p(C) is the posterior and prior of the circuit. Minimizing the upper bound of I(G;C) constrains the irrelevant information that C retains from G. Therefore, we can select the circuit C from after training the IBCircuit with the tractable bounds.
min IBx(G, A (-X G)
(3)
The first term encourages the circuit C to be informative on the targets Y. The second term ensures that C' receives limited information from the whole computation graph G, ie., minimizing task insk-ireeeevant components. However, in practice, the mutual information in Eq. (3) is intractable.
4.3. Circuit Parameterization
To implement IBCircuit, we follow the variational method (Rockafellar & Wets, 2009) and assume the prior of the circuit to be isotropic Gaussian. Given pretrained model composed of components with intermediate ac- tivations h = [hi, fug. ... hz). we introduce IB weights X = A2, to control information flow through A; = (ws with learnable parameter wi. To con- struct circuit, we adopt Gaussian noise e; Niu,, of) scaled by X E (0, to perturb activations, where ir and of are computed from batch activations of h,. Following established circuit analysis protocols (Wang et al., 2022; Conmy et al., 2023), we parameterize the circuit on node and edge level:
4.2. Estimation of Mutual Information
Exact computation of I(Y:C) and C(G:C) is intractable. Hence, we introduce variational approximation to estimate the variational bounds on these two terms and propose the IBCircuit framework to find informative circuits. The deriva- tion is detailed in Appendix A.
Maximizing CCY; We first examine the first term C(Y:C) in Eq. (3), which encourages to be informative of the output Y. We derive the variational lower bound of T(;CC) as follows:
Node-wise Parameterization. Following previous work on node-level circuit discovery (Wang et al., 2022), we focus
3

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
where hy denotes the j-th source node's activation and hy de- notes the distorted target node's activation, and Ajs denotes edge-wise IB weight.
Inject Node Noise
Node Selection
The IB weights À implement differentiable information gates: X; - preserves original activations while X; + 0 induces maximal noise. Crucially, our formulation enables simultaneous optimization of all through holistic gradi- ent flow across the computational graph and adaptive noise calibration via batch statistics. This contrasts with existing patching methods that require sequential intervention.
(a) Transformer
(b) Distorted Node
(c) Node Circuit
Inject Edge Noise
Hidden States
MLP
A Atlention Head
4.4. Objective for Training
Head with More Noise
The circuit C is designed extract information from the pre-trained model G by approaching the original outputs Y. Our framework operates through two complementary mech- anisms: (1) maximizing the mutual information between C's outputs and the target outputs Y to maintain functional consistency, and (2) introducing controlled noise injection to minimize dependence on the pre-trained model G, thereby preserving task-critical components while perturbing non- essential components. Considering the theoretical results in Propositions 4.1 and 4.2, the variational upper bound IBCircuits (9, Y;C) can be formulated as:
Head with Less Noise
E Edge Selection
Selected Head
Residual Stream
O
Forward Edge
Edge with More Noise
0 K
V
Edge with Less Noise
Selected Edge
(d) Distorted Edge
(e) Edge Circuit
Figure 1. (a) Transformer blocks from the perspective of the resid- ual stream. (b) Adding Gaussian noise to the activations of atten- tion heads using node-wise IB weights and optimizing through the Information Bottleneck. (c) Selecting attention heads with less noise as the node-level circuit. (d) Adding Gaussian noise to the activations of source nodes using edge-wise IB weights and opti- mizing through the Information Bottleneck. (c) Selecting edges with less noise as the edge-level circuit.
Σ DKL (v +3Dxrl((()))))(()))
(8)
N
where ,3 is the hyperparameter. Consider our circuit pa- rameterization, the second term [xxl(((())]pp(C) can be further specified as:
on attention heads node components. We perturb the output activations of each attention head in the Transformer architecture and train the distorted model holistically. For individual attention heads:
Dx Dxrl((C(())p((C = log A
A + B
(9)
2n
where A = -- Ar), B = Σ A.(h--m)) and n is the number of component in the language model. Al- though the exact computation of CY: and CGG:C) is intractable, we can calculate their variational bounds. There- fore, we adopt IBCircuit,s (G. C:C) as the overall objective function. Compared to existing methods, (1) IBcircuit is an optimization framework that allows us to optimize the IB weight holistically through end-to-end training: (2) IBCir- cuit uses the original output Y of the transformer model as supervision signals, avoiding the need to manually design corrupted activations.
di
h
X signal preservation
(1 x))
injoction
where fú denotes the i-th attention head's activation and hy denotes the distorted activation, A; is node-wise IB weight.
Edge-wise Parameterization. Inspired by ACDC (Conmy al., 2023), we explicitly model the Transformer's resid- ual stream through two node types: (1) Source Nodes: Token/positional embeddings, attention head outputs, and (2) Target Nodes: Q/K/V projection in- puts and MLP inputs. Following the layer-wise residual architecture, each target node at layer / (denoted as trg (l)) aggregates information from all preceding source nodes up to layer/ (denoted sre(< 1)), forming directed edges E == ú): E ste(< < 1), trg(()). We parameterize the
MLP outputs.
4.5. Circuit Formation
The learned IB weights À encode component importance through their information transmission capacity. We develop circuit formation protocols for node and edge components:
edge through:
Node Component Selection. Corresponding to node-wise perturbation in Eq. (6), we identify critical attention heads:
h,
Σ (FA)EE
Aynhy signal propagation
(1 (1-)
Crode {o [n]]
noise injection
A
Tnode
(10)

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
where the adaptive threshold Tnode is determined via:
Greater-Than (Hanna et al., 2024): In the Greater- Than task, models receive input like "The war lasted from the year 1741 to the year 17", and must predict a valid two-digit end year, i.e., one that is greater than 41. In this paper, we aim to identify which components of the model are crucial for predicting the end year.
Thode = inf {T E (0.1)
I(A,> Σ I(A; > T) S Kande J (11)
with kiddde controlling the maximum allowed nodes.
Edge Component Selection. Similar to node component selection, we select essential residual connections through:
Baselines. We compare the proposed method with the following methods designed for node component selection and edge component selection, respectively:
Codge = - ( EE A5
( Tetge )
(12)
Node Component Selection Methods:
The edge threshold Tedge is determined via:
Subnetwork Probing (SP) (Cao et al., 2021): SP learns mask for each node in the circuit to determine if it is part of the circuit via gradient descent.
Tedge = inf { E (0,1)
Σ
T(ye > T) S Rietge) ). (13)
Automated Circuit DisCovery (ACDC) (Conmy et al., 2023): ACDC is originally designed for edge component selection. We derive the score of a node by summarizing the impact of its removal the model.
with Aedgo controlling the edge sparsity.
5. Experiments
Attribution Patching (AP) (Nanda, 2023): AP as- signs scores to all nodes at the same time by leveraging gradient information and again prunes nodes below a certain threshold to form the final circuit.
In this section, we conduct the evaluation to answer the following research questions (RQs):
RQ1-Grounded in Previous Work: Can IBCireuit ef-
Edge Component Selection Methods:
fectively reproduce circuits taken from previous works that found the circuit explaining behavior for tasks?
Automated Circuit DisCovery (ACDC) (Conmy et al., 2023): ACDC traverses the transformer's compu- tational graph in reverse topological order, iteratively assigning scores to edges and pruning the circuit.
RQ2-Ablatinn Study: Are both KL loss and MI loss used for training IBCircuit necessary? How does the different a affect the effectiveness of IBCireuit? What is the contribution of each component in the IBCireuit?
Edge Attribution Patching (EAP) (Nanda, 2023): EAP is an edge version of AP that takes into account the activations of both the source and target nodes.
RQ3-Faithfulness & Minimality: Does IBCircuit avoid including components that do not participate in the behavior while maintaining better faithfulness?
We also compared two variants of IBCircuit, namely IBCircuit-woMI and IBCireuitonlyyMI in RQ2, which represent IBCircuit models trained solely with KL loss and MI loss, respectively. Further details about experimental setup are in Appendix B.
RQ4-Scalability to Large Models: Does IBCircuit have the scalability to be applied on large models?
5.1. Experiment Setting
Metrics For the IOI task, we use Logit Difference for evalu- ation. Logit Difference measures the difference in logits as- signed to the correct and incorrect answers. For example, for the input "When Mary and John went to the store, John gave drink to." we calculate (ogit(Mary) logit((Joh)), The larger the Logit Difference, the better the performance of the model or circuit. In the Greater-Than task, we use the Greater Prob- ability metric, which sums the total probability assigned to all correct and incorrect options and calculates the differ- ence, e.g., for the input "The war lasted from the year 1741 to the year 17", we calculate Σ1241 P(y) Eysan P(y). A larger difference indicates better model or circuit perfor- mance. For these two tasks, we calculate the KL Divergence between the logits output by the Circuit and the pre-trained model. A smaller KI. Divergence indicates that the results
Tasks We primarily focus on GPT-2 (Radford et al., 2019) for better evaluation, as it is a classical model typically studied from a circuit's perspective. We intentionally choose two tasks (IOI and Greater-Than) that have been studied before for fair comparison with previous work.
Indirect Object Identification (IOI) (Wang et al., 2022): An IOI sentence involves an initial dependent clause, e.g., "When Mary and John went to the store", followed by main clause, e.g.. "John gave drink to Mary." In this case, the indirect object (IO) is "Mary" and the subject (S) is "John". The 10I task is to predict the final token in the sentence to be the IO. IOI Circuit Discovery aims to identify which components of the model are crucial for performing such IOI tasks.
5

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
1.0
1.0
SP
ACDC
True Positive Rate
0.8
True Positive Rate
0.8
AP
IBCircuit
0.6
0.6
SP
0.4
ACDC
0.4
IBCrccuit-onlyMi
IDCircuit-woM/
0.2
0.2
AP
IBCircuit
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
(a) I0I
(b) Greater-Than
Figure 2. ROC curves of SP, ACDC, AP and IBCircuit identifying model components from previous work, across 10I circuit and Greater-Thal circuit.
0.16
0.12
014
Average
0.10
0.09
0.10
0.88
008
1.kk
2k
2,
3
3.5k
4k
1.k
2k
25k
3k
3.5k
4
Training Steps
Training Steps
1.0MI
0.8MI
0.6MI
0.4MI
0.2MI
0.01MI
0.001M1
1.0MI
0.8MI
0.6MI
0.4MI
0.2MI
0.01MI
0.001MI
(a) Average À
(b) KI. Divergence
Figure 3. Comparison of the impact of different trade-off coefficients a on IBCireuit implementing the task.
of the Circuit have greater faithfulness.
Figure 2 illustrates the performance IBCircuit recover- ing the canonical circuit within GPT2-small, compared to existing methods. Our findings are as follows: i) IBCircuit shows competitive performance on both the IOI and Greater- Than tasks, notably outperforming baseline methods the IOI task; ii) however, IBCireuit underperforms compared to ACDC on the Greater-Thar task. This discrepancy might be attributed to the fact that the Greater-Thie task, unlike the IOI task, does not have clearly defined expected out- put. The broader range of possible correct outputs could potentially increase the learning difficulty of the IBCireuit.
Circuit Ablation We ablate the nodes that are not included in the circuit by using activation patching to evaluate the effectiveness of identified circuit. We implement randomly selected activations from the corrupted dataset for patching. In IOI task, we construct corrupted inputs by replacing IO and S with arbitrary names. In Greater-Thar task, the start year's last two digits are changed to "01", leading models to output years prior to the start year.
5.2. RQI-Grounded in Previous Work
Following (Conmy et al., 2023), we formulate circuit dis- covery as a binary classification problem, where nodes are classified as positive (in the canonical circuit taken from previous works) or negative (not in the canonical circuit). We determine series of thresholds for ACDC, SP, AP, and IBCircuit by varying the number of nodes from 10% to 100%, increasing by 10% each time. We plot the pessimistic segments between the Pareto frontiers of TPR and FPR for each method across this range of thresholds.
5.3. RQ2-Ablation Study
The Influence of KL Loss and Loss In Figure 2, we compare the IBCircuit models trained without KL loss or MI loss. We find that: on the IOI task, IBCircuit outper- forms IBCireuit- onlyMI. This can be intuitively explained using Information Bottleneck, as the KL loss primarily serves to align the performance the noisy model with that the pretrained
and significantly surpasses IBCircuit-
6

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
0.95
1.5
Greater Probability
8.9
0.8
0.75
0.5
8.7
0.65
20
40
60
86
100
120
140
20
40
60
80
100
120
140
Number of Nodes
Number of Nodes
IBCircuit Obj)
IBCircuit(hard mask)
IBCircuit 0bj)
IBCircuit mask)
IBCircuit
IBCircuit
(a) Greater Probability
(b) KL Divergence
Figure 4. Comparison of IBCircuit and ablation variants in terms of Greater Probability and KL Divergence metrics.
0.9
Logit Difference
Greater Probability
0.85
0.8
0.75
0.7
0.65
20
40
60
88
100
129
140
20
40
60
80
100
128
146
Number of Nodes
Number of Nodes
SP
ACDC
AP
IBCircuit
SP
ACDC
AP
IBCircuit
(a) TOI
(h) Greater-
Figure 5. Comparison of IBCircuit and related methods terms of Logit Difference and Greater Probability metrics under different node number thresholds. Higher metric scores and fewer nodes correspond better circuits.
model, while the MI loss helps reduce irrelevant information the noisy model. Consequently, the absence of MI loss in IBCircuit-woMI results in slightly worse performance com- pared to IBCircuit, whereas the lack of KL loss in IBCircuit- onlyMI severely diminishes the performance.
Contribution of Each Component To validate the contri- bution of each component in IBCireuit, we further compare the following variants:
IBCircuit (hard mask): We replaced the sigmoid- based mask used for Gaussian perturbations with Hard- concrete Masking (Cao et al., 2021).
Parameter Sensitivity Analysis We further compare the impact of different trade-off coefficients on IBCircuit in implementing the IOI task in Figure 3. In Figure 3a, we found that as a increases, IBCircuit can achieve a smaller average with the same number of training steps, indicating that larger can learn sparser Circuit. Correspondingly, as shown in Figure 3b, as a increases, the Circuit obtained by IBCircuit with the same number of training steps has larger KI. Divergence. This indicates that during the learning process, IBCircuit will sacrifice KL divergence to obtain sparser Circuit. How to set the hyperparameter depends on whether the user wants to obtain a sparser Circuit Circuit with more stable KI. divergence.
IBCircuit (SP Obj): We substituted the mutual information-based loss LMI with the SP (Subnerwork Probing (Cao al., 202/)) Objective, which penalizes the mask based on the probability of it being non-zero.
We conducted two comparative experiments on the Greaterthan task using the Greater Probability and KL Divergence metrics. The evaluation results for these two metries are shown in Figure 4. Higher Greater Probabil- iry (Lower KL Divergence) and fewer nodes correspond better circuits.
Gaussian Perturbations vs. Hard-concrete Masking, We

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
2.5
1.5
2
1.5
KL Div
DIV
1
KL
160
120
140
28
48
68
80
100
120
148
Number of Nodes
Number of Nodes
SP
ACDC
AP I8Circuit
ACDC
18Circuit
(a) IOI
(b) Greater-Thar
Figure 6. Comparison of IBCireuit and related methods of KL Divergence metric under different node number thresholds. Lower metric scores and fewer nodes correspond to better circuits.
0.6
Greater Probability
Logit Difference
0.55
0.5
0.5
0.45
-0.5
-1
0.4
2800
600
880
1800
1200
1408
Number of Edges
Number Edges
ACDC EAP
18Circuit
ACDC
EAP
IBCircuit
(a) IOI
(b) Greater-Than
Figure 7. Comparison of IBCircuit and related methods in terms of Logit Difference and Greater Probability metrics under different edge number thresholds. Higher metric scores fewer edges correspond to better circuits.
observed that, except for cases with a few nodes (fewer than about 45), the circuits optimized via Hard-concreds Masking achieved better Greater Probahility compared to IBCireuit. However, in such small-node scenarios, all methods resulted in high KL Divergence between the identified circuits and the original pretrained model's outputs, indicating signifi- cant degradation of the circuit's capabilities. Since preserv- ing the pretrained model's performance is critical, extreme cases with very few nodes are less meaningful. Focusing on scenarios where pretrained capabilities are maintained (cases with more nodes), IBCircuit outperforms the Hard- concrete Masking variant, demonstrating the superiority of Gaussian Perturbations.
This highlights that the mutual information- based regular- ization (Larz) is more effective at preserving the pretrained model's capabilities while optimizing circuit discovery.
5.4. RQ3-Faithfulness & Minimality
Intuitively, a circuit with fewer nodes or edges that still achieves high metrics is less likely to contain components that do not participate in the behavior (Conmy et al., 2023). We measure the performance of various methods in terms of Logit Difference, Greater Probability, and KL Divergence under different node and edge number thresholds.
Figure and Figure 6 present a comparison of metric scores for various methods across different numbers of node components in the Indirect Object Identification (IOI) and Greater-Than tasks. In the IOI task, IBCireuit achieves higher Logir Difference with fewer nodes, demonstrating perior efficiency in utilizing limited components to maintain
Mutual Information Regularization vs. SP Objective. The model optimized with the SP Objective performed com- parably to IBCircuit only when the number of nodes is fewer than 20. In all other cases, it shows significantly worse per- formance in both Greater Probability and KL Divergence.
8

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
0.4
3.5
0.35
KL Div
KL DIV
2.5
0.25
0.2
0.15
8.1
0.5
0.05
580
1890
1588
2000
400
600
869
1900
1260
1480
Number of Edges
Number of Edges
IBCircuit
ACDC
IBCircuit
(a) IOI
(h) Greater-Than
Figure 8. Comparison IBCircuit and related methods terms KI. Divergence metric under different edge number thresholds. Lower metric scores and fewer edges correspond to better circuits.
For edge component selection presented in Figure 7 and Figure 8, IBCircuit outperforms baseline methods in both IOI and Greater-Than tasks. In the IOI task, the num- ber of edges decreases, IBCircuit consistently achieves higher Logit Difference and lower KL Divergence compared to ACDC and EAP. Notably, in the Greater-Than task, IB- Circuit maintains a Greater Probability above 50% and Divergence below 0.15 even with very sparse edge counts. This highlights its efficiency and reliability in optimizing circuit performance under constrained components.
Logit Difference
26
24
50k
60k
70k
Number of Edges
5.5. RQ4-Scalability to Large Models
EAP
EAP-IG
IBCircuit
Following the experimental setup of EAP-IG (Hanna et al., 2022), compare IBCireuit with EAP-IG and EAP on the IOI task using GPT-2 XI. (1.5B) (Radford et al., 2019). As shown in the Figure 9, IBCircuit achieves performance comparable to EAP and better logit difference than EAP when the number of edges ranges from 30k to 70k (approxi- mately 97%-98.5 88.5% of the total 2235025 edges). This result demonstrates the scalability of IBCircuit to larger models.
Figure 9. Comparison of IBCircuit and related methods terms of Logit Difference metric under different edge numher thresholds. Higher metric scores and fewer edges correspond to better circuits.
predictive accuracy. Although IBCircuit does not perform as well as other methods when the number of nodes is greater than 90, it still maintains a large Logit Difference (around 4). This may not affect its performance compared the pretrained model. Additionally, the proposed method does not require manually designing corrupted data. In Figure 6, larger number of nodes achieved a lower KL Divergence, which also supports the idea that when the Logit Difference large enough, minor differences do not affect the performance of the Circuit. the Greater-Than task, when the number of nodes greater than 70, IBCir- cuit shows comparable performance to ACDC in terms of Greater Probability and KL Divergence. However, when the number of nodes is low, IBCircuit performs worse than ACDC. This may be because the Greater Than task does not have a specific token as the answer, unlike the IOI task. When the model's s sparsity is highly pursued, IBCircuit finds difficult to achieve the best faithfulness.
6. Conclusion
In this paper, we aim to address the challenge of understand- ing the behavior Transformer-basee models, which are often seen black boxes due their complex computations. In this work, we propose an end-to-end approach based on the principle of Information Bottleneck, called IBCircuit, identify informative circuits holistically. IBCircuit is an opti- mization framework for holistic circuit discovery and can be applied to any given task without tediously corrupted activa- tion design. In both the Indirect Object Identification (IOI) and Greater-Than tasks, IBCircuit identifies more faithful and minimal circuits in terms critical node components and edge components.

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Acknowledgment
B., and Thomas, N. Causal scrubbing: A method for rigor- ously testing interpretability hypotheses. In Al Alignment Forum, pp. 1828-1843, 2022.
This research is supported by grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14217622). This work was supported by Damo Academy (Hupan Laboratory) through Damo Academy (Hupan Laboratory) Innovative Research Program. The authors would like to express their gratitude to the reviewers for their feedback, which has improved the clarity and contribution of the paper.
Chang, H., Ye, J., Lopez-Avilas A., Du, J., and Li, Path- based explanation for knowledge graph completion. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 231-242, 2024.
Chintam, A., Beloch, R., Zuidema, W., Hanna, M., and Van Der Wal, O. Identifying and adapting transformer- components responsible for gender bias in an english language model. In Proceedings of the 6th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 379-394, 2023.
Equal Contribution Statement Tian Bian and Yifan Niu jointly proposed the IBCircuit framework. Yifan Niu in- troduced variational approximation, derived the theoretical results, and wrote Section 1-4, Both of them contributed to the carly development of the IBCircuit code. Tian Bian then extended the code the IOI and greater-than tasks, conducted the experiments on GPT-2, and wrote Section 5. Both authors contributed equally this work; listed alphabetically by last name.
Conmy, A., Mavor-Parker A., Lynch, A., Heimersheim, S., and Earriga-Alonse A. Towards automated circuit dis- covery for mechanistic interpretability. Advances in Neu- ral Information Processing Systems, 36:16318-16352. 2023.
Impact Statement
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
This paper presents work whose goal to advance the field of mechanistic interpretability. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S. M., Linzen, T., and Belinkov, Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828-1843, 2021.
References
Achille, A. and Soatto, S. Emergence of invariance and disentanglement in deep representations. Journal of Ma- chine Learning Research, 19(50):1-34, 3018a.
Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstrac- tions of neural networks. Advances in Neural Information Prucessing Systems, 34:9574-9586, 2021.
Achille, A. and Soatto, S. Information dropout: Learning optimal representations through noisy computation. IEEE transactions pattern analysis and machine intelligence, 40(12):2897-2905 2018b.
Goldowsky-Din N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching. arXiv preprint 2.XXv:2300.00969, 2023.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
Hanna, M., Pezzelle, S., and Belinkov, Y. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms, 2024. URL https://arxiv. org/abs/2403..7886, pp. 12, 2022.
Cammarata, N., Goh, G., Carter, S., Voss, C., Schubert, I.., and Olah, C. Curve circuits. Distill, 6()):c000240006 2021.
Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abili- ties in pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Cao, S., Sanh, V., and Rush, A. M. Low-complexity prob- ing via finding subnetworks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 960-9660 2021.
Kim, J., Kim, M., Woo, D., and Kim, G. Drop- bottleoktle Learning discrete compressed representation for noise- robust exploration. In International Conference on Learn ing Representations, 2021.
Chan, L., Garriga-Alonso A., Goldowsky-Dine N., Green-
blatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris,
10

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Li, Y., Zhang, X., Luo, L., Chang, H., Ren, Y., King. I,, and Li, J. G-refer: Graph retrieval-augmentee large language model for explainable recommendation. In Proceedings of the ACM on Web Conference 2025, pp. 240-251, 2025a.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Räuker, T., Ho, A., Casper, S., and Madffeld-Menelle D. To- ward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Con- ference on Secure and Trustworthy Machine Learning (SaTML), pp. 464-483. IEEE, 2023.
Li, Z., Cen, J., Su, B., Huang, W., Xu, T., Rong, Y., and Zhao, D. Large language-geometry model: When Ilm meets equivariance. arXiv preprint arXiv:2502.11149, 2025b.
Rockafellar, R. T. and Wets, R. J.-B. Variational analysis, volume 317. Springer Science & Business Media, 2009.
Liu, Z., Wang, T., Shi, J., Zheng, X., Chen, Z., Song, L., Dong, W., Obeysekera, J., Shirani, F., and Luo, D. Timex++: Learning time-series explanations with infor- mation bottleneck. In Forty-first International Conference on Machine Learning. 2024.
Schulz, K., Sixt, L., Tombari, F,, and Landgraf, T. Restrict- ing the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
Luo, Y., Liu, P., Guan, T., Yu, J.. and Yang. Y. Significance- aware information bottleneck for domain adaptive seman- tic segmentation. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 6778-6787, 2019.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 2000.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L.. Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Marks, S., Rager, C., Michaud, J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.96647 2024.
Vig. J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating gender bias in language models using causal mediation analysis. Ad- vances in neural information processing systems, 33: 12388-12401, 2020.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-17372, 2022.
Nanda, N. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. hnn.ecclanttccieeprreblllaaaaabn 2023.
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: a circuit for in- direct object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022.
Nanda, N., Chan, L.,, Lieberum, T., Smith, J., and Steinhardt, Progress measures for grokking via mechanistic inter- pretability. In The Eleventh International Conference on Learning Representations, 2023.
Wu, A. and Deng, C. Tib: Detecting unknown objects via two-stream information bottleneck. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to circuits. Distill, 5(3)xc00024-000 2020.
Wu, T., Ren, H., Li, P., and Leskovec, Graph information bottleneck. Advances in Neural Information Processing Systems, 33:20437-20448, 2020.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1895, 2022.
Wu, Z., Geiger, A., Icard, T., Potts, C., and Goodman, N. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in Neural Information Processing Systems, 36, 2024.
Qian, K., Zhang, Y., Chang, S., Hasegawa-Johnson M., and Cox, D. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
Yu, J., Xu, T., Rong, Y., Bian, Y., Huang, J., and He, R. Graph information bottleneck for subgraph recognition. International Conference on Learning Representations, 2020.

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Yu, J., Cao, J., and He, R. Improving subgraph recogni- tion with variational graph information bottleneck. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19396-19405, 2022.
Yu, J., Xu, T., Rong. Y., Bian, Y., Huang, J., and He, R. Rec- ognizing predictive substructures with subgraph informa- tion bottleneck. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(3):1650-1665 2024.
12

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
A. Analysis of IBCircuit
A.1. Proof of IBCircuit Objective
We justify the above formulation through the following derivation. Let Ga be a subset of G, which is independent to Y.
Denote G. as the noisy subset of G determined by injected noise, if we select the eircuit C. by dropping Ge in G, the following inequality holds:
(G C) S (G.:;))
(14)
This equation indicates when setting in Eq. (8), the IBCircuit objective upper bounds the mutual information of g. and C. Hence, optimizing the IBCircuit objective encourages C to be less related to components in G, which are irrelevant to Y.
Proof. We follow the proof in (Yu et al., 2022). Suppose g. C, 9, and Y satisfy the Markov condition (Y.9.) g + (Achille & Soatto, 201 8a). Then we have the following inequality:
T(C; 9) 2 T(C,Y, = T(C;g.) T(C;Y|g.).
(15)
Since and G2 are independent, we have H(Y]G,) = HYY and (Y, < (Y))).. Then we have:
I(C; ((C; H( HY C) 2 H(Y H( ((;
(16)
Combine Eq. (15) and Eq. (16), we:
T(C < T(C:g) T(C;Y)
(17)
Suppose G, G, 9, and Y satisfy the Markov condition (Y, + 9 + 9 (Achille & Soatto, 2018a). Then, combine with Eq. (17) we have:
9.) < - -
(18)
deterministic given 9. and C, since we can recover 9 by combining 9. with C. Then for the left part in Eq. (18), we
have:
(())9.) = (G.,C: = Gs) + I(C:9.19. 2 ((:
(19)
Therefore, by combining Eq. (18) and Eq. (19) we have the follow inequality:
T(C; 9.) < (():9.) < -(::9)-1(:
(20)
which proofs Eq. (14).
A.2. Derivation of MI Loss
The Kullback-Leibler (KL) divergence is measure of how one probability distribution diverges from second, expected probability distribution. In this section, we derive the KL divergence formula between two Gaussian distributions Q; and P., and further calculate the average KL divergence over a batch of data.
1. KL DIVERGENCE FORMULA
Given two Gaussian distributions: P. WN o.) and Q N(A,, + (1 -xxe) (1 A1)os). The KL divergence
DEL P.) is given by:
of + log
2g + (mQ, - mp,.i² -1)
DEL (Qrl))))
(21)
Q
Substituting the parameters:
1 1 log
of
(1 - X,) + (Arh, + (1 A,pp 14)2 -1)
(Q.ll)))
+
(22)
(1 A1 P0?
o7
13

---

IBCircuit: Towards Holistic Circuit Discovery with Information Bottleneck
Simplifying the terms:
(QilP:)) 1 log (1 ),)² +
(1 2z) X2 114)
(23)
0
= looo - Ax) + (1 1;)
X7 (h+ 14)
+
(24)
20
A.2.2. AVERAGE KI. DIVERGENCE OVER A BATCH
For batch of data points, the average KL divergence is:
1 DkL.
DKL (Q:1:))
(25)
i=1
Substituting the expression for DkI.(Q:l(P3)
(1- 11-2pp 11)²
X2 - ui)²
DKL = Σ
log(1 - Ax) +
+
(26)
2
20
Breaking down the summation:
(1
11)2
22 - -1)
DKL Σ
log(1 - X1) + Σ
(27)
2n
2no
i=1
1=1
1=1
Assuming Ag -- Ei=r Ar), Bg = Σm=
d11li-pe).
(Ag)²
B3
DKL = -- Ag +
2n +
(28)
2n
B. Experimental Setting
For node-level circuit discovery, we set the learning rate to 0.05, trained for 1300 epochs using the Adam optimizer for IB weights, and a to For edge-level circuit discovery, we set the learning rate 0.1. trained for 3000 epochs using the Adam optimizer for IB weights with learning rate warm-up scheduler (200 warm-up steps), and between 0.01 and I to balance the sparsity of the circuit and the KL divergence (see the analysis of the parameter in Section 5.3), To obtain the performance of IBCircuit with fewer edges, we selected higher values of a.
14


Reviews:
[{"review_id": "rTnTSdE1Wk", "reviewer": "Reviewer_eJCi", "summary": "This paper explores circuit discovery in pretrained language models. The proposed method, i.e., IBCircuit, leverages the information bottleneck principle to holistically identify and optimize circuits without needing specific corrupted activations for different tasks. It is demonstrated that IBCircuit can identify more relevant and minimal circuits compared to existing methods, particularly in tasks like Indirect Object Identification and Greater-Than calculations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper. We appreciate your time and effort in reviewing our work. We would like to address your questions and concerns below. \n\n>**Weakness 1**: Unclear explanation of \"distorted information flow\" in Eq.1: Why use input corruption instead of direct KL minimization?\n\nThanks for the comment. The distorted information flow is used for modeling circuits with learnable parameters. There is no explicit circuit within the LLM, thus we can not directly minimize KL. In the distorted information flow, we model the learnable circuit with Gaussian, i.e. input corruption. Therefore, IBCircuit can optimize the circuit with gradient descent. It is the common practice in variational methods [1][2]. Using input corruption to model circuit is common in optimization framework for circuit discovery. For example, the baseline Subnetwork Probing [3] uses a hard-concrete random masking as input corruption to model the circuit.\n\n[1] Deep Variational Information Bottleneck. ICLR 2017.\n\n\n[2] Auto-Encoding Variational Bayes. ICLR 2014.\n\n[3] Low-complexity probing via finding subnetworks. NAACL 2021.\n\n>**Weakness 2**: The proofs presented for the theoretical bounds lack novelty. The derivation of the lower bounds appears to be a straightforward application of previous works.\n\nThanks for the comment. We derived a novel upper bound, and since previous subgraph discovery work has the same optimization objective as circuit discovery, we adopted the same form of lower bound.\n\n* **Upper Bound:** The traditional upper bound can not be directly applied to the circuit discovery, as the random variables $\\mathcal{G}$ and $\\mathcal{C}$ are computational graphs. Therefore, we derive a new upper bound.\n\n* **Lower Bound:** In previous studies, the lower bound derived for subgraph discovery aims to identify critical subgraphs that preserve information similar to the complete graph. This aligns with the objective of circuit discovery, where the goal is to find circuits that maintain the performance of the full pre-trained model. Therefore, we incorporate the same bound using the notation of Information Bottleneck Circuit to ensure self-consistency in the proposed method.\n\n>**Weakness 3**: No evidence for applicability to larger models.\n\nThanks for the suggestions. We agree that additional results on larger models would enhance our contribution. We include the results in [this anonymous link](https://anonymous.4open.science/r/IBCircuit_ICML25_rebuttal-76E3/). Due to our limited GPU resources, we have not yet completed all experiments. We will keep the remaining results updated in the next few days.", "claims_and_evidence": "Mostly.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked the proof for the VIB.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Circuit discovery is very important to promote the explainability of black-box pretrained language models", "essential_references_not_discussed": "Not I can think of.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Circuit discovery is a crucial field for enhancing the explainability of language models.\n- The paper is well-written and aptly frames the circuit discovery challenge within the Variational Information Bottleneck (VIB) framework.\n- The experiments conducted demonstrate the method's effectiveness on a smaller GPT-2 model, providing preliminary evidence of its utility.\n\n**Weaknesses:**\n\n- Eq. 1 introduces the concept of \"distorted information flow\", which is confusing and not properly explained in the paper. If the goal is to frame this as an Information Bottleneck problem, why is there a need to corrupt the input instead of directly minimizing the KL divergence with the same input? This aspect could benefit from further clarification.\n\n- The proofs presented for the theoretical bounds lack novelty. The derivation of the lower bounds appears to be a straightforward application of previous works.\n\n- It's uncertain whether this method is scalable to larger models like LLAMA, which have been explored in recent studies in circuit detection. Further experimentation on such models would be beneficial to validate the method's applicability across different scales.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please address my concerns in the weakness in \"Other Strengths And Weaknesses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "IlU2FOKUbh", "reviewer": "Reviewer_tPrr", "summary": "This paper addresses the recent surge of interest in discovering circuits inside language models that are sufficient for faithfully explaining the behavior on specific tasks. The key contribution is to provide a method grounded in the Information Bottleneck (IB) method, which is optimized directly (avoiding expensive iterative patching), works without manually constructed corrupted inputs, and outperforms various existing methods.\n\n## update after rebuttal\nThe authors have clarified key aspects and provided new results. The point remains that the clarity of the paper needs to be very strongly improved with regard to how the variational IB is implemented -- I have increased my score trusting the authors that they will keep their promise to revise this aspect thoroughly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your effort in reviewing our paper and providing constructive feedback. We would like to address your concerns below and revise the corresponding parts in the revised version.\n\n**Theoretical Claims:** We would like to further summarize the IBCircuit. IBCircuit can be regarded as the application of the Information Bottleneck principle to circuit discovery with our proposed appropriate adaptations.\n* **Why use variational approximation?** The mutual information is intractable and IB can not be directly applied to circuit discovery. The variational method is introduced to estimate the bound of mutual information. The problem is also analyzed in VAE[1] and VIB[2] in deep representation learning. \n* **Random Variable.** In deep neural network[1][2], the input $X$ are random variables. Each concrete input string $x$ can be viewed as an instance sampled from the task distribution $X$, e.g. IOI task. Similarly, $Y$, $\\mathcal{C}$ and $\\mathcal{G}$ are also random variables, and $[x,y,c,g]$ is an concrete instance. We provide an analysis on random variables, but this formulation can be trained with instances[1][2]. \n* **Gaussian in Appendix A.2.** Gaussian is the variational prior[1][2]. Gaussian is widely used as variational priors due to the reparameterization trick and convenient KL calculation[1][2]. Following existing literatures, in IBCircuit, the prior of circuit $\\mathcal{C}$ is set as Gaussian. IBCircuit will learn the posterior distribution of $\\mathcal{C}$ over task $X$, and we report the mean of the learned distribution $\\mathcal{C}$ as the final result.\n* **$I[Y; \\mathcal{C}]$ Computation.** For an instance $x$ in a batch, we compute the KL divergence between the output of the perturbed model $y_{\\mathcal{c}}$ and the output of the clean model $y$. Then we take the average over the batch. For each instance $x$ in the batch, the corresponding circuit $c$ is generated from Gaussian with reparameterization trick[1].  Therefore, each instance in the training stage is deterministic.\n* **$I[\\mathcal{G}; \\mathcal{C}]$ Computation in Appendix A.2.** We compute the KL between the posterior distribution and Guassian prior of the circuit over the batch. As the prior is Gaussian[2], we can calculate the KL as in Appendix A.2.\n\n**We will make them clear in the revision.**\n\n[1] Deep Variational Information Bottleneck. ICLR 2017.\n\n[2] Auto-Encoding Variational Bayes. ICLR 2014.\n\n>**W1**: The writing around Proposition 4.2. \n\nThanks for the comment. We will provide the definitions of A and B in the main text. $L_{MI}(G; C)$ is defined as the upper bound of $I(G; C)$, i.e. the KL between the posterior distribution and Gaussian prior of the circuit over the batch.\n\n\n>**W2**: Comparison to Subnetwork Probing.\n\n* **Whether the IB or the variational approximation is key.** As mentioned above, the key is IB. The variational approximation makes  IB trainable in deep neural networks.\n* **What is responsible for the performance difference?** We think the objective of IBCircuit is responsible for the performance difference. Gaussian perturbations and hard masking are just two different ways of modeling circuits with learnable parameters, not the key. (1) In $I[Y;\\mathcal{C}]$, IBCircuit calculate KL while SP calculate cross-entropy. KL measures the distribution of the output logit vector. While cross-entropy only focuses on the ground truth, i.e. one element on the logit vector. SP ignores the logits of other tokens. (2) In $I[\\mathcal{G}; \\mathcal{C}]$, IBCircuit ensures the circuit does not receive irrelevant information from the whole computation graph. In contrast, SP only encourages sparsity. Even if the circuit obtained by SP is sparse, it may contain some irrelevant information.\n\n>**W3&Theoretical Claims**: Additional Experimental Results. \n\nThanks for the suggestions. We agree that additional comparisons would enhance our contribution. We include the results about (1) the comparison to IG-EAG, (2) more tasks, and (3) larger models in [this anonymous link](https://anonymous.4open.science/r/IBCircuit_ICML25_rebuttal-76E3/). Due to our limited GPU resources, we have not yet completed all experiments. We will keep the remaining results updated in the next few days.\n\n>**Questions For Authors**: What is the computational cost? \n\nTaking the IOI task as an example, running the official ACDC implementation on a single H20 GPU takes approximately 16 hours, while EAP takes about 1 minute, and the proposed method requires roughly 1.2 hours. Although EAP has a faster runtime, it requires designing corresponding clean and corrupted input pairs to compute the gradients. Moreover, the proposed method outperforms EAP on both IOI and Great-than tasks. Furthermore, our approach is significantly more scalable compared to ACDC.", "claims_and_evidence": "I believe the claims as made in the Abstract, the Introduction, and the Conclusion are largely supported by evidence.\n\nRegarding the claim that the method outperforms \"recent related work\" (made in the abstract), I believe comparison to IG-EAG [1] is missing. I discuss this further under \"Other Strengths and Weaknesses\".\n\n[1] Hanna et al, Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms, COLM 2024", "methods_and_evaluation_criteria": "Yes, the evaluation criteria make sense. The paper uses appropriate standard benchmarks.\n\nA weakness is that evaluation is limited to two tasks (IOI and Greater-Than) and a small model (GPT2-Small), even though a few more relevant tasks are available. In particular, the paper [1] cited above uses six such tasks.", "theoretical_claims": "I have questions about the basic formulation in terms of IB.\nSection 3 defines the circuit and the full model to be computation graphs.\nSection 4 introduces the two key information terms based on IB:\n * informativity about true output: I[Y; C]\n * informativity about full graph: I[G; C]\n\nNow a few questions [NOTE after rebuttal: I now understand this. The point remains that the clarity of the paper needs to be very strongly improved in this regard -- I have increased my score trusting the authors that they will keep their promise to revise this aspect thoroughly]:\n* (a) In general, I[...;...] is applied to random variables, but here it is not clear where the randomness comes from. Equation (3) has the subscript P(C]G) -- literally, this appears to suggest that the circuit C (i.e., a computation subgraph) itself is a random variable, a random subset of the full graph -- but that would seem at odds with later parts of the paper, so probably is not what is meant. I assume that C here denote activations of the nodes in the graph, so that P(C]G) is a distribution over activation patterns in the circuit, defined by the randomness of the Gaussian perturbations, right? \n* (b) Following up, in I[G; C] what is the random variable G? Is it the distribution of activation patterns in the full graph, with the randomness coming from sampling different input strings? But this is actually at odds with Appendix A.2.1, where the KL divergence of two Gaussians is considered. But the activations in G are not Gaussians -- they are deterministic when fixing the input string. Where does Gaussianity come in? Maybe some kind of Gaussianity is imposed also on the full model's activations (which is how I understand Appendix A2), but then this needs to be made very explicit in the main paper.\n* (c) Regarding I[Y; C], it remains unclear in Section 4 how this is computed or approximated. Is the idea to construct a variational bound by computing the average (across the randomness in the Gaussian perturbations \\epsilon) cross-entropy of the perturbed model on the outputs Y?", "experimental_designs_or_analyses": "The soundness of the experimental design appears good. My main worry is that the basic design of the method isn't sufficiently clear from the paper (see under Theoretical Claims -- though from the rebuttal it does make sense to me), but beyond this the design seems fine.", "relation_to_broader_scientific_literature": "The key contributions are to propose a circuit discovery method that\n- is based on a holistic differentiable objective\n- works even without manually created corrupted activations", "essential_references_not_discussed": "All essential references are cited, though I would like to ask the authors to provide further comparison as detailed at other parts of this review.", "other_strengths_and_weaknesses": "Strengths:\n\n- Grounding in IB in principle permits a principled motivation of the approach (though see doubts below under “Theoretical Claims”)\n\n\n- The idea of a circuit discovery method based on end-to-end optimization is very appealing, and may hold the promise to be much more scalable than  iterative methods such as ACDC.\n\n- The approach is competitive or even better than existing methods compared to.\n\n\n- It is remarkable that the proposed method performs well even without using explicit corrupted data (one may wonder if it could perhaps work even better if one replaced the Gaussian perturbations with activations computed on randomly selected corrupted data, for more targeted corruptions).\n\n\n(Other) Weaknesses beyond those mentioned elsewhere:\n\n- [updated after rebuttal] (major) The description of the IB formulation and the variational prior/posterior needs to be made a lot more explicit. I understand the method, but only after the rebuttal. The paper itself needs improvement.\n- (minor) Writing needs to be improved around Proposition 4.2: To the reader arriving here, the statement (5) is meaningless, since A and B have not been introduced. Hence, for the reader of the main paper, L_{MI} essentially remains undefined.\n- The method is very similar to Subnetwork Probing (SP, Cao et al, cited). The method is compared to, but it is hard to tell which conceptual aspects about the proposed method are responsible for outperforming that method. In particular that the current ms resorts to variational approximations, it is not clear whether the generality of IB is key, or some aspect of the variational approximations. The I[Y;C] term doesn't seem so different from the recoverability-part of the SP objective. It seems the clear conceptual difference lies in the fact that the proposed method uses Gaussian perturbations rather  than hard-concrete random masking, and that the regularization is based on Mutual Information rather than the SP objective. Is this responsible for the performance difference? I think this could in principle be checked in additional experiments. [the authors have provided some new material here]\n- The success of the method is only shown on two tasks in a small model (GPT2-small). What are the prospects for scaling the approach to larger models than GPT2Small? Notably, IG-EAG [1] has been evaluated on more tasks and also shown to scale to larger models (Appendix K in [1]). Relatedly, IG-EAG would be good to include as a baseline. [the authors have provided some new material here]", "other_comments_or_suggestions": "None", "questions_for_authors": "I have posed questions about the IB formulation under \"Theoretical Claims\". Satisfactory clarification could, together with addressing of other weaknesses and concerns, change my evaluation of the paper. [the authors have answered this]\n\nWhat is the computational cost? It is clear that the proposed method may be more scalable than ACDC, but then it seems much more expensive than (IG-)EAP, depending on the number of iterations needed for training? [the authors have answered this]", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M2rusiL4Cn", "reviewer": "Reviewer_vc6F", "summary": "This paper proposes a method to identify circuits -- interpretable subgraph of a computation graph of a neural net -- via the information bottleneck principle. The key idea of the paper's' technique is estimating the IBCircuit objective via noise injection. Empirical results on several well-known tasks, such as IOI and Greater-Than, demonstrate that the method can faithfully (measured by e.g. the logit diff metric) recover circuits with fewer circuit components compared to previous methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper and providing constructive feedback. We would like to address your questions and concerns below.\n>**Claims and Evidence**: **Q1**. Discuss the causality of circuit identification. **Q2**. Sampling methods in previous circuit studies IOI, Greater-than.\n\n**Q1**: We appreciate your insight on causality. **We would like to clarify that IBCircuit is causal.** As in causal inference literature [1], our method employs controlled noise injection with learned IB weights as implicit causal interventions. And we do not need explicit data corruption. Unlike conventional ablation-based approaches, IBCircuit avoids distributional biases from discovery-phase ablations. Therefore, we can overcome the robustness limitations mentioned in [2] and find more reliable circuits.\n\n**Q2**: Thank you for highlighting this detail. We acknowledge that IOI introduces mean ablation in Section 2.2, while Greater Than uses permutation sampling. We will clarify these distinctions in the revised version.\n\n[1] A Causality-Aware Perspective on Domain Generalization via Domain Intervention.\n\n[2] Transformer Circuit Faithfulness Metrics Are Not Robust. COLM2024.\n\n>**Methods and Evaluation**: **Q1**. Conduct more rigorous statistical tests. **Q2**. Experiment with larger model architectures.\n\n**Q1**:  We provide additional results under various ablation methods in [this anonymous link](https://anonymous.4open.science/r/IBCircuit_ICML25_rebuttal-76E3/), supplementing the random ablation presented in our initial draft. We also include the suggested evaluations.\n\n**Q2**: We add comparison on CodeLLaMA-13B in the same anonymous link above.\n\n>**Broader Scientific Literature and Essential References**: Reference and comparisons with: (1)Edge pruning, (2)Sparse feature circuit.\n\nThank you for the valuable feedback. We include the comparison with the two baselines in [this anonymous link](https://anonymous.4open.science/r/IBCircuit_ICML25_rebuttal-76E3/).  Due to our limited GPU resources, we have not yet completed all experiments. We will keep the remaining results updated in the next few days. Additionally, we will cite the two papers in the revision.", "claims_and_evidence": "I find that the circuit finding performance (measured in recovery metric and circuit components) is generally well supported with the paper's experiments. The authors have shown that IBCircuit can be (resp.) relative or outperform well-established techniques (e.g. ACDC) in terms of false positive rates on GT and IOI tasks. In the low-node regime, IBCircuit can also outperform these techniques in terms of recovery metrics.\n\nHowever, I do find that the completely foregoing corrupted data to be a bit troubling. The key idea of such a contrast set is to causally intervene on the model's behavior, so that the method can identify model components that are meaningfully contributing to the actual implementation of an algorithmic task. Here, I'd argue that the author is identifying a slightly different subgraph (which may or may not be the circuit), which may or may not be causal to the implementation of this task, and I'd encourage the author to discuss more on these subtleties. \n\nThere is also a small error in the author's claim that the authors of IOI [1] and Greater-Than [2] have used mean dataset values for their circuit discovery analyses (line 93), where they have instead used permutation sampling to obtain such values. I believe these are subtle but critical points in the design philosophy of the goal of circuit finding. This type of resampling ablation has long been advocated in [3] which precedes both [1][2].\n\n[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, https://arxiv.org/abs/2211.00593\n\n[2] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, https://arxiv.org/abs/2305.00586\n\n[3] Causal Scrubbing: a method for rigorously testing interpretability hypotheses, https://www.lesswrong.com/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing", "methods_and_evaluation_criteria": "The method is solid. The authors have clearly defined a learning objective and designed feasible implementations.\n\nI believe the paper could benefit from a more rigorous set of evaluations. For example, several works [1][2] have discussed that some circuit evaluation metrics discussed (and used) in this paper are not robust. Given the closeness of some of these results, it'd be good to have statistical tests, or bring in some of the evaluation methods proposed in these papers, for more rigorous testing.\n\nAdditionally, the authors have stated that \"(existing methods) do not scale well with the model size\", but the authors have conducted their experiments on a similar scale (i.e. GPT-2 small). If this is indeed a problem that the authors are aiming to address, they should experiment with larger architectures (e.g. 7B or 13B models) as other scalable circuit finding methods (e.g. edge pruning) have demonstrated [3].\n\n[1] Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms, https://arxiv.org/abs/2403.17806\n\n[2] Transformer Circuit Faithfulness Metrics are not Robust, https://arxiv.org/abs/2407.08734\n\n[3] Finding Transformer Circuits with Edge Pruning, https://arxiv.org/abs/2406.16778", "theoretical_claims": "IBCircuit is grounded in information theory, which is a good idea to start with. I have checked the authors mathematical claims (e.g. derivations, optimization objectives) and do not see any error.", "experimental_designs_or_analyses": "Please check my review in the __Methods And Evaluation Criteria__ section, where I discussed some concerns regarding the evaluation metrics.", "relation_to_broader_scientific_literature": "The authors have generally situated this paper in classical circuit finding literatures well. But I believe there should be a more direct, head-to-head comparison against some recent developments in circuit finding techniques that are more scalable, e.g. edge pruning [1] and sparse feature circuits [2]. In particular, [1] is able to scale their circuit finding technique to 13B models and [2] does not require a contrast set. There are also some minor errors in the literature review discussed in the __Claims and Evidence__ section.\n\n[1] Finding Transformer Circuits with Edge Pruning, https://arxiv.org/abs/2406.16778\n\n[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, https://arxiv.org/abs/2403.19647", "essential_references_not_discussed": "See previous section on **Relation To Broader Scientific Literature**. The authors should reference [1] as it is also a scalable approach, and [2] is a method that the authors cite, but does not properly discuss its independence from contrast set. \n\n\n[1] Finding Transformer Circuits with Edge Pruning, https://arxiv.org/abs/2406.16778\n\n[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, https://arxiv.org/abs/2403.19647", "other_strengths_and_weaknesses": "- Strength: clearly grounded methods with information theory support.\n- Weakness: lack of proper discussion with prior works (see above), the core philosophy of the goal of circuit finding (can we claim that the circuit found by IBCircuit is causal?), and perhaps more rigor in evaluation.", "other_comments_or_suggestions": "The writing is generally in good condition.", "questions_for_authors": "I'm happy to read your take on several points raised above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R6n02OAOUm", "reviewer": "Reviewer_ZTi2", "summary": "This paper proposes a method, IBCircuit, for discovering circuits based in the information bottleneck method. Specifically, by interpolating between the original activation and the mean batch activation of a component or edge, and learning these coefficients for all nodes or edges simultaneously, one can discover circuits without needing to design counterfactual activations manually (as is often done in prior work).\n\nThe proposed optimizes two objectives: one that encourages the circuit $\\mathcal{C}$ to be informative on output $Y$, and one that encourages the circuit $\\mathcal{C}$ to be sparse. It is found that the proposed method achieves better AUROC than prior methods, that the sparsity term is slightly helpful (but not useful on its own), and that the proposed method is generally better for preserving the full model’s task-specific information—more so on IOI than Greater-Than.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper and providing constructive feedback. We would like to address your questions and concerns below.\n>**Claims and Evidence**: **Q1**: Not sure what claims of avoiding manually constructed counterfactual activations mean. **Q2**: In the evaluation, don’t we need counterfactual activations from other inputs to evaluate the circuits?\n\n**Q1**: We emphasize that our circuit discovery process does not require manually designing counterfactual inputs. While mean ablation also avoids explicit counterfactual input design, it underperforms the resampling ablation [1]. Additionally, we agree with the comment that \"the performance will probably correlate with the similarity of the discovery and evaluation patching methods\"[2]. The correlation arises from the biases introduced by different counterfactual inputs as corrupted data. In contrast, IBCircuit leverages Gaussian as corrupted activations rather than corrupted data, which eliminates such biases.\n\n**Q2**: To ensure fair comparison, we follow existing baselines and utilize counterfactual activations derived from other inputs during evaluation. This demonstrates that our method achieves robust results even when assessed using ablation strategies distinct from the ablation employed in circuit discovery.  \n\n[1] Causal Scrubbing: a method for rigorously testing interpretability hypotheses.\n\n[2] Transformer Circuit Faithfulness Metrics Are Not Robust. COLM2024.\n\n\n>**Methods And Evaluation Criteria**: **Q1**:Treating prior circuits as \"ground-truth\" risks missing valid components. **Q2**: KL divergence recovers far more than just task-specific behavior.\n\n**Q1**: We agree with your opinion that treating manually validated circuits as ground truth may disadvantage newer discoveries. However, since baselines [1][2] also evaluate against these manually validated results as ground truth, we followed their protocol to ensure fair comparison. Beyond this Grounded in Previous Work experiment, we further demonstrate the advantages of our method through comparative analyses on metrics such as Faithfulness and Minimality.\n\n**Q2**: In [1], Appendix C (\"Discussion of Metrics Optimized\") argues that:(1) Optimizing for low KL divergence is the simplest and most robust metric applicable across diverse tasks;(2) KL divergence is universally applicable to next-token prediction tasks because it does not require predefined output labels (unlike metrics such as logit difference, which necessitate specifying the exact tokens between which differences are computed). Based on these insights, we adopted KL divergence as our optimization metric.\n\n[1] Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023.\n\n[2] Attribution Patching Outperforms Automated Circuit Discovery. NeurIPS 2023.\n\n>**Experimental Designs Or Analyses**: Evaluation uses counterfactual activations dissimilar to baseline discovery methods.\n\n**We would like to clarify that our evaluation is the same as the existing methods.** We agree with the statement that \"the performance will probably correlate with the similarity of the discovery and evaluation patching methods\". For this reason, in our experiments, we adopt unified counterfactual activations derived from alternate inputs for all baseline methods, as well as our proposed approach to ensure fair comparison. The proposed method is robust and does not rely on this ablation strategy for circuit discovery. We appreciate your suggestion and will explicitly discuss this point in the revision. \n\n\n>**Citation and writing issues mentioned in the weakness and suggestions.**\n\nThank you for your suggestion. We will revise the corresponding part in the new version. We will discuss and cite the recommended papers.\n\n\n>**Question**: Runtime and GPU.\n\nThe IBCircuit runs the IOI task on one H20 GPU for about 1.2 hours. The official implementation of ACDC takes about 16 hours on the same device. We will add the comparison in the revised version.", "claims_and_evidence": "The main claims are supported by a reasonable amount of quantitative evidence. My main concern is that the circuits found in the IOI and Greater-Than papers are taken as ground-truth circuits; see Methods and Evaluation. \n\nIt is also claimed that one of the main contributions is that we do not need to manually construct task-specific counterfactual activations. But this is already common: mean ablations, which are used in many circuit discovery papers, do not require constructing targeted counterfactual dataset pairs, but instead simply taking the mean activation of a component over some general text corpus. One can instead define the mean as over a task dataset, but this does not have to be the case—it simply performs better this way. Optimal Ablations [4; references provided under “Essential References” section] also does not require task-specific activations. With respect to the “manually constructed” part, I’m not sure what this means, exactly: patching with an activation from a counterfactual input does not require us to manually design the activation either; we just need to provide a counterfactual input and the model will produce the new activation automatically. Also, in the evaluation (Sec. 5), don’t we need counterfactual activations from other inputs to evaluate the circuits?", "methods_and_evaluation_criteria": "In Figure 2, the circuits of Wang et al. for IOI and Hanna et al. for Greater-Than are treated as ground-truth circuits. These two circuits may be *high-precision* circuits that were manually validated, they are not guaranteed (nor likely, in my opinion) to have recalled all of the important causal dependencies. If this is true, then certain methods maybe unfairly penalized for recovering components that actually were important, but were simply not found by Wang et al. (2023) nor Hanna et al. (2023).\n\nThe KL Divergence is also a very broad metric that recovers far more than just task-specific behavior. A sparser circuit will generally have lower faithfulness, even if it perfectly recovers task performance. The logit difference is a good metric, so I don’t know whether targeting low KL divergence will be helpful in addition to the other metrics proposed.", "theoretical_claims": "Proofs in appendix look good to me.", "experimental_designs_or_analyses": "Evaluating circuits with counterfactual activations from alternate inputs is not very similar to the way that the circuit is discovered. This could unfairly penalize methods that discover circuits using very different kinds of activation patching, including the baselines. This actually strengthens the argument in favor of IBCircuits, as being able to generalize to new kinds of patching methods suggests that the discovered circuit is more robust. Nonetheless, I think this should be discussed explicitly, as it doesn’t seem like a fair comparison across all methods; the performance will probably correlate with the similarity of the discovery and evaluation patching methods.", "relation_to_broader_scientific_literature": "The mechanistic interpretability essentials are discussed and well contextualized. However, there is a line of highly related recent work (over 4 months from the submission deadline) that is not cited, but should be. Perhaps most related is the Optimal Ablations paper [4], which proposes a method called UGS that functions similarly to the coefficient learning approach here. Information Flow Routes [3] is conceptually similar in that it uses information-theoretic notions to define a circuit discovery method, but the actual method is quite different.\n\nAdditionally, the use of noise to ablate components was proposed by Meng et al. (who are cited in the paper), but this is not directly attributed to them and should be. Also cite the original activation patching papers: Vig et al. [1] and Finlayson et al. [3].\n\nSee Essential References for references.", "essential_references_not_discussed": "* Activation patching: cite Vig et al. [1], Finlayson et al. [2].\n* When referring to adding Gaussian noise to a component, cite Meng. They are already cited elsewhere in the paper, but this method is not attributed to them as it should be.\n* Cite information flow routes [3]. This method should be discussed and contrasted with the proposed method.\n* Cite optimal ablations [4] when referring to ablating multiple components at once and learning coefficients on them. The UGS method they propose also involves interpolating between real activations and noise; as far as I can tell, the main difference between UGS and this method is that UGS involves learning the ablation value, rather than defining it as the mean within a batch. Otherwise, they are very similar.\n\nReferences:\n\n[1] Vig et al. (2020). “Investigating Gender Bias in Language Models Using Causal Mediation Analysis.” NeurIPS. https://proceedings.neurips.cc/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html\n\n[2] Finlayson et al. (2021). “Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models.” ACL. https://aclanthology.org/2021.acl-long.144/\n\n[3] Ferrando & Voita (2024). “Information Flow Routes: Automatically Interpreting Language Models at Scale.” arXiv (posted ~1 year before ICML submission deadline, then published at EMNLP within the 4-month  window). https://aclanthology.org/2024.emnlp-main.965/\n\n[4] Li & Janson (2024). “Optimal ablation for interpretability.” arXiv (posted > 4 months before ICML submission deadline, then published at NeurIPS within the 4-month window). https://openreview.net/forum?id=opt72TYzwZ", "other_strengths_and_weaknesses": "Strengths:\n* I like that the performance of the circuit and the faithfulness of the circuit to the full model are separate metrics. This should be more common.\n* The writing is generally easy to follow.\n\nWeaknesses:\n* Figure 1 is not very clear. Maybe color could be used in a more consistent way across both nodes and edges to indicate noising? And maybe the default vs. noising colors could be made more distinct from each other?", "other_comments_or_suggestions": "* L78: rephrase final clause of the sentence at end of Sec. 2? I wasn’t sure what this meant.\n* Could T be added as a subscript to Y in Sec. 4 to make it clear which tasks Y is over?\n* Use \\mathcal for C and G in the appendix for consistency.", "questions_for_authors": "* Appendix B: This is quite a few epochs. What was the runtime of your method, and on what kind of GPU? I just want to verify that it’s at least an order-of-magnitude better runtime than ACDC (which is an easy bar to pass), and verify scalability.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

This paper proposes IBCircuit, a novel approach for circuit discovery in language models based on the Information Bottleneck principle. Unlike previous methods that require manually designed corrupted activations, IBCircuit provides an end-to-end optimization framework that can be applied to any task without tedious corrupted activation design. The authors demonstrate that IBCircuit identifies more faithful and minimal circuits in terms of critical node components and edge components compared to recent related work on the Indirect Object Identification (IOI) and Greater-Than tasks. The method shows promising results in terms of faithfulness metrics and scaling potential, offering a new direction for interpretability research in large language models.

**Reasons to Accept**

- Circuit discovery is a crucial field for enhancing the explainability of language models (eJCi, tPrr)
- The method is competitive or even better than existing methods in terms of AUROC and faithfulness metrics (tPrr, vc6F, ZTi2)
- The idea of circuit discovery based on end-to-end optimization is very appealing and more scalable than iterative methods like ACDC (tPrr)
- The method is significantly faster than ACDC, taking about 1.2 hours compared to 16 hours for ACDC on the IOI task using the same hardware (ZTi2)

**Suggestions for revisions**  
1. The formulation and variational implementation of the Information Bottleneck approach is not sufficiently explained in the paper, making it difficult for readers to understand how the method works (tPrr)
2. The method has only been evaluated on a smaller GPT-2 model and limited tasks (IOI and Greater-Than), raising questions about its scalability to larger models like LLAMA (eJCi, tPrr, vc6F) 
3. The proofs for theoretical bounds lack novelty, appearing as straightforward applications of previous works (eJCi) 
4. Missing comparisons with recent scalable circuit discovery methods like Edge Pruning and IG-EAG (vc6F, tPrr) 
5. Equation 1 introduces "distorted information flow" without proper explanation, raising questions about why input corruption is needed instead of direct KL minimization (eJCi) 
6. The evaluation treats previous manually validated circuits as "ground truth," which may unfairly penalize methods that discover valid components not identified in those circuits (ZTi2)
7. Using KL Divergence as a faithfulness metric may recover more than just task-specific behavior (ZTi2) 
8. The paper misses citation of several relevant works, particularly regarding information-theoretic approaches to circuit discovery (ZTi2)
Decision: Accept (poster)