Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
Authors: Yinhan He
Wendy Zheng
Yushun Dong
Yaochen Zhu
Chen Chen
Jundong Li
Venue: ICML 2025 poster (2025)
Keywords: Mechanistic Interpretability
Modular Circuits
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We propose a modular circuits discovery method called ModCirc.

Abstract: Mechanistic interpretability (MI) research aims to understand large language models (LLMs) by identifying computational circuits, subgraphs of model components with associated functional interpretations, that explain specific behaviors. Current MI approaches focus on discovering task-specific circuits, which has two key limitations: (1) poor generalizability across different language tasks, and (2) high costs associated with requiring human or advanced LLM interpretation of each computational node. To address these challenges, we propose developing a ``modular circuit (MC) vocabulary'' consisting of task-agnostic functional units. Each unit consists of a small computational subgraph with its interpretation. This approach enables global interpretability by allowing different language tasks to share common MCs, while reducing costs by reusing established interpretations for new tasks. We establish five criteria for characterizing the MC vocabulary and present ModCirc, a novel global-level mechanistic interpretability framework for discovering MC vocabularies in LLMs. We demonstrate ModCirc's effectiveness by showing that it can identify modular circuits that perform well on various metrics.

Towards Global-Ievel Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
Yinhan He
Wendy Zheng?
Yushun Dong²
Yaochen Zhu'
Chen Chen
Jundong Li
Abstract
in/output
att. head
Mechanistic interpretability (MI) research aims understand large language models (LLMs) by identifying computational
A MLP neuron hidden shate comp, edge
SCS
of model components with associated functional in- terpretations (Fls)~that explain specific model behaviors. Current MI research mainly focus on discovering task-specific circuits, which have two key limitations: (1) low generalization ability across diverse language tasks and (2) high costs due to the need for human or advanced LLMs interpret each computational node. To address these challenges, we propose novel modular circuit (MC) vocabulary of task-agnostic func- tional units, each containing a small computa- tional subgraph with its interpretation obtained by examining the subgraph's behavior on extensive corpora. By allowing different language tasks share common MCs, our approach enables global interpretability while reducing costs by reusing established interpretations for new tasks. Besides, we propose five criteria for characterizing the MC vocabulary and present ModCire, a novel global- level MI framework for discovering MC vocabu- laries in LLMs. We demonstrate ModCire's 's effec- tiveness by showing that it can identify modular circuits that perform well on various metrics."
Components & edges
Common MC
circuits
SCS
(a) MCs provide cross-task insight.
vocabulary
FI: this MC is capturing patients' symptoms.
match
New unseen SCS (b) Func. interpretations (FI) extracted from MC vocab.
Figure I. Advantages of modular circuits (MC) as an LLM mecha- nistic interpretation tool. (a) Common MCs (shown in sky blue) shared across significant computational subgraphs (SCSs) of dif- ferent language tasks reveal cross-tat task interpretability insights. (b) MCs identified in new tasks can be matched against vocabulary of known MCs, allowing direct reuse of existing functional inter- pretations (FI) rather than generating new ones.
2023; Chang et al., 2024; Wei & Liu, 2025) has trig- gered the interest to understand their internal mechanisms and decision-making processes. Mechanistic interpretabil- ity (Wang et al., 2022: Lieberum et al., 2023; Cunning- ham al., 2023; Nanda et al., 2023; Conmy et al., 2023; Bereska & Gavves, 2024; Rai al., 2024; Nainani al., 2024; Nainani, 2024), which aims to reverse engineer how these models process information and make decisions, has emerged as promising approach. For various language tasks such as indirect object identification (IOI) (Wang et al., 2022), modular sum (Nanda et al., 2023), and subject-verb agreement (Marks et al., 2024), current research has suc- cessfully identified many computational circuits, i.e., the significant computational subgraph (SCS) of the LLM that contributes the most to the LLM's output, along with the nodes' functional interpretations (FIs). Specifically, the nodes in these computational subgraphs are LLM compo- nents such as attention heads and feed-forward neurons, with edges representing direct computational flow between
1. Introduction
The rapid advancement of Large Language Models (LLMs) (Achiam et al., 2023; Anthropic, 2024; Team et al.,
'Department of Electrical and Computer Engincering, Uni- versity of Virginia, Charlottesville, VA, USA 'Department of Computer Science. University of Virginia, Charlottesville, VA. USA DDepartment of Computer Science, Florida State Univer- sity, Tallahassee, FL, USA. Correspondence to: Jundong Li <jl6gk@ virginia.edu.>.
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'hee code project available httpt::////// //tt:::///ttttttt com/ i nhanHe ModCire

---

Towards Global-level Mechanistic Interpretability; A Perspective of Modular Circuits of Large Language Models
nodes. For each node, the Fl provides textual explanations of its semantic role in completing the language task. To- gether, the nodes and their interactions form circuit that explains how an LLM performs a specific language task.
each reusable computational subgraph. Finally, we design novel prompting strategy generate FIs for the modular SCSs to obtain the MC vocabulary with an advanced LLM.
Our contribution can be concretely summarized as follows: (1) Novel formulation: we formulate the novel problem of MC vocabulary discovery for LLMs. The problem is built upon our designed comprehensive set of evaluation crite- ria that characterize the quality of an MC vocabulary. (2) Algorithm Design: we propose a principled MC vocabulary discovery framework that provides cross-task interpretabil- ity insights and alleviates the financial cost of circuit dis- covery. (3) Experimental Validation: we conduct extensive evaluations on an LLM across diverse language tasks from multiple domains, which both quantitatively and qualita- tively verify ModCire's effectiveness.
However, existing methods can only identify circuits on a task-by-task basis (i.e., analyzing how models handle each individual task), which presents two key limitations: (I) Lack of Generalization ability: Since the current cir- cuit discovery methods find circuit task by task, they fail to capture LLMs' broader computation patterns shared across different tasks. Hence, the insights gained from studying one task provide little information about how the model approaches other potentially related tasks. (2) High Cost: Current methods generate Fl for each node of the identified computational subgraph, which requires costly human in- spection (or most advanced LLMs such GPT-4 (Achiam et al., 2023)) to understand its role. This becomes increas- ingly impractical as LLMs rapidly scale in size.
2. Preliminaries
Here, we elaborate on how the significant computational subgraph (SCS) of a circuit is computed in existing literature, and the related terminology used throughout our work.
To address these limitations, we propose a novel modular circuits (MC) vocabulary, as inspired by Nanda (2023)'s hypothesis on the existence of circuit modularity. As shown in Fig. I, each MC in our vocabulary is a task-agnostic functional unit consisting of small computational sub- graph (referred to as "modular SCS") within an L.M and its associated FI. First, it provides global interpretability by enabling different tasks to share common MCs in their circuits. As illustrated in Fig. I(a), we identify & SCSs across i different tasks. These SCSs share a common mod- ular SCS (highlighted in sky-blue). whose corresponding FI offers cross-task insights. Second, the MC vocabulary re- duces the computational cost of circuit discovery, as shown in Fig. 1(b). When analyzing an unseen SCS for a new language task, we can match it against our existing MC vocabulary. This allows us to directly apply the cached Fls from the vocabulary to the matched portions of the SCS, eliminating the need for costly manual inspections.
In existing research, the SCS of circuit on task T, denoted as Cr, is identified through "activation patching" (Wang et al., 2023; Marks et al., 2024; Zhang & Nanda, 2023; Hanna et al., 2024a; Lieberum et al., 2023) across all com- putational nodes (abbreviated as "'noes",, which refer to LLM components such as attention heads and feed-forward neurons) of the examined LLM. For a node v, "activation patching" calculates the its causal indirect effect (IE):
IE(LLM, B) EADD LLM(dr |do(= = paatch)) - (LM(dd delean
Here, D is the input text dataset for task T (c.g.. in a disease diagnosis task, the input is a patient's 's symptom descrip- tion and the output is the corresponding disease name), and d = (delean dpatch) is a data sample from dataset D. Specifically, delean is the original input text, while daach is a corrupted version of the input, typically created by re- placing key words in gelean with random words. The term LLM (dd represents the logit value from the LLM's output corresponding to the ground-truth answer token (typ- ically multiple choice letter like "A" or "B"). Mcanwhile, LLM gdean [do(v = ypatch)) represents the logit value for the same answer token, but with a key difference: while the input remains delean the activation (i.c., output) at node v is replaced with patch (the activation that would have been produced by inputting dmatch). Nodes with "TE" val- ues higher than predefined threshold compose the nodes of Cr. Two nodes in CT are connected if there is direct computation flow between them. Causal tracing will serve as a useful part in our proposed method for finding MC vocabulary S = (Si,.., S } (l: the number of MCs).
In our work, we take the first step to find the MC vocab- ulary in an LLM. We first identify five evaluation criteria (consistency, locality, reusability, composability, and glob- ality) that characterize the properties an ideal MC vocabu- lary must possess. Based on these evaluation criteria, we subsequently propose a novel Modular Circuit vocabulary discovery method termed ModCire. Specifically, with a set of training tasks' SCSs of an LLM, our ModCirc first dis- covers the reusable computational subgraphs across those circuits according to the reusability criterion. For each of the reusable computational subgraph, we process it with a reinforcement learning-bast based graph neural partitioning approach to obtain the modular SCSs (each partition in each reusable computational subgraph is considered as a modular SCS). In this process, ModCire optimizes the con- sistency, locality, and composability of the partitioning for
2

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
3. Evaluation Criteria and Problem Definition
Sy needs to produce the desired outpur. However, since both the ground-truth output p and the inverse function 5 are not accessible, we instead evaluate composability using the expected mutual information between S1 (d) and S2(S) (d)):
The MC vocabulary is characterized by the following criteria derived from initial discussions of Nanda (2023).
The first criterion is Consistency. This criterion stipulates that an MC (denoted as S) should maintain functional con- sistency across different tasks. Denote the Fl of S when LLM performs task T E T (where T is a set of tasks in con- sideration) as (S, T), the consistency can be evaluated as the maximum number of synonymous Fls of the MC S across tasks T (denote synonymity by =)
Composability((S + 52) = (I( (d))
(5)
Here, the data input d complies with the distribution D.
The fifth criterion is Globality. The total number of unique MCs and the number used in any individual circuit must stay below certain thresholds N and M, respectively, ensur- ing manageable vocabulary size. With those criteria, we formally define the MC vocabulary Discovery problem:
M
Consist.7($) = {
MTT, E,,
FI(S,T) = Fb).
Definition 3.1. (MC vocabulary Discovery) Given (1) an LLM, (2) set of tasks T within task domain (e.g., tasks in medicine such as clinical diagnosis prediction). (3) dataset D for each task, find MC vocabulary that opti- mizes/satisfies all evaluation criteria.
(2)
The second criterion is Locality. This criterion mandates that nodes within MC exhibit both physical and semantic proximity. Physical locality means that nodes must be inter- nally connected and localized with respect to their physical positions within the LLM. Semantic locality indicates that nodes must serve closely related semantic functionalities.
4. ModCire: The Proposed Methodology
In the previous section, we have formulated the novel prob- lem of MC vocabulary discovery. Here, we present our proposed ModCire framework which effectives solves the MC vocabulary discovery problem.
To evaluate locality, we construct two types of embeddings for all nodes in the LLM, a semantic embedding measuring functional behavior similarities between MCs, and a physi- cal embedding measuring structural proximity between the MCs. For semantic embedding, we build a space where each node v E V maps to a vector, with distances d=em(.) (---) representing functional similarities (see Section 4). For physical embedding, we place nodes in a Fuclidean grid based on their proximity (Liu et al., 2023), where the distance duhuss,,) between nodes is their Manhattan dis- tance (Black, 2006). We define the locality for an MC S as Locality(S) = Lsem. + Lppys.. where
4.1. Model Overview and Design Motivation
In general, ModCire finds vocabulary of MCs with three steps, as illustrated in Fig. 2. In the first step (Section 4.2), we preprocess and unify all tasks into classification tasks, enabling us to identify SCS in the LLM using causal tracing techniques. In the second step (Section 4.3), we utilize a reinforcement learning (RL) based graph partitioning strat- egy to discover the modular SCSs from the SCSs of the given tasks through optimizing the MC vocabulary evalua- tion criteria. Finally (Section 4.4), we design LLM-based Fl prompting strategy to explain the function of each dis- covered modular SCS. Till then, we have acquired our MC vocabulary where each MC is a modular SCS with its FI.
En, Eninees baceesssldemm//m Iddem/ phyys (0), U2)
Lism/pphye
(3)
EveS, s Ideem//phee phyy (v, wI
The third criterion is Reusability. An MC is applicable across different tasks, enabling transfer of functionality, The reusability of an MC S on a set of tasks T can be measured by the proportion of the tasks that S appears in the LLM's circuits for those tasks, i.e.,
4.2. Task Format Unification and SCSs Collection
As in Definition 3.1, we are provided with an LLM and a set of tasks T and their corresponding datasets However, some tasks may not be suitable for circuit discov- ery because eircuit discovery requires the desired output for each input text to be only one token in length. This ensures that the logit difference on the first decoded token caused by patching the computational node (Wang et al., 2023; Goldowsky- -Dil et al., 2023) faithfully reveals the causal effect of the node. To solve the issue, we uniformly convert the formats of different tasks (c.g., medical diagnosis and medical documents summarization in the medical domain)
Reusability = 1 STE Cr). iT
1
(4)
The fourth criterion is Composability. When two MCs are connected, their combined operation should reflect a logical composition of their individual functions. For two MCs S₁ and S2 with computation flow S₁ + Sz. idcally we would verify that (d) = Sy (p) for any input d and the output given by ground-truth logical computation. This would ensure that S₁ provides exactly the information that
3

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
FI: this MC identifies symptoms... FI: consult medi, with symetoms MC vocabulary
SCSs acquisition
S
FI interpreter
P(A E
0.74
Consistency Composability
Locality
p () 0.26
Compute node features (one graph os e.g))
Opt. GNN w. Policy Grad.
Calculate reward
Re: ee-partition the node if not articulate
Compute partition probability
Balanced Initial partition
Select node examine
Articulate node detection
GNN aggregation
Reusable subgraphs
Figure 2. Overview of our proposed MC vocabulary discovery framework ModCire.
into multi-choice question answering. Specifically, for a data sample consisting an input text and desired output text, we compose two other output choices by prompting capable LLM (e.g., GPT-4) for similar but incorrect output.
(i.e., {uprid pleaan i patcc m- 1.
Ji= i.c.
LLM |do(v = uf patchh)) I.M reeann
V, LLM peppori (wgrid v.iid)) )
After task format unification, proceed to find the SCSs for all tasks on the examined LLM. Since we need to com- pute the SCSs for all the tasks on the LLM, performing activation patching can be very time consuming. Therefore, we adopt edge attribute patching (EAP) (Syed et al., 2023; Marks et al., 2024). EAP the linear approximation (i.e., first-order Taylor expansion) of the activation patching for- mula at the output of the interested computational node with the clean input. Specifically,
V LLM lempeend (u - pelean).
Finally, collect all nodes in the LLM that have the top ten values for their indirect effect in each transformer layer and connect them with all possible edges in the LLM com- putation graph as the discovered SCS for the examined task.
4.3. Modular SCSs Discovery with Neural Partitioning
With the SCSs of all tasks achieved, we proceed to discover the modular SCSs from them, illustrated in the bottom right of Fig. 2. Advised by the reusability property of MCs (see Section 2), we first conduct reusable subgraphs genera- tion to find the subgraphs that appear least two SCSs by conducting pair-wise intersection for each pair of the SCSs. This procedure guarantees that the modular SCSs reused across the tasks while ensuring the integrity of po- tential MCs (see details in Section 4.3.1). Then, we design an RL-based neural partitioning method to partition each of the reusable subgraphs to become modular SCSs ccording to our proposed evaluation criteria.
LLM dolean do(c ((oocch)) - LLM (L gllaan).
V,LLM pppoiean (npatch yclean).
This approximation significantly reduces the computation of finding circuits since, originally, we need to perform two LLM inferences for each node. Now, we only need two LLM inferences and one backward propagation to compute the indirect causal effect for all of the nodes.
However, the estimation of the indirect effect given by EAP degrade when pPatch is not close enough to clean (by definition of first-order Taylor expansion). To counter the estimation degradation, we utilize integrated gradient (IG) (Sundararajan et al., 2017; Hanna et al., 2024b; Marks et al., 2024), which instead estimates the indirect effect with first-order Taylor expansion all m equally spaced points
Due the complexity of the RL-based neural partitioning method, introduce it separately with three parts: (1) Node Feature Acquisition: we craft for each node a fea- ture vector as a preparation for the partitioning algorithm. (2) RL-based Neural Partitioning:
the general framework description for our designed RL-based neural partitioning method. (3) Objective Design: a detailed elaboration on

---

Towards Global-Level Mechanistic Interpretability; A Perspective of Modular Circuits of Large Language Models
how we design the RL rewards to optimize the locality, com- posability, and globality of the discovered modular SCSs.
Algorithm 1 Initial Partitioning of Reusable Subgraphs
Input : :g CCC,,G,....C J: Set of reusable subgraphs; G:
the reusable subgraph to be partitioned:
1. REUSABLE SUBGRAPH GENERATION
Output : P: Initial partitioning for each G E G;
Gent < Flnddaterseetionraaaae (9);
In Section 2, MCs must maintain reusability, meaning that each modular SCS should be able to appear in SCSs of multiple tasks. While frequent subgraph mining methods, like gSpan (Yan & Han, 2002), could identify subgraphs that appear across multiple circuits, these methods present significant limitation: they identify frequent subgraphs of varying sizes, with smaller subgraphs typically appearing more frequently (the smallest being just two nodes con- nected by one edge). These small frequent subgraphs risk fragmenting the modualr SCSs, as they may represent only partial components rather than complete modular SCSs.
C P + www.sisssaaaattaaa. (G. C);
(Gou):
U + \
// Unassi gned nodes in G
while U * do
foreach partition p E P do
t f Random (p)
// Random node from
partition
No <- neighbors of v in U
if N. + 0 then
u u Random (NV.) + U {u} // Assign u
to partition of v
else if Size < max partition size in N.'s partitions then
u f neighbor with largest partition; to v' a partition
// Assi gn "
To preserve the integrity of modular SCSs, we instead unify all the circuits and preserve any overlapping components that have more than one node. This approach finds the reusable subgraph between circuits. ensuring that potential modular SCSs remain intact. However, this procedure leads to situation that a reusable subgraph may contain multiple modular SCSs. Therefore, we follow with a partitioning procedure to differentiate individual modular SCs.
end
end
end
circumvent this non-differentiabiliyy issue by policy gradi- ents (Kakade, 2001; Shah et al., 2024), an RL techmique, which allows us to optimize the GNN as policy network by sampling node partitioning actions and their corresponding rewards. Next, we specifically explain how our RL-based neural partitioning applies to reusable subgraph.
4.3.2. NODE FEATURE ACQUISITION
We construct node features for reusable subgraphs by com- bining semantic and physical features. For semantic fea- tures, we collect each node's activation patterns across all data samples and tasks. Since different node types (like attention heads) have different dimensions, we use separate MLPs to project their activations into a common dimen- sional space. The semantic feature for each node is the mean vector of these processed activations across all sam- ples and tasks. For physical features, we use Euclidean grid embedding, mapping each node to coordinates based on its transformer layer and position (attention head/feed forward neuron index). The final node feature concatenates both embeddings, capturing both functional behavior and structural position to optimize modular SCS localities.
For any reusable subgraph, we first generate an initial par- titioning, as illustrated in Algorithm 1, that ensures the following properties: (1) internal connectivity within each partition (supporting physical locality in our proposed eval- uation criteria); (2) balanced partition sizes across all par- titions due to the initial partitioning serving as the starting point for our RL-based neural partitioning approach; and (3) partitioning compatible across all reusable subgraphs, meaning that any nodes in one partition of a reusable sub- graph, if exist in another reusable subgraph, should not be separated into different partitions in that other subgraph.
To do this, we first guarantee the partitioning compatibil- ity requirement by finding the intersection graph of all the reusable subgraphs and assigning each of its weakly con- nected components as partition. Then, we expand those partitions iteratively in the reusable subgraph. Specifically, in each iteration, we randomly select a node from each cur- rent partitions and visit one of its neighbors, adding the neighbor to the same partition of the source node (for in- ternal connectivity). If the neighbors of the source node are all assigned and the partition of the source node has a smaller size than the largest partition of its neighbor nodes, we reassign the neighbor that has the largest partition to the partition of the source node which allows for balanced partitioning. We repeat the iteration until all nodes in the
4.3.3. RL-BASED NEURAL PARTITIONING
In the Section 4.3.2, we equip each reusable subgraphs with node features. Given the reusable subgraphs and their node features, our motivation is to train a graph neural network (GNN) to partition the reusable subgraphs, whose output logits for each node represent the probability of the node be- longing to various partitions. Furthermore, the GNN should be trained with loss function that optimizes our proposed evaluation criteria. However, since the optimization space (i.e., all possible partitionings) is discrete, the loss functions are non-differentiable. This makes it impossible to train the GNN in traditional supervised manner. Fortunately, we may
5

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
reusable subgraph are assigned a partition.
I feel headache, now take 1 pill of) ibuprofen every day, I suspect that it
Modular circuit
After the initial partitioning, inspired by Shah et al. (2024), we train the GNN partitioner for each reusable subgraph upon its initial partitioning with policy gradient. As policy gradient methods operate within a Markov Decision Process (MDP) (Puterman, 1990), we will now formulate the specific MDP that our policy gradient operates within. In our MDP, decisions unfold through a sequence of states ("s"), actions ("a") and rewards ("r") known repartition trajectory ("r"), we aim to optimize the GNN as policy network to maximize the expected reward of repartition trajectories.
caused by the food I ate yesterday.
have a fever since 2:00 am and
took Tylenol 8 in the morning. was reading book, suddenly feel intense stomachache.
Activations
FI: this MC
feel headache, now take 1 pill of
is primarily ibuprofen every day, I suspect that it is focusing on:
caused by the food I ate yesterday.
identifying
2. I have a fever since 2:00 om and
the patients'
took Tylenol at in the morning,
symptoms.
3. I was reading book, suddenly I feel intense stomachache.
In our formulation, a repartition trajectory is a sequence of repartitioning processes, where each repartition process refers to one following procedure applied on the reusable subgraph: (1) From the currently partitioned reusable sub- graph (which corresponds to state "s"), sample candidate repartition node within current partition boundaries, where each node's 's selection probability is proportional to the frac- tion of heterogeneous partitions in its neighborhood; (2) De- tect if the node is articulation node (Tian et al., 2017) (i.c., the node by removing which separate the nodes within a partition to isolated connected components); (3) If the node is not an articulation node, repartition the selected node by current GNN partitioner (which corresponds to action "a")). The reward "r" (Shah et al., 2024) of the repartitioning is
Figure 3. Fl generation for the MCs.
Equation 8). Here, we design our modular SCS partitioning objective with the evaluation criteria specified in Section 3.
Here, we incorporate the evaluation criteria into the objec- tive function by: (1) Consistency: as quantifying semantic consistency requires the LLM prompting efforts which is rather financially and time intensive, we do not incorporate this metric into the training stage; (2) Locality: we calculate the physical and semantic locality exactly by the Equ. 3, where the physical locality computation only considers the node's layer index in the model. (3) Reusability: this metric was previously addressed in Section 4.3.1 during the gener- ation of reusable computational subgraphs. We exclude it from the current objective function as a compromise in our optimization approach, accepting lower reusability thresh- old for modular SCSs. (4) Composability: we compute the Equ. 5 with KDE (Rosenblatt, 1956) estimation for each pair of adjacent modular SCSs and take their average value as an evaluation of the composability of the partitioning. (5) Globality: as described Section 4.3.3, we set a maximum threshold for partition numbers and a minimum size require- ment relative their containing circuits. These constraints, combined with our selection of only the top N modular SCSs based on their metric values, ensure globality.
Obj(G, prefore) Obj poster) r(a) Obj(G, phofore) + Obj (G, Pofter))
where Obj(G, Obe fore and (b) pafter) are the objec- tive functions designed with our proposed MC vocabulary evaluation criteria, which will be specified in Section 4.3.4.
With each repartitioning process's 's reward specified, the com- prehensive reward of the repartition trajectory is hence the discounted summation (by the discount factor 0 < < 1) across the horizon of length L > 0
R(T) = r(ao) + yr(1)) + ............................... + r(a--1).
(9)
Therefore, denote our GNN partitioner (8) where 0 is its parameters, our optimization target can be written as
4.4. Functional Interpretation (FI) Generation
To complete MCs discovery, we perform Fl generation as illustrated in Fig. 3 for each modular SCSs by utilizing the tfate-of-the-an LLM such as GPT-4 (Achiam et al., 2023) (referred in the following as "FI generator"), along with an integrated dataset which is the combination of all training data. For each data item in the integrated dataset, we process it as input to the examined LLM and extract its activation for each modular SCS. We calculate the token-wiis læ norm of the activation, which creates teken-level heatmap for the data item. Then, we feed these heatmaps to the Fl generator to determine the function of certain modular SCS. Notably, we may expand the text dataset for FI generation to any text datasets within the examined task domain.
( Fe, (R(T)).
(10)
Here, we adopt Markov Chain Monte Carlo (MCMC) (Hast- ings, 1970) estimate the REINFORCE gradient (Williams, 1992) of Equation 10 for optimization of the GNN.
4.3.4. OвJECTIVE DESIGN
According to Section 4.3.3, we view the GNN partitioner as a policy network and optimize it with policy gradient. In the process, one significant part that informs the optimiza- tion direction is the objective in the reward function (i.e.,
6

---

Towards Global-level Mechanistic Interpretability; A Perspective of Modular Circuits of Large Language Models
5. Experimental Evaluations
MedAltr
MedSlalus
Corsference
PubMedSumm
0.40
1.1
In this section, we discover the MC vocabulary of Med- LLaMA (8B) (Labs, 2024), an LLM finetuned on LLaMA (8B) with medical domain context, when it performs medi- cal tasks by our ModCire. Specifically, we investigate RQI: How well can ModCire find the MCs compared with other baselines? RQ2: To what extent each component of Mod- Cire contributes to the overall performance? How will the choice of the hyperparameters affect the performance of ModCirc? RQ3: How does an MC involve different task circuits with consistent functiooiigg? RQ4: How well can FIs of MCs transfer to new tasks? Please refer to the Ap- pendix for supplementary results of parameter analysis and MC examples.
Reusebiiity
0.35
Composability
0.0
0.30
0.8
0.25
0.20
10
15
0.6
100 Node Feature Dimension (b)
Top for Discovering Circuits (k)
(e)
Figure 4. Parameter analysis.
Circ demonstrates notably superior reusability scores (0.32- 0.37) across all datasets compared to baselines whose scores are below 0.21. While ModCire maintains competitive com- posability scores between 0. 0.70-and 1.00, it does not consis- tently outperform other approaches like Clust. and Random, which sometimes achieve perfect scores. In terms of con- sistency, ModCirc performs comparably to other methods, with scores ranging from 0.26-0.40, indicating reliable but not necessarily dominant performance in this aspect. The slightly lower consistency and composability scores can be attributed to ModCirc's 's requirement maintain connected partitions.
5.1. Experimental Settings
Datasets For training, we use "MEDAL" (Wen et al., 2020) for multi-class medical abstract classification, "Medical Ab- stracts" (Schopf et al., 2023) for patient condition classifi- cation from abstracts, "MedMCQA" (Pal et al., 2022) for medical multiple-choice question answering, and "Symp- tom to Diagmosis" (Gretel.ai, 2022) for mapping symptom diagnoses. For evaluation, we employ four datasets with distinet tasks: "MedStatus" identifies medication status in clinical text, "MedAttr" recognizes medical entities, "Coref- erence" "aaiiee"AAggwaaaa (Agrawal et al., 2022; Moon et al., 2014) resolves clinical coreference relations, and "PubMed Summ. "(Cohan ct al., 2018) evaluates biomedical abstract summarization.
We explore the effectiveness of different components in ModCire through three variants. "ModCire-NCire" ran- domly selects circuits from the LLM's computational graphs. "ModCire-NInis skips the initial partitioning step and di- rectly applies reinforcement learning-base based reusable sub- graph partitioning. "ModCire-NNeuro" only performs ini- tial partitioning, omitting the reinforcement lcarning-based reusable subgraph partitioning step. In Table 2, we show the performance of our ModCirc and the three variants on the Medi Attr dataset. We can observe that our ModCire pos- sesses a consistent performance gain compared to all three variants w.r.t. consistency. Composability decreases, prob- ably due the additional condition of finding connected MCs. We have similar observations in other datasets.
Baselines Our baselines include (1) Random. We randomly select nodes from the circuits of each dataset and find their ego graphs with radius of three. (2) Frequent Random. We propose variant of the Random baseline by first finding the reusable subgraphs as in ModCire. Then, we perform Random on the reusable subgraphs to obtain frequent ran- dom MCs. Kmeans. Similarly, we first find the reusable subgraphs and then perform KMeans clustering to partition the subgraphs into MCs. (4) Activation. We assume that if two computational nodes are in an MC, then they would have high activation values on the same inputs. Thus, we compute the top 10 activation values for each computational node across all data samples. Then, we group nodes together if their activated samples intersect and find the connected components within the node groups.
Additionally, we examine the parameter sensitivity of our ModCirc by varying two most significant hyperparameters. Specifically, we adopt "topk (k) nodes", which is the num- ber of selected nodes in each transformer layer for SCS construction (see Section 4.2); and "node feature dimension" referring to the node feature dimensions (see Section 4.3.2). In Fig. 4 (a) shows that increasing the k leads to improved reusability (from 0.20 to 0.40) across all datasets, with Me- dAttr and PubMedSumm most notable. Fig. 4 (b) shows that composability first increases then decreases with the in- crease of node feature dimension (peak at 64). Other criteria are stable by varying these hyperparameters.
5.2. Quantitative Analysis
In this section, we aim to answer RQI and RQ2. In Table I, we measure the composability, consistency, and reusability of the MCs identified by ModCire and the baselines on four evaluation datasets. We observe that ModCire, in general, achieves the best performance, especially in the reusability criterion, proving the effectiveness of our framework. Mod-
7

---

Towards Global-Level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
Table Effectivvnness of ModCire and baselines in finding MCs. The optimal results in bold, and the runner-up are underlined.
Datasets
Metrics
Act.
Clust.
Freq-
Random
ModCire
Composability
0.75 0.00
0.80 = 0.10
0.65 0.12
0.75 + 0.00
0.70
Coreference
Consistency
0.35
0.39 = 0.02
0.39 +
0
0.40 0.02
Reusability
0.06 0.00
0.13 = 0.01
0.21 = 0.01
0.11 + 0.01
0.37 = 0.01
Composability
1,00 0.00
1.00 = 0.00
1.00 =
1,00 + 0.00
0.100
MediAttr.
Consistency
0.01
0.34 0.01
0.02
0.01
Reusability
0.06
0.11 = 0.01
0.18
0.10
0.34 0.01
Composability
1.00
1.00 = 0.00
0,00
1,00
1.00
MediStatus
Consistency
0.04
0.26 = 0.00
0.20
0.26
0.26 0.01
Reusability
0.01
0.10 0.01
0.11
0.09
0.33 0.00
Composability
0.00
0.75 0.00
0.05
0.70
0.70
PubSumm.
Consistency
0.02
0.33 = 0.02
0.34 0.01
0.33 0.02
0.34 0.03
Reusability
0.05 = 0.00
0.10 0.01
0.18 0.01
0.10
0.32 0.01
data" supports determining medication statuses, while in coreference resolution, its specialization enables accurate tracking of medical terms across text. The absence of this MC in PubMed Summ is reasonable, as this task focuses more on high-level summarization rather than specific entity identification. This semantic alignment between the MC's Fl and its usage across tasks validates interpretation.
Table Performance comparison of ModCirc variants specified in Section 5.2 on MediAutr. "Compo.", "Const." and "Reusa." represent composability, consistency, and reusability, respectively. Const.
Method
Compo.
Reusa.
ModCire
0.95 # 0.10
0.33 # 0.01
0.34 # 0.01
ModCire-NCire
1.00 = 0.00
0.30 0.01
0.20 = 0.00
ModCire-Nine
1.00 = 0.00
0
0.34 = 0.01
ModCire-NNeurg
0.95 = 0.10
0.31 = 0.01
0.34 = 0.01
5.4. MC Transferability and Composability
In this section, we address RQ4. We test ModCire's trans- ferability and composability by training ModCire GPT-
A2B
A22
2 Small (Radford al., 2019) with four training NLP tasks: AGNews (Zhang et al., 2015), MPQA (Wiebe et al., 2005), Universal Dependency (Silveira et al., 2014), and TREC (Voorhees & Tice, 2000). We then generate MCs for two new unseen tasks whose ground-truth circuits (in- cluding Fls) are established existing work: indirect object identification (IOI) (Wang et al., 2022) and acronym detec- tion
A19
C431
A24
D8
A22
A25
Figure 5. An example MC spanning two layers. Green squares show self-attention components (A: attention heads, O: self- projection neurons) while blue triangles are feed-forwart networks (I: MLP input neurons, D: MLP down-pron pro-jproecction neurons).
et al., 2024).
The IOI task aims to predict the indirect object (IO) in sentence where the subject (S) appears twice. For example, in the sentence "Mary (IO) and Amy (S) met, and Amy (S) handed pen to -," the blank represents the position where an indirect object should be identified based on the sentence context. The acronym detection task Cearcia-Carrasee et al., 2024) predicts the acronym of three- word phrase (e.g., the acronym of "Limited Liability Company" is L.CC).
5.3. Qualitative Analysis
In this section, we demonstrate the quality of our MC vo- cabulary with a case study to answer RQ3. In Fig. 5, we illustrate representative MC, which provides compelling evidence for the effectiveness and semantic consistency of the identified MC. This MC is highly reusable, since it ap- pears in SCSs of three out of four evaluation tasks (MedAttr, MedStatus, and Coreference), strongly suggesting its funda- mental role in medical language understanding.
For the IOI task, ModCirc recovers 92% (23/25) of the ground-truth circuits' components, and Fig. 6 shows the analyzed MCs. We observe from Fig. 6 that the FIs of the MCs generated by ModCire correspond to the FIs of the ground-truth circuit (Wang et al., 2022), indicating the transferability of ModCire. Specifically, MC O identifies two Induction Heads, whose Fl is to highlight prior occur- ring subjects (Wang et al., 2022), complementing the MC's Fl of "identifying specific entities," such as nouns. MC 1 contains two Name Mover (NM) Heads that "attend to pre- vious names" and one S-Inibbiti (SI) Head that "removes duplicate names identified by the NM Heads" (Wang et al.,
The MC's FI, which focuses on "identifying and classifying biological entities and interactions, particularly in health and disease," demonstrates remarkable alignment with the semantic requirements of these three tasks. For medication attribute identification, the MC's ability to recognize bio- logical entities serves the task of identifying medications and their characteristics. For medication status classifica- tion, the MC's capability to analyze "complex biological
8

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
MC
6. Related Work
Backup nome mover head
L5H5
L11H9
6.1. Mechanistic Interpretability
Name mover head
L7H9
S-inhibition head
Mechanistic interpretability (MI) reverse-engineers com- ponents of machine learning models into comprehensible concepts, allowing for a deeper understanding of model be- havior (Bereska & Gavves, 2024; Rai et al., 2024; Singh et al., 2024). Recent works in MI concentrate on identi- fying end-to-end circuits, pathways that trace the flow of information from the inputs to the outputs (Conmy et al., 2023; Wang et al., 2022; Hanna et al., 2024a). However, these circuits are task-specific, raising questions about their applicability to other tasks. Some studies attempt to ad- dress this limitation by investigating the generalizability of existing circuits. For instance, researchers have tested the performance of the indirect object identification (IOI) circuit on variations of the original task (Nainani et al., 2024) and examined its reusability in a different context (Merullo et al., 2023). Building on these efforts, this work aims to compile an MC vocabulary that can be reused across various tasks.
LSHB
L11H10
Induction head
identify& & extract specific type of entities"
Uncolored to avoid distraction
MC1
MC 8
L10H1
L9H6
L8H6
L9H9
L10H10
L11H2
L10HO
L9HO
"extracting and assessing contextual relevance"
"creates pointers between entities" name moving
Figure 6. MCs identified by ModCire for IOI. "L" and "H" repre- sent the layer and head indices, respectively.
MCO
MC1 L11H4
Letter mover head
L5H8
L9H9
"extracting specific wentities entities.from.teet trem_
"extracting contextual relevonce""
Information movement head
6.2. LLM Modularity
LLM modularity refers to the integration of multiple com- ponents, each specializing in a specific task, to construct LLM with diverse functionalities (Munikoti et al., 2024; Shao et al., 2024; Kaddour et al., 2023). One approach emphasizes model-centric perspective, designing compo- nents to specialize in specific tasks (Xiao et al., 2024; Feng et al., 2024; Ostapenko et al., 2024; Muqeeth et al., 2024; Kang et al., 2024). Another approach focuses on the train- ing process, encouraging the development of modularity during training (Liu al., 2023; Nainani, 2024). However, these methods incur high training costs. posing a signifi- cant challenge for LLMs. Our work addresses this issue by examining pre-trained models for modularity and applying causal mediation analysis to uncover modular circuits.
Figure 7. MCs identified by ModCire for acronym detection.
This aligns with the MC's FI of "extracting and assessing contextual relevance," where NM Heads extract names, and SI Heads assess and remove duplicates. MC 8 contains one NM Head and three Backup Name Mover (BNM) Heads. Unlike MC 1, MC 8's Fl is to "create as- sociations or pointers between entities and their attributes," which matches the NM Head's role to associate and output the indirect object. Furthermore, the FIs of the MCs exhibit composability: MC 0 detects duplicate tokens, MC 1 as- sesses and removes duplication, and MC 8 outputs answers, matching the existing ground-truth circuit structure.
2022) 2
For the acronym detection task, ModCire recovers 87.5% ground-truth circuit nodes (7/8), and the analyzed MCs are shown in Fig. 7. We observe from Fig. 7 that ModCire produces FIs that align with the ground-truth FIs (Gareía- Carrasco et al., 2024): (1) MC 0 captures Head L5H8, and its FI "extracting entities" agrees with the head's ' FI "find capital letter"; (2) MC 1 contains letter mover (LM) heads 1,999 and 1,4,4, aligning the MC's FI "assessing context" with head's ground-truth FI "copy capital". The "assess" " here implies the copying procedure. Additionally, MC 0 identifies the capital letter position, and MC 1 copies the capital letter to answer position, composing the ground-truth circuit's 's FI CCarcia-Carrasee et al., 2024).
7. Conclusion
We take initial step towards global-level mechanistic interpretability (MI) which address the task-specific and high computational costs limitations of existing MI meth- ods. First, we formulate a novel problem of discovering a modular circuit (MC) vocabulary. Then, we propose Mod- Circ, a principled method for MC vocabulary discovery that addresses both limitations. Finally, we examine ModCire an L.M and successfully discovered its MC vocabulary with extensive quantitative and qualitative evidence.
Impact Statement
Name Mover (NM) Heads, S-Inhibition (SI) Heads and the Backup Name Mover (BNM) Heads (referred in the following texts) are defined and identified by existing work.
This work on modular circuit discovery represents a pos- itive direction for Al research, as aims to enhance our
9

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
understanding of how language models process information through purely observational analysis of existing systems. By improving model interpretability and efficiency through identifying reusable components, this research could lead to more reliable and resource-efficient Al systems while reducing computational costs. The approach enables better auditing and validation of behavior through enhanced transparency, which aligns with broader goals of developing trustworthy AI technology that benefit society. Since the research focuses analyzing already-trained models rather than developing new capabilities, it contributes con- structively to scientific knowledge while maintaining re- sponsible approach to Al advancement.
Acknowledgement
This work is supported in part by the National Science Foundation (NSF) under grants IIS-2006844, IIS-2144209, IIS- IIS-2223769, CNS-21
BCS-2228534 and CMMI- 2411 the Office of Naval Research (ONR) under grant N000142412636; and the Commonwealth Cyber Initiative (CCT) under grant VV-1Q24-011.
10

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
References
Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching. arXiv, 2023.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv, 2023.
Gretelaii Symptom diagnosis dataset, 2022. URL https://hugginggtn co/ datasets/ pretelai/sympoe to diagnosis. Dataset available on Hugging Face.
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., and Son- tag. D. Large language models are few-shot clinical information extractors. In EMNLP, 2022.
Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abili- ties in pre-trained language model. NeurIPS, 2024a.
Ansel, J. and Yang, Edward, a. PyTorch Faster Machine Learning Through Dynamic Python Bytecode Transfor- mation and Graph Compilation. In ASPLOS, 2024.
Hanna, M., Pezzelle, S., and Belinkov, Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv, 2024b.
Anthropic. Claude 3 model card, 2024.
URL
httpp::/wwwwwee anthroric. com/ httpps//wwwecccc..aaaaaoaeeecoor de@ba daebbe9ho.ccaa37ccabbe33333333
Model_Card_Claudee pdf.
Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. 1970.
Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety-a review. arXiv, 2024.
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileamu, R., and McHardy, R. Challenges and applications of large language models. arXiv preprint arXiv:2307.10110 2023.
Black, E. Manhattan "istance"" dictionary of algorithms and data structures. http://slinxx. nist. gov/dads// 2006.
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang. L., Zhu, K.,
Kakade, M. A natural policy gradient. NeurIPS, 2001.
Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology. 2024.
Kang, J., Karlinsky, L., Luo, H., Wang, Z., Hansen, J., Glass, J., Cox, D., Panda, R., Feris, R., and Ritter, A. Self-moe: Towards compositional large language models with self- specialized experts. arXiv preprint arXiv:2406.12035 2024.
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. In NAACL (Short Papers), 2018.
Labs, J. S. Isl-medllama---8b--2200 2024. URL https://hugiingee JSI.-Med11 awaeedllaa---e.. 3- 8B-v2.0.
co/ johnsnowlabs/
Conmy, A., Mavor-Parker, A., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. NeurIPS, 36, 2023.
Lieberum. T., Rahtz, M., Kramár, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V. Does circuit analysis inter- pretability scale? evidence from multiple choice capabili- ties chinchilla. arXiv, 2023.
Cunningham, H., Ewart, A., Riggs, L., Huben. R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. arXiv,, 2023.
Liu, Z., Gan, E., and Tegmark, M. Seeing is believing: brain-inspiree modular training for mechanistic inter- pretability. Entropy, 2023.
Feng, S., Sorensen, T., Liu, Y., Fisher, J., Park, C. Y., Choi, Y., and Tsvetkov, Y. Modular pluralism: Pluralistic alignment via multi-llm collaboration. arXiv preprint 2xXiv22406.15951, 2024.
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv, 2024.
Fey, M. and Lenssen, J. E. Fast Graph Repre- sentation Learning with PyTorch Geometric, May 2019. URL https://t// /ttp://gtttt/////////t com wwt.coooo/pyaanneeeeeeeeee wytorch_geometee
Merullo, J., Eickhoff, C., and Pavlick, E. Circuit component reuse across tasks transformer language models. arXiy preprint arXiv:2310.08744, 2023.
J., Maté, A., and Trujillo, J. C. How does gpt-2 predict acronyms? extracting and understanding circuit via mechanistic interpretability. In International Conference on Artificial Intelligence and Statistics, pp. 3322-3330. PMLR, 2024.
Moon, S., Pakhomov, S., Liu, N., Ryan, J. O., and Melton, G. B. A sense inventory for clinical abbreviations and acronyms created using clinical notes and medical dictio- nary resources. JAMIA, 2014.

---

Towards Global-Level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
Munikoti, S., Stewart, I., Horawalavithana, S., Kvinge, H., Emerson, T., Thompson, S. E.,, and Pazdernik, K. Gener- alist multimodal ai: A review of architectures, challenges and opportunities. arXiv preprint arXiv:2406.0599, 2024.
Shah, R., Jain, K., Manchanda, S., Medya, S., and Ranu, S. Neurocut: A neural approach for robust graph partition- ing. In SIGKDD, 2024.
Shao, M., Basit, A., Karri, R., and Shafique, M. Survey of different large language model architectures: Trends, benchmarks, and challenges. IEEE Access, 2024.
Mugeeth, M., Liu, H., Liu, Y., and Raffel, C. Learning to route among specialized experts for zero-shot generaliza- tion. arXiv preprint arXhv:2402.0585,, 2024.
Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., and Manning, C. D. A gold stan- dard dependency corpus for English. In LREC, 2014.
Nainani, J. Evaluating brain-inspired modular training in au- tomated circuit discovery for mechanistic interpretability. arXiv preprint ::XY:22401.03646, 2024.
Singh, C., Inala, J. P., Galley, M., Caruana, R., and Gao, J. Rethinking interpretability in the era of large language models. arXiv preprint NxNv:220220016.., 2024.
2024.
Nainani, J., Vaidyanathan, S., Yeung, A., Gupta, K., and Jensen, D. Adaptive circuit behavior and generalization in mechanistic interpretability. arXiv, 2024.
Sundararajan, M.. Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In ICML, 2017.
Nanda, N. Actually, othello-gpt has a linear emergent world model, 2023. URL <https://neet /neen//nda io aepechanistic--nteeeeeeeaaae othello>.
Syed, A., Rager, C.,, and Conmy, A. Attribution patching outperforms automated circuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv, 2023.
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, Progress measures for grokking via mechanistic inter- pretability. In The Eleventh International Conference on Learning Representations, 2023.
Tian, L., Bashan, A., Shi, D.-N., and Liu, Y.-Y. Articulation points complex networks. Nature communications, 2017.
OpenAl. Gpt-4o mini, 2025. URL https://oppent com. Large language model.
Ostapenko, O., Su, Z., Ponti, E. M., Charlin, L.., Roux, N. L.. Percira, M., Caccia, L., and Sordoni, A. Towards modular iims by building and reusing library of loras. arXiv preprint ar 2rXiv:2055.11333 2024.
Voorhees, E. M. and Tice, D. M. Building a question an- swering test collection. In SIGIR, 2000.
Wang. K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, Interpretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint 1rXiv:22110005333 2022.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medm- cqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In CHIL. 2022.
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. In ICLR, 2023.
Puterman, M. L. Markov decision processes. Handbooks in
operations research and management science, 1990.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et Language models are unsupervised multitask learners. OpenAl blog, 2019.
Wei, X. and Liu, L. Are large language models good in- context learners for financial sentiment analysis? arXiv, 2025.
Rai, D., Zhou, Y., Feng, S., Saparov, A., and Yao, Z. A practical review of mechanistic interpretability for transformer-based language models. arXiv, 2024.
Wen, Z.. Lu, X. H., and Reddy, S. MeDAL: Medical ab- breviation disambiguation dataset for natural language understanding pretraining. In Proceedings of the 3rd Clinical Natural Language Processing Workshop. 2020.
Rosenblatt, M. Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statis- tics, 1956.
Wiebe, J., Wilson, T., and Cardie, C. Annotating expressions of opinions and emotions in language. LREC, 2005.
Schopf, T., Braun, D., and Matthes, F. Evaluating unsuper- vised text classification: Zero-shot and mmilarity-baseed approaches. NLPIR, 2023.
Williams, R. J. Simple statistical gradient- algo- rithms for connectionist reinforcement learning. Machine learning, 1992.
12

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
Wolf, Tr, Debut, L., Sanh, V., Chaumond, J., De- langue, C., Moi, A., Cistac, P. Ma, C., Jernite, Y,, Plu, J. Xu, C., Le Scao, T. Gugger, S., Drame, M.. Lhoest, Q.. and Rush, A. M. Transformers: State-of-the-Art Natural Language Processing. pp- 38-45. Association for Computational Linguistics, Oc- tober 2020. URL https:/ /www. aclweb.org/ anthology /0nggy/2222 2000 .e020.emmilln ww..pp-demos......
Xiao, C., Zhang, Z., Song, C., Jiang, D., Yao, F., Han, X., Wang. X., Wang, S., Huang, Y., Lin, G., al. Config- urable foundation models: Building Ilms from a modular perspective. arXiv preprint arXiv:2409.02877 2024.
Yan, X. and Han, J. gspan: Graph-based substructure pattern mining. In ICDM, 2002.
Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods. In ICLR, 2023.
Zhang, X., Zhao, J., and LeCun, Character-level convo- lutional networks for text classification. NeurlPS, 2015.
13

---

Towards Global-Level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
A. Limitation
The proposed ModCire framework, while demonstrating promising results in discovering modular circuits within large language models, has several important limitations that should be acknowledged: First, our current implementation relies heavily on the quality and diversity of training task circuits. The discovered modular circuits (MCs) are inherently limited by the scope and variety of the training tasks used. If the training tasks do not adequately cover certain functional aspects of the model, the resulting MC vocabulary may miss important modular components that could be present in the model but not activated by our chosen tasks. Second, the evaluation of consistency and reusability across tasks remains challenging. The current metrics may not fully capture the nuanced ways in which circuits might be reused or adapted across different contexts, potentially leading to an oversimplified understanding the model's modularity.
B. Implementation Details
ModCire is implemented in PyTorch (Ansel & Yang, 2024) using the Transformers package (Wolf et al., 2020) to access MedLlama and the Py PyTorch Geometric package (Fey & Lenssen, 2019) for constructing the GNN model. We also used the OpenAl API to access GPT-4o-Mini (OpenAI, 2025) for the functional interpretation and evaluation phase. In our experiments, the learning rate is 0.001, and the training epoch number is 10. Experiments are carried out on an Nvidia RTX A6000, and the reported numerical results are averaged across five different runs.
C. Supplementary Experiment Results
C.I. Parameter Analysis
In this subsection, present additional parameter analysis of our proposed ModCirc. We observe from Fig. 8 that there exist four steady relationships: composability and reusability are steady with the increase of top k discovering circuits and node feature dimensions, respectively. Besides, consistency remain intact with the increase of both the top k discovering circuits and node feature dimensions.
MedAttr
MedStatus
Coreference
PubMed Summ.
0.95
0.45
0.90
Composablity
0.85
Cousistency
0.40
0.80
0.35
0.30
0.70
0.65
0.60
0.20
10
10
Top Discovering
Top Discovering Circuits
(a)
(b)
0.375
0.45
0.350
Consistancy
0.40
Reusabiltt
0.325
0.300
0.275
0.30
0.250
0.225
0.20
0.200
Node Feature Dimension
Node Feuture Dimension
(c)
(d)
Figure Additional parameter analysis of our proposed ModCire.
14

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
C.2. Example Modular Circuits
We show example modular circuits in Fig. 9, For complete of the modular circuits, see https://github. com Htt./iinhaanellll 23/ModCirc.
4_proj
k. proj
X_pooj
proj
gate_pee pote_proj
up proo
proj
imput
ourput
Modular Circuit This MC specializes infectious disease classification from symptom patterns
ALpro
proj
Y_proj
o_proj
_poj
up_proj
down_pro)
imput
output
Modular Circuit This MC focuses on respiratory and neurological condition diagnosis
Figure 9. Modular Circuit 1. This MC specializes in infectious disease classification from symptom patterns
15

---

Towards Global-level Mechanistic Interpretability: A Perspective of Modular Circuits of Large Language Models
C.3. Functional Interpretations of the Modular Circuits
We show two example modular circuits' functional interpretations as in Table 3. For complete set of the functional interpretations, https: htttt:///itthii http:/tttttt..tt/oo....ooooccttt (ccomm//nnaaneeee///oooooo
Table 3. Descriptions of Modular Circuits and Their Functions
Circuit Description
0
The main function of modular circuit O appears to be the classification and differentiation of medical conditions based on presented symptoms and clinical information. High importance scores are associated with texts that describe specific symptoms (e-g., runny nose, fever, itching) and medical conditions (e.g.. common cold, malaria, renovascular hypertension). The circuit likely focuses identifying patterns symptomatology and diagnostic tests accurately categorize various health issues, particularly in the context of infectious diseases and conditions related to organ function.
1
The main function of modular circuit 1 appears to be the classification and diagnosis of medical conditions based on symptom descriptions and clinical contexts. The high importance scores are associated with texts that detail specific symptoms (e.g., cough, pain, stiffness) and treatments (e.g.. chemotherapy regimens), indicating that the circuit is designed to interpret and categorize health-related information effectively. focuses on identifying patterns symptoms and their potential links specific diseases or conditions, particularly in respiratory, neurological, and musculoskeletal contexts.
C.4. Ablation Study
Here, we demonstrate the extended ablation results on additional datasets in Table 4. These results show that ModCire consistently outperforms (up to 5%) its ablation variant without RL-based subgraph partitioning. Note that "Reusability" is excluded from this comparison since it is primarily influenced by the initial partitioning rather than the RL-based subgraph partitioning.
Table 4. Comparison of ablation and original results across datasets
Dataset
Metric
Ablation Results
Original Results
Composition
0.65 = 0.12
0.70 0.10
Coreference
Consistency
0.37 = 0.01
0.40 : 0.02
Composition
0.65 = 0.12
0.70 # 0.
PubmedSumm
Consistency
0.34 # 0.01
0.34 # 0,03
16


Reviews:
[{"review_id": "QzKX1kKRi5", "reviewer": "Reviewer_aN6A", "summary": "The authors address two key limitations in current MI research: (1) task-specificity of circuit identification and (2) high computational costs for interpreting new tasks. Their solution introduces a modular circuits (MC) vocabulary - a collection of task-agnostic functional units, each consisting of a computational subgraph with an associated functional interpretation. These modular circuits can be reused across different language tasks, enabling modular circuit interpretability while reducing costs. The core innovation is conceptualizing a task circuit as modular components shared across different language tasks rather than building separate task-specific interpretations for each new scenario.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the invaluable feedback. Here, we reply to the mentioned concerns and questions. The [anonymous repo](https://anonymous.4open.science/r/ModCirc-4887/README.md) (\"the link\") stores supplementary results.\n\n> **Reviewer**: For a new task, verify whether the functional interpretations (FIs) derived by ModCirc are transferable and composable to match the ground-truth FIs.\n\n**Answer**: We test ModCirc's transferability and composability on two new tasks whose circuits and ground-truth FIs are established in existing work: indirect object identification (IOI) [1] and acronym detection [2]. Specifically, we train ModCirc on GPT-2 Small (the LLM [1, 2] perform on) with four NLP tasks (Trec, AGNews, etc.). ModCirc generates ten modular circuits (MCs), which contain 270 computational nodes selected from over 50k nodes. The results confirm that ModCirc can generate transferable and composable FIs:\n\n**IOI task** aims to predict the indirect object (IO) in a sentence where the subject (S) appears twice. For example, Mary (IO) and Amy (S) went out, and Amy (S) handed a pen to \\_\\_. [1] identifies the IOI's circuit as a set of attention heads. Each of them has a name such as \"induction head\" served as its FI. Our findings are:\n- ModCirc covers 92% (23/25) of ground-truth circuit heads in [1]; \n- **Transferability**: \n    - MC 0 identifies two Induction Heads, whose FI in [1] is to highlight prior occurred subjects. This aligns with the MC's FI of \"identifying specific entities,\" such as nouns. \n    - MC 1 contains two Name Mover (NM) Heads that \"attend to previous names [1]\" and one S-Inhibition (SI) Head that \"removes duplicate names identified by the NM Heads. [1]\" This aligns with the MC's FI of \"extracting and assessing contextual relevance,\" where NM Heads extract names and SI Heads assess and remove duplicates.\n    - MC 8 contains one NM Head and three Backup Name Mover (BNM) Heads. Unlike MC 1, MC 8's FI is to \"create associations or pointers between entities and their attributes,\" which matches the NM Head's role to associate and output the indirect object.\n- **Composibility**: MC 0 detects duplicate tokens, MC 1 assesses and removes duplication, and MC 8 outputs answer, matching the circuit structure in [1].\n\nThe **acronym detection** [2] predicts the acronym of a three-word phrase (e.g., Limited Liability Company -> LLC), where we cover 87.5% ground-truth circuit nodes (7/8):\n- **Consistency**: MC 0 captures Head 5.8, aligning MC FI \"extracting entities\" with head FI \"find capital letter\"; MC 1 contains letter mover (LM) heads 9.1.&11.4, aligning MC FI \"assessing context\" with head FI \"copy capital\". The \"assess\" here is copying procedure.\n- **Composability**: MC 0 identifies the capital letter position, and MC 1 copies the capital letter to answer position, composing circuit's FI in [2].\n\n[1] Wang, Kevin Ro, et al. \"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.\" ICLR, 2023.\n\n[2] García-Carrasco, Jorge, Alejandro Maté, and Juan Carlos Trujillo. \"How does gpt-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability.\" AISTATS, 2024.\n\n\n> **Reviewer**: ModCirc optimizes and evaluates on the same metrics, causing circular evaluations. Further, this discriminates against the baselines as they are not explicitly optimized upon those metrics—test FI transferability for comparisons to address the concern.\n\n\n**Answer**: \n- We use different metrics for evaluation and training. During evaluation we incorporate LLMs to score generated FIs for computing composability and consistency. However, we use non-LLM proxies (see Section 4.3.4) to estimate the metrics during training for efficiency and cost reasons. Thus, we do not discriminate against the baselines because the exact evaluation metrics are not used in training. \n- To avoid the concern that \"ModCirc performs well because the designed metrics discriminate against baselines,\" we further examine the FI transferability and composability for baselines. Specifically, we conduct experiments for the baselines in the same setting as in the last response. We observe that the baselines perform poorly in transferring FIs. For example, the Random baseline can only recover one node out of 25 nodes in IOI's circuit; it does not recover any nodes from the acronyms detection's circuit, making minimal nodes that modular circuits' FIs can apply to. Besides, Random's FIs are very similar across different MCs, failing to capture various functionalities of circuits' components.\n\n**Remark**: \n- We avoid the same circuits appearing in different tasks to serve different functions by restricting tasks within a domain, such as medical language tasks (see Definition 3.1 in the paper). \n- Supplementary C.2 and C.3 are only used to display MCs, so we do not write any claims. Please see the complete circuits in \"saved_results/ploted_circuits\" & \"saved_results/func_interp.jsonl\" in the link.", "claims_and_evidence": "The paper claims that \"By allowing different language tasks to share common MCs, our approach enables global interpretability while reducing costs by reusing established interpretations for new tasks.\" However, this claim lacks sufficient experimental validation in several important ways:\n\n- Missing Validation of Interpretation Transfer: The authors don't experimentally verify whether the functional interpretations (FIs) derived from modular circuits (MCs) actually match the ground truth interpretations when applied to new tasks. When a new task's significant computational subgraph (SCS) maps to multiple MCs from their vocabulary, there's no evaluation of how accurately the combined MC interpretations represent the true function of the new circuit.\n- Assumption Without Verification: The paper assumes that the functional interpretation of a new task's SCS can be composed from the interpretations of its constituent MCs. This core assumption - that interpretations are compositional and transferable - remains untested in the presented experiments.\n- Focus on Clustering Rather Than Interpretation: The evaluations primarily demonstrate the effectiveness of their clustering technique compared to baselines (basic clustering techniques) but don't measure the accuracy of the resulting interpretations when applied to new tasks.", "methods_and_evaluation_criteria": "### Lack of Validation for Interpretation Transfer and Compositionality\n\nThe paper critically fails to validate its fundamental assumption that functional interpretations (FIs) derived from modular circuits can accurately represent new tasks. While the authors claim their MC vocabulary enables reusable interpretations across different language tasks, they provide no experimental evidence comparing these composed interpretations against ground truth. The evaluation omits any verification that subgraphs can be meaningfully decomposed into modular circuits while preserving semantic interpretation. Specifically, the authors should have compared the FI of the original circuit with the FI derived from combining its modular components to verify whether their decomposition maintains interpretative fidelity. Instead of assessing the practical utility of their dictionary approach for interpreting new tasks, the paper focuses solely on the clustering quality of extracted modular circuits, leaving the core premise of transferable, compositional interpretations completely untested.\n\n### Problematic Baseline Selection and Circular Evaluation Design\nThe paper's evaluation methodology employs questionable baselines (Random, Frequent Random, KMeans, and Activation-based methods) that are inherently disadvantaged against ModCirc's reinforcement learning approach. This comparison lacks rigor as ModCirc is evaluated on the same metrics it was explicitly optimized for, creating a circular evaluation framework that virtually guarantees its superior performance. More critically, these experiments fail to address the paper's central claim about functional interpretation transfer, as they demonstrate only clustering effectiveness rather than validating whether the resulting modular decomposition preserves accurate interpretations of the original circuits.", "theoretical_claims": "The paper's central theoretical claim—that functional interpretations of neural circuits are compositional—remains unverified. This key assumption that a circuit's interpretation can be accurately reconstructed by combining interpretations of its modular components receives no experimental validation. The authors do not consider that circuits appearing in multiple tasks might serve fundamentally different functions when combined in novel ways. Simply because modules overlap across tasks doesn't guarantee their functional roles remain consistent when integrated into new contexts.", "experimental_designs_or_analyses": "As mentioned earlier, the experimental design has two significant flaws:\n\n- The paper lacks validation for its compositional interpretation assumption. It fails to compare the functional interpretations derived from combining modular components against ground truth interpretations of the original circuits, leaving the core premise of transferable interpretations untested.\n- The evaluation employs problematic metrics that don't address the central claim. Rather than assessing whether the clustering produces meaningful functional interpretations, the experiments focus solely on clustering quality. The true measure of success should be whether the functional interpretation of a new task's circuit can be accurately represented by combining the interpretations of its constituent modules.", "relation_to_broader_scientific_literature": "This paper extends mechanistic interpretability research by proposing a modular approach to understanding LLM behavior. While prior work focused on identifying task-specific circuits, this research introduces a method for discovering reusable functional units that can be combined to interpret new tasks. The approach offers a potentially more efficient framework for understanding model behavior across diverse language tasks.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "As discussed above, I reiterate the key points:\n\nStrengths:\n\n- If the main claim about compositional functional interpretations is supported through proper evaluation (comparing the FI of original circuits to the FI derived from combined modular components), this could be a very interesting and valuable contribution to mechanistic interpretability.\n\nWeaknesses:\n1. The paper's central theoretical claim—that functional interpretations are compositional—remains unverified. The authors do not validate that a circuit's interpretation can be accurately reconstructed by combining interpretations of its modular components, nor consider that shared circuits might serve different functions when integrated into new contexts.\n2. The evaluation methodology employs problematic baselines and metrics that don't address the central claim, measuring clustering effectiveness rather than whether the resulting modular decomposition preserves accurate interpretations of the original circuits.", "other_comments_or_suggestions": "None", "questions_for_authors": "If the authors address weakness 1 (details in earlier sections), I am happy to increase the score as the contribution is potentially interesting (strength). Specifically, compare the FI of a new task circuit derived using the proposed modular approach with the FI obtained directly from the new task circuit (more details mentioned earlier).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ugyUDgwq4j", "reviewer": "Reviewer_Ezqf", "summary": "This work aims to address key challenges in the current state of mechanistic interpretability literature, namely: (1) the limited generalization of results from task-specific circuit analysis and (2) the high human effort required to determine the functional interpretation of each computational node. To tackle these issues, it proposed the concept of modular circuits (MCs)—subgraphs within a model’s computation graph that are frequently utilized across different tasks. In addition to defining MCs and outlining their key desiderata, it introduced a framework for discovering MCs given a LLM and a set of tasks. Applying this framework to medical tasks, it identified MCs and found that both quantitative and qualitative evaluations show promising results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your dedicated time and effort in reviewing and providing invaluable feedback. We also thank you for recognizing the novelty and the significance of our contributions. We provide a point-to-point reply below for the mentioned concerns and questions. The [anonymous repo](https://anonymous.4open.science/r/ModCirc-4887/README.md) (\"the link\") is used for supplementary results.\n\n\n> **Reviewer**: When constructing a significant computational subgraph (SCS) of a task, ModCirc's computation node filtration criterion may cause the SCS to be as large as 320, so explore alternative node selection methods. For instance,  retain the top N nodes necessary to achieve a faithfulness score of 1.\n\n**Answer**: We apologize for any confusion. Our node selection process identifies the top ten nodes for each of the four node types (attention head, o_projection, mlp_in, and mlp_out) at each layer. This results in approximately 320*4=1280 nodes in total. We follow your suggestion to apply faithfulness as an alternative node selection criterion, it produces slightly more nodes: approximately 2400-2500 nodes across different tasks (Symptom2Disease: 2488, Medal: 2493, MedMcqa: 2496, MedicalAbstract: 2503). We believe both selection methods can produce suitable results for modular circuit generation. However, due to time constraints, we only perform experiments on finding SCSs for different datasets on MedLLaMA, but do not derive modular circuits from the faithfulness-based selection.\n\n> **Reviewer**: Conduct causal intervention to test if the generated functional interpretations are correct.\n\n**Answer**:  The functional interpretations of the modular circuits generated by our ModCirc method can faithfully reveal its functionality. Here, we show some example results for modular circuit 18 in MedLLaMA through causal intervention (see Fig. 3 and 4 in the link). Our ModCirc identifies this modular circuit as performing \"dosage detection.\" To verify this with causal intervention, we create a test dataset. In the dataset, each data item is a sentence containing at least one word related to dosage. We corrupt the dataset in two ways: (1) masking dosage tokens and (2) masking random non-dosage tokens. Results in Fig.3 and Fig. 4 show that masking dosage tokens significantly reduces circuit 18 activation, while random token masking either maintains or increases activation levels. The increased activation with random masking may occur because shorter sequences allow the circuit to focus more effectively on dosage detection. These findings confirm our interpretation that modular circuit 18 specializes in dosage detection.\n\n> **Reviewer**: Enumerate all the modular circuits found in MedLLaMa \n\n**Answer**: Thank you for the suggestion; please see the complete modular circuits set along with their functional interpretations in \"saved_results/ploted_circuits\" and \"saved_results/func_interp.jsonl\" in the link.\n\n> **Reviewer**: How would removing the modular circuits impact the LLM's performance?\n\n**Answer**: For each discovered modular circuit, we test how its removal affects the logits of the ground-truth token on the Symptom2Disease dataset, as shown in Fig.5 in the link. Removing these circuits causes significant performance degradation, in some cases up to 60%. Interestingly, removing a few specific modular circuits increases the logits. While this performance increase is not yet well explored, we can explain why the removals caused no degradation. These particular circuits were irrelevant to the specific task (i.e., the task circuit had no intersection with those modular circuits). Note that we also have similar observations on other datasets.\n\n> **Reviewer**: Writing suggestions: 1) improve the explanation of locality; 2) include a brief conclusion for Section 4.3; 3) Cite \"Causal Tracing\".\n\n**Answer**: Thank you for the suggestions; we will carefully incorporate them into our work.", "claims_and_evidence": "This work presents two primary claims:  \n1. The formalization of a modular circuit vocabulary that defines concrete desiderata, namely Consistency, Locality, Reusability, Compositionality, and Globality.  \n2. The introduction of ModCirc, a framework designed to identify modular circuits that satisfy these desiderata.\n\nBoth claims are supported by evidence. Specifically, the ModCirc framework demonstrates the practical implementation of the proposed vocabulary. Furthermore, its evaluation on medical tasks highlights its effectiveness.", "methods_and_evaluation_criteria": "Overall, while I am broadly satisfied with the soundness of the methods and evaluation criteria, the paper could be improved by incorporating the following points (in addition to those mentioned in the Experimental Designs or Analyses section):\n- The proposed ModCirc framework applies Edge Attribution Patching to identify causal components. However, their filtration criterion retains the top 10 nodes at each layer, meaning that for the LLaMA-8B model, the SCS would consist of 320 components.\n  - While this criterion is not entirely unreasonable, I encourage the authors to explore alternative selection methods based on existing metics such as faithfulness, minimality, and completeness. For instance, they could retain the top N nodes necessary to achieve a faithfulness score of 1.\n- Additionally, functional interpretation is performed using GPT-4 through prompting. While the generated text can suggest hypotheses about the functionality of SCS components, it does not verify their correctness. Existing research often employs causal intervention techniques to validate such hypotheses. I recommend incorporating these techniques into the framework to strengthen the analysis.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The paper could be improved by incorporating following suggestions:\n- The paper presents an example of a modular circuit that identifies biological entities and interactions. However, providing additional examples would strengthen the argument. Ideally, the authors could enumerate all identified MCs, perhaps in the appendix.\n- I would have liked to see ablation experiments on the identified MCs. For instance, if the previously mentioned MC were removed from the model, how would it impact the model’s performance across various tasks?\n  - Such an analysis would also help validate the claims in Section 5.3 regarding the usefulness of the investigated MC for different medical tasks.\n- Finally, while ModCirc generally outperforms other baselines, the differences in scores are not substantial (based on results in Table 1), indicating significant room for further improvement in the proposed framework.", "relation_to_broader_scientific_literature": "The paper effectively outlines key limitations in the current state of mechanistic interpretability literature, namely the limited generalization of results from task-specific circuit analysis and the high human effort required. The proposed modular circuit vocabulary approach offers a potential solution to both issues. Specifically, developing such a vocabulary could accelerate our understanding of deep neural networks by enabling researchers to leverage an existing library of modular circuits, reducing the need to repeatedly discover similar results.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "- The explanation of the Locality desideratum could be improved; I initially found it difficult to grasp.\n- Additionally, I encourage the authors to include a brief conclusion paragraph after Section 4.3.4 and before Section 4.4. This summary would help reinforce the key points of Section 4.3, particularly for readers who may not be familiar with GNNs and/or RL.\n- Section 2 (line 97) mentions “Causal Tracing” without proper citation. I would recommend authors to cite [1].\n\n[1] Meng et al, “Locating and Editing Factual Associations in GPT”, 2023.", "other_comments_or_suggestions": "List of typos:\n- Section 4 (line 125): effectives -> effectively.\n- Section 4.3 (line198): ccording -> according.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVrm8oKj1A", "reviewer": "Reviewer_h4Pi", "summary": "This paper proposes a novel formulation of the circuit discovery problem, which is a mechanistic interpretability task concerned with identifying a small subset of an LLM’s components responsible for a specific task. The authors propose a variant of this problem that involves identifying multiple subsets of the model’s components (a circuit vocabulary) that are modular and are used in multiple different tasks. The paper defines 5 criteria to evaluate a modular circuit vocabulary: consistency, locality, reusability, composability, and globality.\nThe paper then presents a method to tackle the proposed problem. The method consists of (1) identifying an initial set of computational subgraphs for each task considered, (2) generating an initial partitioning of the subgraphs, (3) training a GNN to partition the reusable subgraphs using an RL-based procedure that optimizes for the proposed evaluation criteria. The method is evaluated and compared to a set of baselines on 4 tasks in the medical domain. Additionally, the authors present an ablation study and an analysis of the effect of the method’s hyperparameters.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for providing invaluable feedback. We provide a point-to-point reply below to address the concerns and questions. We use the [anonymous repository](https://anonymous.4open.science/r/ModCirc-4887/README.md) (termed \"the link\") to store supplementary results.\n\n> **Reviewer**: Explain how consistency is calculated in evaluation.\n\n**Answer**: \"Consistency\" is calculated by following steps:\n- For each modular circuit (MC) that intersects with the task's corresponding circuit, we use GPT-4o mini to generate a task-specific functional interpretation (FI) for it.\n- We then prompt GPT-4o mini to rate (on a scale of 1-5) how consistent this task-specific interpretation is with the corresponding FI in our MC vocabulary.\n- Finally, we calculate the mean value of these consistency scores across all MCs to determine the overall consistency for the method performed on a given dataset.\n\n> **Reviewer**: The Consistency score of ModCirc does not achieve a significant lead than baselines.\n\n**Answer**: \n- Our method prioritizes reusability, where ModCirc has much higher performance than the baselines in Table 1 in the paper. This addresses a fundamental requirement: MC vocabulary should cover the circuit of new unseen tasks. All baselines fail to meet this requirement, as their generated vocabularies only capture a small portion of the task's circuit. This creates bias—their consistency and composability scores appear high because they are only tested on small circuit segments (it is easier to be consistent on partial circuits than complete ones).\n- This metric relies on GPT 4o-mini to provide scaled scores evaluating FI consistency. However, GPT 4o-mini may inflate scores, acknowledged as Overconfidence Bias [1], making the lead less distinguishable.\n- We use this metric due to the lack of theoretically principled FI consistency metrics in the community. We view this more as an opportunity for future work rather than a limitation of our research.\n- We believe that having domain experts manually score FI consistency would be optimal. Unfortunately, budget constraints prevented us from pursuing this approach.\n\n[1] Li, Haitao, et al. \"Llms-as-judges: a comprehensive survey on llm-based evaluation methods.\" arXiv preprint arXiv:2412.05579 (2024).\n\n> **Reviewer**: Removing RL-based subgraph partitioning leads to a low-performance drop.\n\n**Answer**: In fact, the RL-based subgraph partitioning is contributing significantly to the ModCirc's performance:\n- Please refer to the extended ablation results on additional datasets in Table 1 (linked). These results show that ModCirc consistently outperforms (up to 5%) its ablation variant without RL-based subgraph partitioning. (Note: Reusability is excluded from this comparison since it is primarily influenced by the initial partitioning rather than the RL-based subgraph partitioning.) \n- As mentioned in our previous response, GPT-4o mini tends to exhibit overconfidence bias, which may understate the actual performance gap. To better illustrate the importance of RL-based subgraph partitioning, we include a qualitative case study. Specifically, we apply ModCirc to GPT-2 Small using four training datasets and evaluate two unseen datasets: Indirect Object Identification (IOI) and Acronym Detection [2, 3]. Our results show that ModCirc recovers the ground-truth circuits with consistent and composable FI (see Fig. 1-2 in the link). However, the variant without RL-based partitioning fails to retrieve the correct circuits and produces nearly identical FIs across different MCs, making it impossible to match the ground-truth circuit's FIs.\n\n[2] Wang, Kevin Ro, et al. \"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.\" ICLR, 2023.\n\n[3] García-Carrasco, Jorge, Alejandro Maté, and Juan Carlos Trujillo. \"How does gpt-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability.\" AISTATS, 2024.\n\n> **Reviewer**: Explain why the tasks are restricted to a particular domain and why using the medical domain. \n \n**Answer**: \n- This is to guarantee the functionality transferability assumption. When two tasks are in different domains, their task semantics can significantly vary, making generalizing functionality challenging. \n- We choose the medical field as it is a high-stakes field where interpretability is significant for model deployments. Furthermore, in the medical domain, some well-performed medical specialized LLMs such as MedLLaMA and many high-quality medical language datasets are available, facilitating us to explore the LLM's behavior more precisely.", "claims_and_evidence": "The authors claim that a modular circuit vocabulary should exhibit consistency, among other properties. However, the empirical results indicate that the proposed method achieves consistency scores that are nearly on par with simple “random” and “frequency” baselines.  A high degree of functional consistency of a component across different tasks seems a necessary condition to claim modularity, and this is not clear from the empirical results.", "methods_and_evaluation_criteria": "- The paper defines the consistency criterion in Section 3, which relies on a “synonymity” operator computed using an LLM. However, the implementation details of this operator and the practical computation of this quantity are not clearly described.\n- The ablation study in Table 2 shows only a minimal performance drop when the RL-based subgraph partitioning is removed. This result suggests that the proposed optimization procedure may not be essential for achieving the desired task performance.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The experimental setting assumes that the tasks under consideration belong to a specific domain (as noted in Definition 3.1, line 117). The rationale behind this domain-specific assumption is not well explained, and the choice of the medical domain for evaluation is not adequately motivated.\n- The paper relies on an auto-interpretability procedure to generate functional interpretations of components. This reliance is potentially problematic given that LLM-generated interpretations can be imperfect and may not fully capture the underlying functionality.", "relation_to_broader_scientific_literature": "The contributions of the paper connect with the increasingly larger literature about LLM circuit finding and behavior localization. This is adequately discussed by the authors in the related work section.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "- The use of abbreviations, such as “MC” for modular circuit vocabulary and “ModCirc” for the method, can be confusing at first reading. \n- line 222: “modualr”", "questions_for_authors": "1. Could you provide more details on how the “synonymity” operator is implemented, and explain the practical steps involved in computing the consistency metric using an LLM?\n2. The ablation results in Table 2 show only a minimal performance drop when the RL-based subgraph partitioning is removed. Could you clarify the specific contribution of the RL component, and whether a simpler method might achieve similar performance?\n3. The consistency scores achieved by ModCirc are nearly identical to those of the “random” and “frequency” baselines. How do you interpret this finding, and what additional evidence supports the claim that your method attains the high degree of functional consistency required for a truly modular circuit vocabulary?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QyOf6JcUI3", "reviewer": "Reviewer_4yNw", "summary": "The paper introduces ModCirc, a framework for global-level mechanistic interpretability of LLMs by discovering modular circuits -- task-agnostic functional units that enable cross-task interpretability while reducing computational costs. It defines the MC vocabulary discovery problem with five evaluation criteria: consistency, locality, reusability, composability, and globality. The proposed reinforcement learning-based graph neural partitioning method identifies reusable computational subgraphs and partitions them into modular circuits, which are then assigned functional interpretations. Experiments on MedLLaMA-8B across diverse medical tasks demonstrate that ModCirc effectively identifies modular circuits with strong reusability and competitive composability and consistency, enabling scalable and interpretable LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you've dedicated to reviewing and providing invaluable feedback. We provide a point-to-point reply below for the mentioned concerns and questions. We use the [anonymous repository](https://anonymous.4open.science/r/ModCirc-4887/README.md) (termed \"the link\") to store supplementary results.\n\n> **Reviewer**: Scale the experiments to general LLMs.\n\n**Answer**: Thank you for the suggestion. We further conduct our ModCirc in GPT-2 Small (see Fig. 1 and Fig. 2); we train the ModCirc on four datasets: Trec, AGNews, MPQA, and Universal Dependency. We generate ten modular circuits (270 computational nodes) from over 50k nodes. Our generated modular circuits successfully uncover the ground-truth circuits with consistent and composable functional interpretations on tasks researched in two existing works. Due to the space limit, we kindly refer you to our response to reviewer aN6A's first question for more observations.\n\n> **Reviewer**: How to verify GPT-4 generated FIs are correct?\n\n**Answer**: We may verify it with causal intervention. Here, we show some example results for modular circuit 18 in MedLLaMA through causal intervention (see Fig. 3 and 4 in the link). Our ModCirc identifies this modular circuit as performing \"dosage detection.\" To verify this with causal intervention, we create a test dataset. In the dataset, each data item is a sentence containing at least one word related to dosage. We corrupt the dataset in two ways: (1) masking dosage tokens and (2) masking random non-dosage tokens. Results in Fig.3 and Fig. 4 show that masking dosage tokens significantly reduces circuit 18 activation, while random token masking either maintains or increases activation levels. The increased activation with random masking may occur because shorter sequences allow the circuit to focus more effectively on dosage detection. These findings confirm our interpretation that modular circuit 18 specializes in dosage detection.\n\n\n> **Reviewer**: The FIs are not examined by human experts.\n\n**Answer**: We recognize that the FIs are not thoroughly inspected by human experts. However, due to the budget limit, we do not have access to perform the expert inspection. We plan on releasing our experiment results online and invite broad crowdsourcing support to finish the human verification.\n\n> **Reviewer**: Explain why ModCirc does not always outperform baselines in composability and investigate methods to improve functional stability across tasks.\n\n**Answer**: \n- Our approach emphasizes reusability and demonstrates significantly better performance than the baselines shown in Table 1. This meets a crucial need: a modular circuit vocabulary must be able to cover the circuits for new, unseen tasks. The baselines do not satisfy this need, as their generated vocabularies only contain a limited segment of unseen task circuits, leading to bias. Their reported consistency and composability scores appear high because they are only evaluated on smaller circuit portions, which is inherently easier to maintain consistency than entire circuits.\n- This metric depends on GPT-4o mini to yield scaled assessments of functional interpretation consistency. However, there are concerns that GPT-4o mini may produce inflated scores, an issue recognized by the community as Overconfidence Bias [1], potentially diminishing their distinctiveness.\n- We design this metric due to the absence of well-acknowledged metrics for assessing functional interpretation consistency in the field. We perceive this more as a chance for future exploration rather than a limitation of our research.\n- We think having domain experts to evaluate functional interpretation consistency would be the ideal approach to measure the consistency more faithfully. However, we are unable to finish it due to the budget limit.\n\n[1] Li, Haitao, et al. \"Llms-as-judges: a comprehensive survey on llm-based evaluation methods.\" arXiv preprint arXiv:2412.05579 (2024).\n\n> **Reviewer**: Provide run time comparison for ModCirc and the baselines. \n\n**Answer**:\nThe table below illustrates the average wall-clock time to experiment with the methods. We observe that ModCirc is about 2.5x slower than most baselines, except for \"Clust.\", where it is 2x slower. However, it should be considered that most baselines are heuristic methods requiring much less time than a machine learning method. Thus, we deem that the time difference is a necessary trade-off for the generally improved performance of ModCirc displays when compared to the baselines. \n\n|  Method | Time (s) |\n|:-------:|:--------:|\n|   Act.  |  941.14  |\n|  Clust. |  1321.00 |\n|  Freq.  |  957.13  |\n|  Random |  961.75  |\n| ModCirc |  2357.27 |", "claims_and_evidence": "Claim1 -- Functional Interpretations Accuracy: The paper relies on GPT-4-generated FIs, but does not rigorously evaluate their correctness.\n\nClaim2 -- Scalability to General LLMs: All experiments focus on MedLLaMA-8B, leaving open the question of whether ModCirc generalizes to larger or non-medical models.\n\nClaim3 -- Composability and Consistency Improvements: ModCirc does not always outperform baselines on these metrics, suggesting that task-agnostic modular circuits might not always maintain strong interpretability across different contexts.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria align well with the problem of global-level mechanistic interpretability in LLMs. However, the evaluation is limited to MedLLaMA-8B and medical NLP tasks, which restricts generalizability. Moreover, functional interpretation quality is not rigorously validated beyond GPT-4 outputs,", "theoretical_claims": "The paper does not present formal mathematical proofs but introduces a set of evaluation criteria and an algorithmic framework (ModCirc) for discovering modular circuits in large language models.", "experimental_designs_or_analyses": "The study focuses only on MedLLaMA-8B, a domain-specific model, and does not test whether ModCirc works for general-purpose LLMs.\n\nClaims about scalability and generalization should be validated on non-medical tasks.\n\nFunctional Interpretations are not rigorously evaluated.", "relation_to_broader_scientific_literature": "The paper builds on mechanistic interpretability (MI) research, particularly prior work on task-specific circuit discovery (e.g., activation patching and causal tracing in LLMs) but extends it by introducing modular circuits (MCs) that generalize across tasks.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n(1) The paper introduces a novel formulation of modular circuit discovery, extending mechanistic interpretability research by making circuits reusable across tasks rather than task-specific.\n\n(2) The paper is well-structured, with clear explanations of evaluation criteria, algorithmic steps, and experimental setups, making it easy to follow.\n\n(3) Code with a detailed readme file is provided.\n\n\nWeaknesses:\n\n(1) The experiments are restricted to MedLLaMA-8B and medical NLP tasks, limiting claims of scalability to general LLMs.\n\n(2) The GPT-4-generated FIs are not validated by human experts, raising concerns about potential biases or hallucinations in interpretations.\n\n(3) The reinforcement learning-based neural partitioning may be computationally expensive, and alternative differentiable or self-supervised methods could be explored for efficiency.", "other_comments_or_suggestions": "(1) Provide runtime comparisons between ModCirc and baselines\n\n(2) Explain why ModCirc does not always outperform baselines in composability and investigate methods to improve functional stability across tasks.", "questions_for_authors": "(1) Have you tested ModCirc on general-purpose LLMs or non-medical tasks?\n\n(2) How do you ensure the accuracy of GPT-4-generated FIs?\n\n(3) How does the runtime of ModCirc compare to baselines?\n\n(4) Why does ModCirc sometimes underperform baselines in composability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

This paper addresses limitations in current mechanistic interpretability (MI) research by introducing the concept of modular circuits (MCs) - task-agnostic functional units within LLMs that can be reused across different language tasks. The authors formalize five criteria for evaluating modular circuit vocabularies: consistency, locality, reusability, composability, and globality. They propose ModCirc, a novel framework using reinforcement learning-based graph neural partitioning to discover modular circuits, and demonstrate its effectiveness on MedLLaMA-8B across various medical tasks. This approach enables more global interpretability while reducing the costs associated with interpreting new tasks by reusing established interpretations.

**Reasons to Accept**

- The formalization of modular circuit vocabulary and the ModCirc framework offers a potential solution to key challenges in mechanistic interpretability and could accelerate our understanding of deep neural networks (Ezqf).
- The approach extends current task-specific mechanistic interpretability patterns by enabling cross-task interpretability while reducing computational costs (4yNw).
- The paper is well-structured with clear explanations of the evaluation criteria, algorithmic steps, and experimental setups (4yNw).
- In response to reviewer concerns, the authors demonstrated that ModCirc successfully recovers ground-truth circuits with consistent and composable functional interpretations on unseen tasks (aN6A, 4yNw).

**Potential concerns and the authors' responses**

1. Reviewer aN6A raised the concern about the composability and transferability. The authors addressed this in their rebuttal with new experiments on indirect object identification and acronym detection tasks.
2. The paper relies on GPT-4-generated functional interpretations without rigorous validation by human experts (4yNw, Ezqf). The authors acknowledged this limitation due to budget constraints but demonstrated causal intervention techniques to verify interpretations.
3. The consistency scores achieved by ModCirc were nearly on par with simple "random" and "frequency" baselines, raising questions about the method's effectiveness (h4Pi). The authors explained that baselines' consistency scores appear high because they only cover small circuit segments, while ModCirc covers much more of the task circuit as shown by its higher reusability scores.
4. The ablation study showed only a minimal performance drop when the RL-based subgraph partitioning was removed (h4Pi). The authors provided additional ablation results showing more significant performance differences across datasets and demonstrated qualitative differences in the quality of the generated interpretations.
Decision: Accept (poster)