Verifying Chain-of-Thought Reasoning via Its Computational Graph
Authors: Zheng Zhao
Yeskendir Koishekenov
Xianjun Yang
Naila Murray
Nicola Cancedda
Venue: ICLR 2026 Oral (2026)
Keywords: Mechanistic Interpretability
Chain-of-Thought Reasoning
Attribution Graphs
Primary area: interpretability and explainable AI
TL;DR: We introduce CRV, a white-box methodology that treats attribution graphs as execution traces, and use it to provide evidence that flawed reasoning has a verifiable computational structure.

Abstract: Current Chain-of-Thought (CoT) verification methods predict reasoning correctness based on outputs (black-box) or activations (gray-box), but offer limited insight into \textit{why} a computation fails. We introduce a white-box method: \textbf{Circuit-based Reasoning Verification (CRV)}. We hypothesize that attribution graphs of correct CoT steps, viewed as \textit{execution traces} of the model's latent reasoning circuits, possess distinct structural fingerprints from those of incorrect steps. By training a classifier on structural features of these graphs, we show that these traces contain a powerful signal of reasoning errors. Our white-box approach yields novel scientific insights unattainable by other methods. (1) We demonstrate that structural signatures of error are highly predictive, establishing the viability of verifying reasoning directly via its computational graph. (2) We find these signatures to be highly domain-specific, revealing that failures in different reasoning tasks manifest as distinct computational patterns. (3) We provide evidence that these signatures are not merely correlational; by using our analysis to guide targeted interventions on individual transcoder features, we successfully correct the model's faulty reasoning. Our work shows that, by scrutinizing a model's computational process, we can move from simple error detection to a deeper, causal understanding of LLM reasoning.

Published as a conference paper at ICLR 2026
VERIFYING CHAIN-OF-THOUGHT REASONING VIA ITS COMPUTATIONAL GRAPH
Zheng Zhao 133- Yeskendir Koishekenov Xianjun Yang² Naila Murray' Nicola Cancedda'
FAIR at Meta Meta Superintelligenee Labs University of Edinburgh
zheng zhao@ed. ac. uk, yeskendir@meta, com, xianjunyangeucsb.. edu
murrayn@meta.com com, canceddan Srr.ddddnnamm.....
ABSTRACT
Current Chain-of- ohhouhhh (CoT) verification methods predict reasoning correct- ness based outputs (black-box) or activations (gray-box), but offer limited in- sight into why a computation fails. We introduce a white-box method: Circuit- based Reasoning Verification (CRV). We hypothesize that attribution graphs of correct CoT steps, viewed as execution traces of the model's latent reasoning cir- cuits, possess distinet structural fingerprints from those of incorrect steps. By training a classifier on structural features of these graphs, we show that these traces contain powerful signal of reasoning errors. Our white-box approach yields novel scientific insights unattainable by other methods. (I) We demonstrate that structural signatures of error are highly predictive, establishing the viability of verifying reasoning directly via its computational graph. (2) We find these sig- natures be highly comaann-specifie revealing that failures different reasoning tasks manifest as distinct computational patterns. (3) We provide evidence that these signatures are not merely correlational: by using our analysis to guide tar- geted interventions on individual transcoder features, successfully correct the model's faulty reasoning. Our work shows that, by scrutinizing a model's com- putational process,
move from simple error detection to deeper, causal understanding of LLM reasoning.
INTRODUCTION
Chain-of-Thougee (CoT; Wei al. 2022 Kojima al. 2022) prompting has proven be powerful
method for boosting the performance of Large Language Models (LLMs). This capability is now central the latest generation of reasoning models, such DeepSeek-RI (DeepSeek-Al et al. 2025) and OpenAl's ol (OpenAl et al. (024).. Despite this success, a fundamental vulnerability persists across the spectrum of these systems: the reasoning process itself sometimes flawed
(Turpin et al. 2023 Li et al.] 2025b Arcuschin et al. 2025 Lindsey et al., 2025 2025b).
Chen et al.,
This reliability gap has spurred research into automated verification. Current methods fall into two main categories. Black-box approaches analyze the generated text or final logit distribution (Jacovi al. 2024: Wang al., 2025b Baker et al. 2025). Gray-box approaches look at the model's internal state, using simple probes on raw activations or analyzing the trajectory of hidden states (iee et al. 2025 Zhang al. 2025 Afzal et al. 2025 Bi et al. 2025 Wang al. 2025a). While insightful, these methods are fundamentally limited; they can detect that a model's 's internal state is correlated with error, but not explain why the underlying computation leads
error.
This limitation motivates a deeper, more mechanistic approach. We postulate that models implement latent algorithms that solve specific tasks through specialized subgraphs, circuits (Olah et al.) 2020 Elhage et al. 202I). From this perspective, reasoning failure is not merely an erroncous state, but flaw in the execution of latent algorithm. To diagnose such flaws requires inspecting the underlying computational process, akin to examining an execution trace in classical software. We propose to approximate this trace by constructing an attribution graph (Dunefsky al. 2025 structural representation of the causal information flow between model components.
Work done during internship Meta FAIR

---

Published as a conference paper at ICLR 2026
For such graph to as meaningful trace, its components must be interpretable. We therefore first create an interpretable surrogate model by replacing its standard MLP modules with trained transcoders (Dunefsky al., 2025). We then construct and analyze attribution graphs sparsely activating features of such surrogate model (Ameisen et al. 2025). Finally, to formally test whether these traces contain detectable signal of error, train diagnostic classifier on their structural properties. This entire methodology. which we call Circuit-basee Reasoning Verifica- tion (CRV), is thus designed as scientific instrument to investigate our central hypothesis: that reasoning failures manifest as detectable structural signatures on their computational execution traces, which can be leveraged for automated verification.
the
As scientific instrument, CRV requires a controlled experimental setting. While advanced reason- ing models employ complex mechanisms like search and backtracking, their convoluted reasoning paths can obscure the fundamental computations of single reasoning step. Our work therefore focuses on standard, instruction-tune vides a clearer window into the primitive computations that underpin emergent reasoning. While our approach, despite being effective, is too computationally intensive to be intended as a prac- tical, drop-in verifier, it yields novel scientific insights unattainable by other methods. Our main contributions are therefore not just about performance, but about understanding:
models generating autoregressive CoT, as this paradigm pro-
We introduce Circuit-based Reasoning Verification, a white-box method for analyzing rea- soning failures, showing that verifying reasoning via its computational graph is feasible.
We find that the structural signatures of error are highly domain-specific, revealing that failures in executing different reasoning tasks manifest distinct computational patterns.
We establish the causal role of these error signatures, successfully correcting faulty reason- ing via targeted interventions on individual transcoder features.
To support future research, we release datasets with step-level correctness labels for CoT reasoning on synthetic and real-world tasks, along with our trained transcoders
2 PROBLEM FORMULATION AND PRELIMINARIES
2.1 PROBLEM STATEMENT
Let an LLM generate Chain-of-Thought S = (81, 82, , 8m) to solve a problem, where each step 8 is a sequence of tokens. During the generation of step 8, the underlying model produces a computational state M,, From this state, we construct attribution graph G; = (V, E), where vertices V represent interpretable features and tokens, and edges represent the causal influence between them (see Section 3.2) From each graph Gü extract fixed-size feature vector x; (GG)), where ø is a feature extraction function designed to capture the graph's 's structural properties. We term this vector the step's structural fingerprint. Our goal is to learn a diagnostic classifier fo that takes this structural fingerprint as input to predict the correctness of the reasoning step:
j fx()
where i E (correct, incorrect].
2.2 PRELIMINARIES: CIRCUITS IN TRANSFORMERS
The term "cireuit" in mechanistic interpretability refers to specific subgraph within neural network that implements a uuman-underrtanddable algorithm (Olah et al., 2020). In Transform- (aaswani al. 2017), these circuits are composed of attention heads and MIP computations. Our work is conceptually motivated by the prospect of finding patterns distinguishing sound and faulty activations of circuits involved in reasoning. While our method does not observe these cir- cuits directly, our hypothesis is that they cast detectable structural fingerprints onto the attribution graphs we construct. A primary goal of our subsequent analysis therefore to interpret the graph- based features that are most predictive of failure as the signatures of these underlying error patterns.
2

---

Published as a conference paper at ICLR 2026
Repiaos MLP Modules with Transcaders
2) Canstruct Altribution Graphs
Extract Graph Features
000O
0000
(0000
O
Layer3
Node Stons
Topoingpcal
Fasturee
Layer
O O
M.P
O O
PLT
0000
0000
(0000
ooo
Layer
Diag Classifier
p000
0000
oo00
Layer
Enbeddings
Comrect
Incomret
Neiinn
Transoder
Steam
Feeeume
Figure 1: The CRV pipeline. (1) The LLM's MLP modules are replaced with per-layer transcoders (PLTs), making it interpretable. (2) For given CoT step, we generate attribution graph captur- ing causal flow between interpretable features and model components. (3) Structural features are extracted from this graph, and (4) fed to diagnostic classifier to predict the step's correctness.
2.3 PRELIMINARIES: TRANSCODERS FOR INTERPRETABLE FEATURES
A significant challenge in analyzing model activations is their high dimensionality and lack of direct interpretability. A powerful approach to this challenge is to learn a sparse, overcomplete basis for these activations using a sparse autoencoder (SAE: Cunningham et al. (023). An SAE is trained to reconstruct an activation vector & E IRd from a much higher-dimensional but mostly zero, feature set of learned, interpretable
vector E RD, where D > d. The elements of f correspond features, sparsely activated by inputs. While the canonical SAE objective is to reconstruct its own input (fz) æ),, our work leverages variant known as a transcoder (Dunefsky et al. (2025), which is instead trained to approximate the input-ouiput function of target component, such as an MLP (f(z) = MLP(x)). This approach makes the transcoder true functional substitute for the original module. Its objective is not mere reconstruction, but the emulation of computational step in an interpretable, sparsely activated basis. By replacing model's standard MI. module with a trained transcoder, we force its intermediate computations to be represented not by dense vector, but by sparse combination of these meaningful features.
METHODOLOGY
Unlike in Process Reward Modeling (PRM), where the goal is limited to judging the correctness of reasoning step, we take the perspective of model developer interested in debugging reasoning failures in a specific model to which they have full access. We introduce Verification (CRV), method for detecting flawed reasoning by analyzing its structural fingerprint.
Reasoning
3.1 DATASET CURATION AND STEP-LEVEL ANNOTATION
A prerequisite for developing our method is a dataset with reliable step-level correctness labels. Furthermore, our white-box methodology imposes a critical requirement that distinguishes our data needs from prior work. Since CRV analyzes the causal computational graph that produces rea- soning step. we must capture the full internal state of our specific model during the generation process. Consequently, existing text-only datasets such as PRM800K (Lightman et al., 2024) and REVEAL (Jacovi et al. 20244,, which provide static '(text, label)' pairs and are designed for training black-box verifiers, our own model's CoT outputs to create the necessary '(text, label, computational trace)' tuples for analysis. We therefore created new benchmark covering both controlled synthetic tasks and the real-world GSM8K dataset (Cobbe et al. 2021).
incompatible with mechanistic approach. We must generate and label
Synthetic Datasets (Boolean and Arithmetic). To study reasoning failures in controlled environ- ment, we generated two datasets. The first involves evaluating complex boolean expressions, while the second involves multi-step arithmetic problems. The motivation for these datasets is the unam-
3

---

Published as a conference paper at ICLR 2026
biguous ground truth: the correctness of any step in the reasoning chain (e.g.. "15 + 22") can be verified automatically by simple parser and evaluator. This allows us to generate a large, labeled dataset for initial training and analysis. Furthermore, these tasks are intrinsically compositional, and the complexity of samples can be fully controlled. Further details are provided in Appendix A
Step-Level Annotation for GSM8K. Annotating a real-world dataset like GSM8K is challenging.
To scale, we used a an expert judge. For each CoT, the judge evaluated step correctness given the full problem context. We validated these labels through manual review of substantial subset, yielding high-fidelity dataset for real-world reasoning. Further details are provided in Appendix
process with stronger LLM (e.g., Llama 3.3 70B Instruct) as
3.2 CIRCUIT-RASED REASONING VERIFICATION (CRV)
CRV is a four-stage pipeline designed to classify the correctness of CoT step by analyzing the computational graph of a modified, interpretable LLM. An overview is presented in Figure
3.2. STEP I: REPLACING MLPS WITH INTERPRETABLE TRANSCODERS
The foundation of CRV is an architectural modification that makes the target LLM interpretable. For each MLP module the model, we train a corresponding transcoder on a large, diverse dataset of activations harvested from the original LLM The training objective combines an L2 reconstruc- tion loss with TopK activation function, which enforces sparsity by preserving only the k-largest feature activations. Once trained, we replace the MLP module for each layer in the LLM with its corresponding transcoder. The forward pass of the model is now forced to flow through these sparse, interpretable bottlenecks. All subsequent analysis is performed on this modified, interpretable re- placement model. Full details of the transcoder architecture and training are provided in Appendix
3.2.2 STEP 2: CONSTRUCTING STEP-LEVEL ATTRIBUTION GRAPHS
With our transcoder-infused replacement model, we require a principled method trace information flow and construct causal graph of the computation. To this end, we adapt the recent circuit analysis methodology of Dunefsky et al. (2025). Applying their greedy path-finding algorithm allows us to trace high-attribution connections backward from the final logits, yielding a sparse, weighted, directed graph G, = (V,E) for each reasoming step s;. This graph represents the core computational subgraph, where the nodes V are the disjoint union of input tokens, active transcoder features, and output logits. The directed edges E represent the high-attribution causal pathways between these components (e.g., from an early-layer feature to a later-layer feature, or from a feature to a logit), with weights quantifying the strength of their influence. For a complete derivation and description of the circuit-finding algorithm, we refer the reader the original work (Dunefsky et al. 2025
3.2.3 STEP 3: EXTRACTING INTERPRETABLE GRAPH FEATURES
From each attribution graph Gj we extract fixed-size feature vector x; structural fingerprint of the computation. We prune the graph to its most influential components, retaining nodes and edges accounting for threshold (c.g.. 80%) of total influence to the final logits. The feature set, calculated on this pruned subgraph (unless stated otherwise), is organized into three hicrarchical levels.
Global Graph Statisties: These features capture high-level summary of the computational sub- graph, including the count of active feature nodes after pruning and the final logit probability and entropy. They provide a coarse measure of the computation's 's complexity and uncertainty.
Node Influence and Activation Statistics: This group quantifies the properties of the inter- pretable feature nodes. We compute statistics (mean, max, std) on their activation values and in- fluence scores. This helps distinguish computations driven by few highly active, decisive features from those driven by diffuse combination of many weak features. We also include histogram of active features by layer, which characterizes the computational depth of the reasoning step.
Thiis also referred as per-layer transcoders (PLTs) by Ameisen ct al. 2025).
We use implementation from Hanna 2025) for computing attribution graphs in work.
4

---

Published as a conference paper at ICLR 2026
Topological and Path-Based Features: To analyze the structure of the information flow, com- pute rich set of topological features on the pruned subgraph. These include graph density, centrality measures (degree, betweenness) to identify computational hubs, and connectivity metrics.
This comprehensive feature set provides the foundation for our diagnostic classifier. full list and detailed motivation for each feature is provided in Appendix C.1
3.2.4 STEP 4: DIAGNOSTIC CLASSIFIER
For the final classification step, we use a Gradient Boosting Classifier (GBC) trained on the extracted feature vectors: Jo (x:) = Øt- GBC suits for heterogeneous, tabular features and provides robust feature importance measures, which we leverage to identify the most predictive structural properties of eπтоr circuits. We also benchmark against several alternative classifiers in Appendix C..4
4 EXPERIMENTS
We conduct a series of experiments designed to validate the central hypothesis of our work: that the attribution graphs of reasoning steps contain rich, structural signal of their correctness. Our evaluation is structured around three primary research questions. First, we investigate whether CRV's white-box approach significantly outperforms comprehensive suite of gray-box and black- box baselines verification accuracy and test robustness to domain shifts and increasing task difficulty (RQ1). Next, we analyze our trained models to identify which specific computational structures within the graph are most predictive of failure, moving from detection to mechanistic un- derstanding (RQ2). Finally, we conduct exploratory studies assess if these mechanistic insights can be used to perform targeted, causal interventions that correct faulty reasoning (RQ3).
4.1 EXPERIMENTAL SETUP
Models and Datasets. Our experiments conducted on the Llama 3.1 8B Instruct model (AI@ Metal 2024). We select the instruction-tunee variant, as its prompt-following optimization critical for reliably eliciting the CoT reasoning traces for our analysis. This model is then mod- ified with our trained transcoders as described Section 3 We evaluate performance on our three datasets: Synthetic (Boolean), Synthetic (Arithmetic), and the annotated GSM8K benchmark.
Baselines. We compare CRV against two categories of baselines. First, black- box methods that use the final logit distribution: Maximum Softmax Probability (MaxProb), Perplexity (PPL., Entropy, Temperature Scaling (Temp. Scaling; (hhhh et al. box methods that operate on internal states. This includes trajectory-based methods that analyze hidden state dynamics across layers, such as Chain-of-Embedding (with its real-space CoE-R and complex-space CoE-C variants; Wang et al. 2025a) and CoT-Kinetics (Bi et al. 2025), as well as a standard logistic regression probe (LR Probe) and a two-layer multilayer perceptron probe (MLP 4
2023). and Energy (Liu al. 2020). Second, gray-
Probe) trained on the step's ' average hidden state While CoE and CoT- were originally designed for full CoT evaluation, they prove to be strong step-level baselines. All implementation details are deferred to Appendix
Evaluation Metrics. We use AUROC, 995, and AUPR to evaluate verifier performance. As our goal the detection of reasoning failures, we treat the incorrect label as the positive class for all metric calculations. AUROC assesses how well the method ranks correct versus incorrect steps across thresholds. AUPR captures the precision-recall trade-off for the positive (incorrect) class. @95 measures the false positive rate when 95% of positives are correctly identified, reflecting reliability under strict conditions; a lower score indicates the verifier can detect most errors with minimal false alarm. Together, these metrics provide complementary views of performance. Precise mathematical formulations and implementation details for all metrics are provided in Appendix C33
We also evaluated last-token probe, but found that using the average representation yielded slightly hetter nerformance.
5

---

Published as a conference paper at ICLR 2026
Table 1: Verification performance. Arrows indicate preferred direction (1 higher is better, lower is better). Best and second-best results are highlighted for each metric. The low AUPR on the Boolean dataset reflects extreme label imbalance, with the incorrect label only 0.2% (Appendix A.5).
Paradigm
Method
Synthefic (Boolean)
Synthetic (Arithmetic)
GSMBK
AUROC
AUPR
AUROC
AUPR
IPR
AUROC
AUPR
FPR
MaxProb
58.81
54.91
Black-Box
PFL
Eniropy
53.56
57.37
0.24
0.29
97.55
91.02
60.19
60.03
1.52
1.68
85.40
85.52
53.46
56.67
7.29
8.12
87.08
90.69
Temp. Scaling
58.77
0.36
91.41
59.67
1.56
86.96
54.42
8.24
92.28
Energy
51.08
0.28
95.11
76.45
5.59
73.86
62.55
9.11
86.34
Gra--Bx
CoE-R
53.17
0.33
92.85
58.47
1.93
76.68
8.34
96.20
CoE-C
$1.03
0.38
92.07
69.39
3.03
63.33
53.57
10.80
96.33
CoI-Kinctics
53.62
0.24
60.83
85.09
56.54
7.35
86.83
LR Prohe
$2.91
0.25
88.42
54.22
1.50
91.90
55.86
7.99
90.32
MLP Probe
53.63
0.26
88.56
54.41
1.30
90.98
56.02
8.63
93.94
White-Bes Box CRV (Ours)
75.87
0.97
79.17
92.47
28.92
37.09
70.17
14.3
79.61
4.2
VERIFICATION PERFORMANCE AND ROBUSTNESS (RQ1)
We first address RQI by evaluating CRV against all baselines on the task of reasoning step verifica- tion and then probing its robustness under more challenging conditions.
Main Verification Performance. The results, presented in Table [ provide strong empirical sup- port for our central hypothesis: that the structural signatures present in a reasoning step's compu- tational trace contain directly verifiable signal of its correctness. CRV consistently outperforms all black-box and gray-box baselines across every dataset and metric. The strength of this structural signal is particularly evident on the synthetic datasets. On the Arithmetic task, for instance, CRV achieves an AUROC of 92.47, significant leap over the strongest baseline score of 76.45. This advantage in reliability is further underscored by the FPR @95, where CRV reduces the false posi- tive rate to 37.09% from the baseline's 63.33%. The performance gap most pronounced on these structured, synthetic datasets. We hypothesize that the structured nature of algorithmic reasoning in- duces highly consistent execution traces for valid solutions. Consequently, the structural signatures of error manifest more uniform deviations from this baseline, rendering them highly detectable.
Analysis of Cross-Domain Generalization. A key difference between CRV and most base- lines is that its diagnostic classifier requires training. A critical question, therefore, is whether CRV learns domain- correla- tions or more fundamental, generalizable signa- tures of flawed reasoning. To test this, we con- duct comprehensive doos-ddmain evaluation. We train a CRV classifier on each of our three datasets individually and evaluate its zero-shot performance on the other two unseen tasks. We also evaluate model trained on the union of all training sets to test if a single classifier can learn diverse error patterns simultaneously.
Table 2:
Cross-domain generalization perfor- mance. For each test dataset, we compare the strongest baseline (based on AUROC) against CRV trained in-domain and out-of-domain. Best out-of-domain results are highlighted.
Test Set
Method (Train Set)
Metries
AUROC
AUPR
FPR@95
Baseline (MaxProb)
58.81
0.34
95.20
(GSM8K)
Boolean
CRV (Arithmetie)
61.58
0.51
87.55
CRV (Combined)
65.18
0.65
80.15
CRV (Bootean)
75.87
0.97
79.17
Baseline (Energy)
76.45
5.59
73.86
(GSM8K)
91.91
Table 2 shows that CRV's zero-shot transfer is limited, suggesting that error fingerprints are highly task-specific. For example, CRV trained solely on Arithmetic achieves an AUROC of 57.04 GSM8K, falling short of the Energy baseline. This reveals that errors different reasoning tasks (e.g., boolean logic vs. natural language arithmetic) produce distinct structural patterns in the computational graph.
Arithmetic
CRV (Boolcan)
69.59
2.64
72.87
CRV (Combined)
90.51
30.57
44.23
CRV (Arithmetic)
92.47
28.92
37.09
Baseline (Energy)
62.55
9.11
86.34
GSMBK
(Boolean)
44.37
6.33
95.71
CRV (Arithmetie)
$7.04
7.85
94.37
CRV (Combined)
70.62
12.66
80.22
CRV (GSMAK)
70.17
14.3
79.61
However, this domain specificity does not preclude the creation of a robust, multi-task verifier. As shown in the CRV (Combined) rows, when the classifier exposed diverse signatures during

---

Published as a conference paper at ICLR 2026
AUROC
AUPR
FPR
17.5
15.0
Fibis Pobiiv Hale (16)
12.5
80
AUROC
70
7.5
70
5.0
2.5
n=10
1=10
n=5
Number of Operators
Number of Operanors
Number of Opernnors
Performance of the step correctness predictors on the synthetic arithmetic task function Figure 2:
of difficulty (number of operators). CRV retains a clear advantage as complexity increases.
training, performance rebounds. On the Arithmetic task, the combined model achieves an AUROC of 90.51, nearly matching the specialist in in-domain model (92.47) and outperforming baselines. Re- markably, on GSM8K, the combined model (70.62 AUROC) slightly outperforms the in-domain specialist (70.17). This suggests that while the structural boundaries of error are distinct across tasks, they not incompatible; single classifier can learn the union of these complex "failure geometries", and in the case of GSM8K, potentially benefit from the inductive bias provided by synthetic logical tasks.
Performance Under Increasing Difficulty. To further probe CRV's robustness, we analyze its performance on the synthetic arithmetic dataset as a function of problem complexity, controlled by the number of operators (n E {5, 7, 10)) Figure 2 plots the performance of CRV against key baselines across these difficulty levels. While most methods show stable AUROC and FPR@95 @95, CRV maintains consistent advantage across all difficulty levels. AUPR generally improves for all methods as difficulty rises because harder problems increase the proportion of incorrect examples (a condition which AUPR is sensitive). Importantly, CRV's advantage persists despite these shifts, highlighting the robustness of its structural signals across task difficulty and class balance.
4.3 MECHANISTIC ANALYSIS OF ERROR COMPUTATIONS (RQ2)
Having demonstrated CRV's predictive power, we now turn to its key advantage: interpretability. To address RQ2, dissect our graph representation identify structural "fingerprints" of error, from high-level feature ablation to fine-grained analysis of the most predictive structures.
Ablation of Feature Families. leave-one- out ablation study on the Synthetic (Arithmetic) dataset reveals a clear hierarchy of feature im- portance, summarized in Table 3 The Node Influence & Activation features are demonstra- bly the most critical; their removal causes the most performance degradation across all met- rics, most notably increasing @95 by over 12 points. The Global Graph Statistics also provide substantial contribution. Interest-
Table 3: Leave-one-out ablation study on the Syn- thetic (Arithmetic) dataset.
Arithmetic
Feature Set
AUROCT
AUPR
1PR045
CRV (All three familles)
92.47
28.92
37.09
Ablation:
wlo Global Stats
89.62
24.35
44.54
w/o Node Stats
88.31
23.25
49.07
Topological Stats
90.89
26.83
39.19
ingly, the Topological & Path Bath- features appear least critical for this specific task, suggesting that the state of key local features is a more dominant signal than the holistic graph structure. Nevertheless, the full CRV model, which inte- grates all three signal types, is required to achieve optimal verification performance.
To provide qualitative evidence for our hypoth- esis, we visualize the "structural fingerprints" learned by our classifier. Figure ssows distributions of five highly predictive features for correct versus incorrect GSM8K reasoning steps. Across di- verse feature types, from graph topology (c.g., Graph Density) to node statistics (e.g., Total Active We exclude as the model's high accuracy yields few incorrect examples for reliable evaluation.
Visualizing the Structural Signatures of Error:
7

---

Published as a conference paper at ICLR 2026
Actre
Moce Nedo staeee
Franet huru ze Rota Crant
More DAe Wereeo
awiide audide
100 312
05 IN 150
Figure 4: Topological Fingerprints of Error on GSM8K. Distributions of five selected graph features for correct (blue) vs. incorrect (red) reasoning steps. The visual separation statistically significant for each feature shown (independent t-test, p < 0.001) and represents a medium-to-lare effect size (Cohen's d). This provides quantitative evidence that attribution graphs contain a clear, separable structural signal of a computation's 's integrity.
Features), distributions are clearly distinct. Similar patterns are observed on our synthetic datasets (see included in Appendix [C55) confirming that the graph representation captures separable struc- tural differences between valid and flawed computations.
While individual features are predictive, CRV's strength lies in their combination. To illus- trate this, we project the full high-dimensional feature vec- tors into two dimensions via Principal Component Analysis (PCA). Figure reveals that in- correct steps form a dense sub- set within the broader distribu- tion of correct steps. Crucially, correct steps also occupy dis- tinct region not shared by incorrect computations. This suggests many reasoning failures are com- putational "near misses", structurally similar to valid steps, which explains the overlap. Yet there exists a zone of computational integrity, region defined by structural properties accessible only to correct reasoning. complementary t-SNE visualization in Appendix Ch5wshows the same structure, confirming that CRV succeeds by learning the complex boundary of this zone rather than separating two simple clusters.
(a) Boolean
(b) Arithmetic
(c) GSM8K
Figure 3: Distributions of features after PCA for correct (blue)
vs. incorrect (red) reasoning steps.
4.4 FROM DIAGNOSIS TO CAUSAL INTERVENTION (RQ3)
Finally, we conduct an exploratory study to test whether CRV's diagnostic insights can guide pre- scriptive interventions. A key advantage of our white-box approach is that failure-predictive graph features can be traced to specific components of the model's computation. We illustrate this with case study on the arithmetic task, correcting an order-of-operationn eтrоr.
Given the expression (7* (5+9) +7) ), the model incorrectly computed 7+14-98 in its second step (Table 4.. CRV flagged this step as incorrect, and feature importance highlighted an unusually high activation of a late-layer transcoder feature. Tracing the signal back to the graph, we found a single highly active last-layer transcoder feature (ID 918 14) strongly associated with multiplication.
We hypothesized that this premature activation caused the error, and performed a direct intervention. We re-ran the generation up to the point of failure and used forward hook to manually clamp the activation of this specific multiplication feature to zero. The effect was immediate: as shown in the right column of Table 4 with the premature multiply impulse suppressed, the model correctly generated the next step 1 4 + 7-21 and proceeded to the correct final answer.
While not a general-purposs solution, this proof-of-concept is crucial step. Beyond suppressing faulty features, we also corrected an error by amplifying an under-active feature (Appendix C.5). The success of both interventions provides closed-loop evidence that CRV's structural signatures are causally implicated in errors, opening a promising direction for targeted model interventions. Crucially, because these interventions rely on the semantic interpretation of specific transcoder fea-
8

---

Published as a conference paper at ICLR 2026
Table 4: Side-by-side comparison of a reasoning trace before and after causal intervention. The highlight indicates the point of divergence where suppressing a single multiplication transcoder feature corrects the model's computational path.
Before Intervention (Incorrect)
After Intervention (Correct)
Evaluate the arithmetic expression below.
Evaluate the arithmetic expression below.
(7 * ((5+ 9)+ )))
(7 ( 5 9 + )
To evaluate this expression, we need to follow
To evaluate this expression, we need to follow
the order of operations (PEMDAS):
the order of operations (PEMDAS):
1. Evaluate the expression inside the innermost
1. Evaluate the expression inside the innermost
parentheses: 5 4 9= 14
parentheses: 5 9= 14
2. Multiply 7 by the result: 7 * 14 98
2. Add 7 to the result: 14 + 7
3. Add 7 to the result: 98 + 7 = 105
3. Multiply by the result 7 * 21 = 147
Therefore, the expression evaluates to 105.
Therefore, the value the expression is 147.
tures (e.g., identifying a missing "multiplication" concept), they demonstrate the potential of CRV to serve as a foundation for interpretable debugging, moving beyond simple error flagging to actionable model repair.
RELATED WORK
CoT Prompting for Improved Reasoning. Chain-of-Thought (CoT) prompting was introduced by Wei et al. (2022) to elicit more complex reasoning from LLMs through few-shot examples that demonstrate step-by-step problem solving. Subsequent work has expanded this idea in several direc- tions. Kojima et al. (2022) showed that
simple zero-shot instruction such as "Let's think step by step" can trigger coherent reasoning traces. While this reduces the need for handcrafted prompts, providing structured examples often remains beneficial. To scale this process, recent studies gener- ate CoT exemplars synthetically (Zhang et al. 2023) Shao et al. [2023] Li et al. 2025a). Other work leverages test-time compute scaling extend reasoning chains, enabling longer and more elaborate solutions (Snell et al. (2024). For comprehensive surveys of CoT techniques and their applications, see Chu al. (2024) and Chen et al. 2025a).
Verifying and Improving CoT Reasoning. The transparency of CoT has also made it a focal point for research into model interpretability and reliability. While some work assumes reasoning traces some extent faithful representations of the model's internal process (Wei Jie et al. 2024 Korbak et al. 2025), a significant body of evidence highlights their unreliability (Arcuschin et al. 2025 Bentham et al. 2024; Chen et al.) 2025b; Turpin et al. (002).. This has spurred a rich field of research dedicated to veritying and improving COT traces. This research broadly investigates (i) the model's ' intrinsic ability to self-evaluate its reasoning steps (Zhang al. (025). (ii) how measure
the faithfulness of reasoning chain to the final answer (Lanham et al. 2023] Bi et al. 2025 Tutek al. 2025), and (iii) when reasoning steps are needed or useful (Bogdan et al. 2025 Wang al. 2025b). A parallel line of work aims to improve reasoning chains through various forms of neuro- symbolic reasoning (Lyu et al.) 2023), correction (Tyen et al. 2024), uncertainty calibration [Ji cet al. 2025), or by enforcing internal consistency (Xie et al. 2025 Wang et al. 2025a). A distinct approach involves training auxiliary models, such as Process Reward Models (PRMs), to assess step-level correctness and guide post-training (Lightman et al. 2024 Wang et al.
2024 Guan al.)
2025). While all these methods aim to improve reasoning outcomes, they primarily operate on the textual or hidden state representations. We are not aware of previous attempts verify reasoning by analyzing the structural properties of its underlying computational graph.
Mechanistic Interpretabilityy of CoT Reasoning. Our work is most directly situated within the field of mechanistic interpretabiliity which seeks to reverse-engineer the algorithms learned by neu- ral networks, moving beyond the surface-level analysis of CoT traces (Wei Jie et al. 2024, Korbak et al. 20255 Baker et al. 2025). A central tenet of this field is that models develop specialized sub- graphs, or circuits, to perform specific computations (Olah et al. 2020). Recent work has begun to apply this lens reasoning, providing evidence that CoT traces are driven by latent learned algo-

---

Published as a conference paper at ICLR 2026
2025) and even characterizing specific self-verification circuits in toy models (Lee et al. 2023). Other research aims to improve performance by eliciting or steering behavioral eircuits (Zhao et al. 2025: Ward et al. 2025). A particularly powerful and increasingly popular tool in this area is the use of sparse autoencoders (SAEs), which learn to decompose a model's 's dense ac- tivation vectors into sparse basis of interpretable features (Bricken al. 2023) Cunningham al.) 2023). Our work builds directly on a variant, the transcoder (Dunefsky et al. (025), which acts as functional, interpretable substitute for an MLP module. While prior work has used transcoder-basee attribution graphs to qualitatively analyze the faithfulness of CoT reasoning (Ameisen et al.) 2025), our work the first to operationalize this approach for automated verification. We move beyond visual inspection by systematically extracting quantitative, structural features from these graphs and demonstrating that they can be used to diagnose computational failures.
rithms (Venhoff et al.
CONCLUSION
In this work, we introduced CRV, a white-box methodology for studying the computational struc- ture of reasoning failures. By treating attribution graphs as execution traces of latent circuits, we showed that correct and incorrect reasoning leave distinct structural fingerprints. CRV revealed that these error signatures not only enable accurate verification but are also domain-specific, with fail- ures in different reasoning tasks manifesting as distinct patterns. Moreover, targeted interventions on transcoder features demonstrated that these signatures are causally implicated, allowing us correct faulty reasoning. Together, these findings establish CRV as a proof-of-concept for mechanistic anal- ysis, showing that shifting from opaque activations to interpretable computational structure enables a causal understanding of how and why LLMs fail to reason correctly.
ETHICS STATEMENT
Our research yields insights into success and failure patterns in LLM reasoning. Such knowledge could theoretically be used for malicious purposes, such as designing adversarial attacks engineer- ing more subtle, undetectable reasoning failures. However, the computationally intensive nature of CRV, which also requires deep expertise and white-box model access, positions it tool for deep scientific analysis rather than a scalable method for generating exploits. The primary and intended application of our work is defensive: by providing scientific instrument for developers to diagnose why model fails, we aim to accelerate the development of more robust, reliable, and safer Al sys- tems. We believe the benefits of enabling deeper, causal understanding of Al failures for safety and alignment research significantly outweigh the risks of misuse.
REPRODUCIBILITY STATEMENT
We are committed to the reproducibility of our work. Our newly generated datasets with step-level labels and our trained transcoders are publicly available. We provide details setup in Section 4.1 Comprehensive details are provided throughout the Appendix, including: our
experimental
dataset construction, annotation prompts, and full data statistics (Appendices [A.1] A.2] and A.5
our transcoder training procedure (Appendix [); the attribution graph computation (Appendix B.2 and all classifier and baseline configurations (Appendix
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their valuable feedback. We are grateful to Edan Toledo and Karen Hambardzumyan for their constructive discussions and insightful feedback on this project. We also thank Shuangrui Ding and Yunzhen Feng for their helpful input, and Megan Ung for assistance with setting up the computing environment.
REFERENCES
Anum Afzal, Florian Matthes, Gal Chechik, and Yftah Ziser. Knowing before saying: LLM rep- resentations encode information about cfain-of-thought success before completion. In Wanx-
10

---

Published as a conference paper at ICLR 2026
ang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Find- ings of the Association for Computational Linguistics: ACL 2025, pp. 12791-12800, Vi- enna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-256-
5. doi: 10. 2011663//1/2225 http:////.......nnnnanao......
findings-acl.6622 URL. https: /tap://aclaann aacclannhooogy oggpoo/000.rr///5.....//
Findings-ach 6627
Ae@Meta. Llama 3 model card. 2024.
URL https://github. htt.ggguuuu.coomtaaaaala
aaama3/blobmmalt ma n/ /MODELL CARD ..DD.mo. md
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner. Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URLphttps .ttll
Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild is not always faithful. In Workshop on Reasoning and Planning for Large Language Models, 2025. URL htttps://openrett
net// Soet/Eorummid==L00440 1dm.mmmm21dd0889
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Guan, Aleksander Madry, Wojeiech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation, 2025. hRll:cttss//arxii.oor///////////
Oliver Bentham, Nathan Stringham, and Ana Marasovic. Chain-of-thought unfaithfulness as dis- guised accuracy. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
net/ fereet/oooumiii wow.mmdddydcrrrraeeer
Reproducibility Certification.
Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang. Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. Cot-kinetics: A theoretical modeling assessing Irm reasoning process, 2025. URL https: /ttp///aavvv/v.///ev//eeeeeaaeva org/ abs 2505 13408
Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which Ilm rea- soning steps matter?, 2025. URL htttts:
org/ abs 2506 19143
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, et al. Towards monosemanticity: Decomposing language models with dictionary learn- ing. Transformer Circuits Thread, 2023. features/index.htm..
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain- of-thought for reasoning large language models, 2025a. URL htt.ps://exxi..rrg/////////////e
(2503.00 095677
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't 't always say what they think, 2025b. pattts::/aaxine
abs////ss//55 2505. 05410)
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth survey of chain of thought reasoning: Advances, frontiers and future. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1173-1203, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/253//l/222444alllloooo httpe//org/222222 .e...acl-l1 acl- 657
URL
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
prr//abs/2100.. 14168
11

---

Published as a conference paper at ICLR 2026
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models, 2023. URL https://arxiv: org/abs/ 2309. 08600
DeepSeek-AI al. Deepseek-rl: Incentivizing reasoning capbiiliyy in Ilms via reinforcement learn- ing, 2025. URL httpt:://taxiiiieee wttpaa/wwww..r.a///........ 12948
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable IIm feature circuits. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025, Curran Associates Inc. ISBN 9798331314385
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac HaffieldDoddss, Danny Hernandez, Andy Jones, Jackson Keriinn, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer cetteffoormerrrrutsssso circuits. wwrcuits.pub//2221111
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations 2025. URL
net./
forum? d=t.csst 9 9ZNKD.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small LLMs can master math reasoning with self-evolved deep thinking. In Forty- second International Conference Machine Learning, 2025. htLhttps:: http:::ttp/s/eeeetl //tp:rev///ppeevveee
fnet/oorumiid-52
FGizaa
Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen.
circuit-tracer. com http://attttessereccciiiooo wwtprrcccccccitttttrreeee 2025. The first two
authors contributed equally and are listed alphabetically.
Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roce Aharoni, and Mor Geva. A thain-of-thought is as strong as its weak- link: A benchmark for verifiers of reasoning chains. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 4615 4634, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.1 18653////222..clloongg 254. URL org. 2004.22244.00 /ettps:/aclanthol./ 20lo:/oorroor//2222..l1l----- 2547]
Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. Calibrating verbal uncertainty as a linear feature to reduce hallucinations, 2025.
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Saes (usu- ally) transfer between base and chat models. Alignment Forum, 2024.
URL
https: wwwww//... http:://wwwwwwgnmeetttcece hwwp:/etttouuuoorg///oooo
fmwk 6qgppp/d4d saes usualull 11y wtansrrrannfe-eeeee wewteeerrbbwwennn..e caneeeenn--aeeee base and models
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817138710SS
Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, et al. Chain of thought monitorability: A new and fragile opportunity for ai safety, 2025. URL wttpthtt///// httppss////axvvva
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, et al. Mcasuring faithfulness in chain-of-thought reasoning, 2023. URL http::ttt///tm 10033///LLLppp///rr/v/.r///////
13702
Andrew Lee, Lihao Sun, Chris Wendler, Fernanda Viégas, and Martin Wattenberg. The geometry of self-verification in a task-specific reasoning model, 2025. URL https: // /arxiv...... org/ abs/ 2504 14379
12

---

Published as a conference paper at ICLR 2026
Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured thain-of-thougee prompting for code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1 2025a.
Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang. Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. When thinking fails: The pitfalls of reasoning for instruction- following in Ilms, 2025b. URL httptt:ttsa.iiiiiiee /ttp:///axvvvvoo.o abs 1.555.111233
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. pp. 278-300, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 1 8653/v1/224.. blackboxnlp- 19. URL wwwp:tttt////
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL (Rtths:: /ttt:::://openevieer
net feet/Eormmlle 1ieerummdddd88000
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas I.. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL http:://ttaassfoomrr-rrlatt pub/ 20l//2255/a0 http:/atttuuttonnngttt bionooy.iioo.....ooooo htmll
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch Faithful ofain-of-thought reasoning. In Proceedings of the 13th In- ternational Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguisties (Volume I: Long Papers), pp. 305-329, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: (0.23915/dissilll.0004..00
OpenAl et al. Openai ol system card, 2024. URL
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: generating chain-of thhtuught demonstrations for large language models. In Proceed- ings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Long horizon temperature scaling, 2023. URL
https: //arxiv.//riiv////a or abs/ 2202. 03686
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute comppti can be more effective than scaling model parameters, 2024. URL httts:://aaxii.....
abs 0oo/24088 24082408..00 03314
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't 't always say what they think: unfaithful explanations in chain-of- prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
13

---

Published as a conference paper at ICLR 2026
Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, and Yonatan Belinkov. Measuring chain of thought faithfulness by unlearning reasoning steps, 2025. URL https: /arp:s:///axv... arxxi..
org/ ht///b///5/ 2502. 14829
Gladys Tyen, Hassan Mansoor, Victor Cärbune, Yuanzhu Peter Chen, and Tony Mak. Lims cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics ACL 2024, pp. 11894-13900 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Base models know how to reason, thinking models learn when, 2025. wtt.httsss///rxii...ot
07364
Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhi- fang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v1/ 2024.acl-long.5 100. URL / aalaaaclanntttt howgy.ooooyy.oooooo..o.o org/ 2024 acl -on--long--. 510/
Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. Latent space chain-of- embedding enables output-free LLM self-evaluation. In The Thirteenth International Confer- ence on Learning Representations, 2025a. httph: httt:::://opnnevieeww net.swiiwwwweeeeeeeee.e.ee wwt.://wwwtttttrmm/// ffrummrrummmii
jx070B9EQS
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Yufei Wang, Liangyou Li, Yasheng Wang. Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. Chain-of-probe: Examining the necessity and accuracy of CoT step-by-step. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 2586-2606 Albuquerque, New Mcxico, April 2025b. Association for Computational Linguistics. ISBN 979-8-89176-199 URL
7. doi: 2..18653/v1/022 http://222.....ss.....a
Jake Ward, Chuqiao Lin, Constantin Venhoff, and Neel Nanda. Reesoonnng-fnntuuinnggrruupppppp la- tent representations in base models, 2025. BtRKl//tps///aa.......o/////////
Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao. Percy Liang. Christopher Re, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URI.tttps::/ opw.reviee.eeeeeeeeeeee 2 net.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. -fain-of-thouuht prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS 22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Yeo Wei Jie, Ranjan Satapathy, Rick Goh, and Erik Cambria. How interpretable are reason- ing explanations from prompting large language models? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2148-2164 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.118653/ www.diinnnngnnncc..... 13.. URL. https: /ectpa://acler anthellltthlloooooooooooo oggp://ooooor//22////// Findings- 138/
Zhihui Xie, Jizhou Guo, Tong Yu, and Shuai Li. Calibrating reasoning in language models with internal consistency. In Proceedings of the 38th International Conference on Neural Informa- tion Processing Systems, NIPS '24, Red Hook, NY, USA, 2025, Curran Associates Inc. ISBN 9798331314385.
14

---

Published as a conference paper at ICLR 2026
Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan. Ujjwal Karn, Rui Hou, Madian Khabsa, and Yuning Mao. Diversity-driven data selection for language model tuning through sparse autoencoder, 2025. BBLLttts////axx..rrg abs wttp:o//raab//...1100..0..
Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reason- ing models know when they're right: Probing hidden states for self-verification. In Second Conference on Language Modeling. 2025. URL https:://////.... https:://penzeeiiett net.leewweeeeeeeeeeeeeeeeeeeee.e wtwp::/umteeeoommmmmimmmm forummidrmm 0610Av7683
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models, The Eleventh International Conference on Learning Representations, 2023. URL htttttp:///peneeiieeee
net wetet/oorrmnnnnnnCttee
UHKK
Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du Aryo Pradipta Gema, Hongru Wang, Xu- anli He, Kam-Fai Wong, and Pasquale Minervini. Steering knowledge selection behaviours in LLMs via SAE-based representation engineering. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume Long Pupers), pp. 5117-5136, Albuquerque, New Mexico, April 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-189-6, doi: 18.18653/v 1111155/112222.... hww.:00555aaaa-lnn.2.66.. URL
naw.i-aaacllloo 2647
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMMSSSchat-IIm large-scale real-world LLM conversation dataset. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL //tp:://ppoe /tttps://openreiiie net./
Forum?id=BOFDKKe
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS ON DATASETS
Here we provide a detailed description of our dataset construction, our labeling protocol, and the final dataset statistics.
A.1 SYNTHETIC DATASET CONSTRUCTION
To create a controlled environment for studying reasoning failures, we procedurally generated two synthetic datasets: Boolean and Arithmetic. For each, we first generated ground-truth expression, then prompted our base model (Llama 3.1 8B Instruct) to produce Chain-of-Thouge solution towards solving the expression. We provide the prompt template used to generate CoT in Table 5 Once the CoT is generated, we split them into steps using regular expression.
Table 5: Prompts used for CoT generation across the three datasets. Placeholders for dynamic content are shown in italics.
Datasel
Llama 3.1 8B Instruct Prompt Template
Boolean
Deginnof.texx x startt. seaattthedderidd >aystems (end_header.idd>
Evaluate the boolcan expression below.
dleot_id]>ss at ttadder.dddddd huaers] hen.eeeeeddddeeeeeeee...
(boolean espression)
<le id]><] heades id hessistant < end heades id
Arithmetic
<bbegnnof.Lee. |ttlstart. wel.arr.eeeeerr.. >aysLems hdd
Evaluate the arithmetic expression below.
starttttheeererrrrrr >user< lend_header_idl> (orithmetic expression)
<I cot.idd /t/sstttt.ttttt. www.r.hhaader.i..... baasistant hnddaterrdddddadddiiiddd...
GSM8K
< <lbegin.of.teen
htw.:attttmm/edd..eeeee...dldl (nn.heacer..dd
Solve following math problem step by step.
>user< end_heeder.id]>
[gmmSk.peettin)
Expression Generation. The complexity of the expressions was controlled by the number of oper- ators, n E (3,5, 7, 10). For the Arithmetic task, we generated expressions with single-digit integers and the operators {+,-, x For the Boolean task, operands were simply {True, Fal False). and possible operators are (and, or, not]. We generated 10,000 unique expressions for each difficulty level, with the exception of the Boolean task 3, where the number of unique expressions is inherently limited. 6 examples from these datasets with their associated CoTs generated by Llama 3.1 8B Instruct model.
A.2 LABELING PROTOCOL
Our primary goal was to create labels of the highest possible quality. For the synthetic datasets, we developed a rigorous two swo-pronged strategy that leverages both a powerful LLM judge and direct programmatic verification, keeping only the annotations where both methods agree.
Method 1: LLM-as-a-Judge. Our first annotation method uses powerful, external LLM Llama 3.3 70B Instruct as an expert judge. For each reasoning step, the judge model was provided with the full problem context and the generated step, and was prompted to output binary correct- ness label and a justification. This method was used for all three datasets including the GSM8K dataset. We provide the prompts used for the judge in Table (Boolean). Table g (Arithmetic), and Table (GSM8K)
Method 2: Programmatic State Verification (Synthetic Datasets only). For the synthetic tasks, developed programmatic method to verify the logical integrity of the entire reasoning trace.
16

---

Published as a conference paper at ICLR 2026
Table 6: Examples of generated expressions and the corresponding CoT generated by our base model for the synthetic datasets.
Task
Example
Boolean (n 3)
Expression: (( (T:ue oz Truel and (True and True)) oz ICrue and Palse):
Generated Col:
To ovaluate the boolean expression. need follow order operations (PEMDAS):
1. Evaluste the expressions inside the innermost parentheses:
(Izue or Tzue) True
tirue and Tree)
2. Now the expression becomes: (True and Trie) or (True Folse)
3. Evalunte the expressions inside parentheses:
(True and Tria) True
False)
4. Now the expression hecomes: Tzue or False
Evaluate the final expression:
True or False Tcue
The final answer Tzue.
Arithmetic (n = 7)
Expression: ((1 (- + (-6)) (
Generated CoT: To evaluate this expression, we need follow the order of operations (PEMDAS):
Evaluate expressions
innermost parenibeses:
(-3) (-6)
2. Multiply 9 amd 6:
+9 6 54
3. Multiply 99 and 54:
54 -486
4. Add 44 486:
9-886 (-4) - 4490
The final answer -490.
This goes beyond simply checking the correctness of single calculation. After each reasoning step i generated by the model, we use a specially crafted prompt to ask the model to output the current, simplified state of the original expression.
For example, the original expression (3+5) +2 and the model's first step is 1. 3+5=8, we then prompt it: "Now the original expression becomes: ". The model is expected to return the reduced expression 8.2. We then programmatically evaluate the ground-truth value of both the original expression and this new, simplified expression returned by the model. A step is labeled 'correct' only if the two values are identical. If at any point the value of the simplified expression diverges from the ground-truth value of the original, that step is labeled 'incorrect'. While occa- sionally the model outputs a reduced expression which evaluates to the same value despite being incorrect, this method filters significant amount of errors.
Final Label Agreement. To create our final, the intersection of the labels from both methods. That is, reasoning step was only included in final dataset both the I.LM-as-a-Judge and the programmatic verifier agreed on its label. This strict agreement protocol ensures exceptionally clean dataset by filtering out ambiguous cases or potential errors from either annotation method
label set for the synthetic tasks, we took
A.3 HUMAN VALIDATION LLM- JUUDGE LABELS
To validate the quality of our LLM-as-a-Judh annotation pipeline, a subset of 100 randomly sam- pled Boolean and Arithmetic expressions (=/00 steps) was independently annotated by four authors. Each annotator labeled half of the set, with every step covered by at least annotators. To mitigate the rarity of incorrect steps, we upsampled the positive class. Because of the extreme class imbal-
While this significantly increases our confidence in the label quality, it also has the effect of making the class distribution imbalanced, ambiguous incorrect cases more likely to be filtered
17

---

Published as a conference paper at ICLR 2026
Table 7: Prompt used for step-level annotation by the Llama 3.3 70B Instruct judge model on the Synthetic Boolean dataset. Placeholders for dynamic content are shown in italics.
Llama 3.3 Instruct Prompt Template
<I lbbeiinoo ftxett I sttart. Staar.hesdder.1 b Peystems end.bessder.dd
expert in logical reasoning boolean algebra. evaluate
reasoning steps in boolcan expression
evaluation with high precision.
dot_id d I start headar,ie >userk end_haader__ddi>
Evaluate this reasoning step for logical correctness:
Original Boolean Expression: (ariginal.sspressing expression)
Correct Truth Value: (correctavllee)
Context (previous steps): {contexr)
Stop to ovaluate: {step]
Evaluation criteria:
Is the boolean operation upplied correctly?
Does slep follow proper onler of operations?
truth values computed accurately?
Is the reasoning logically sound!?
Respond with exacily one the following:
CORRECT: if the logically sound and mathernaticlll
INCORRECT: the step contains logical emors, mathematical mistakes, or invalid reasoning
Your response should start with either "CORRECT" or "INCORRECT" fallowed by brief explanation.
eot_id [>-] starttttttt.... itt.errtteeeeriddiieeeo.
Table 8: Prompt used for step-level annotation by the Llama 3.3 70B Instruct judge model on the Synthetic Arithmetic dataset. Placeholders for dynamic content are shown in italics.
Llama Instruct Prompt Template
<lbegin_offfr |><]
attart_header__ aalllaayaa payatams, headdrrrddrrd
expert mathematical reasoning arithmetic operations.
comectness reasoming steps arithmetic
expression evaluation with high precision.
<I cot >>< start www.hhr.idddd.......... >user< end.header.idi
Evaluate reasoning step for mathematical
Original Arithnetic Expression: [priginullesprressionn)
Correct Value: (cornee volue)
Context (previous steps): (contexr)
Step to evaluate: [step]
Evaluation critcria:
operations upplied correctly?
Does the step follow proper order of uperations (PEMDAS/BODMAS?
numerical computations accurate?
Is the mathematical reasoning sound?
Respond with exacily one of the following:
CORRECT: step mathematically sound computationally accurate
INCORRECT. the step contains mathematical errors, computational mistakes, invalid reasoning
Your response should start with either "CORRECT" or "INCORRECT" followed by a hrief explanation.
<I iet start wheader. idl passistant end.heederr_dd
ance, Cohen's Kappa (k) can underestimate agreement, report both and raw percentage agreement to give a fuller view inter-annotator reliability.
The results are summarized in Table 10 The agreement among human annotators was moderate as measured by Cohen's 's Kappa (x = 0.42) but high in simple agreement (87.3%). When compar- ing the consensus human labels to the LLM-as-at a--ge labels, found fair agreement by Kappa (K = 0.26) and similarly high simple agreement 84).%) A qualitative review of the disagreements revealed recurring pattern: the vast majority of discrepancies, both among humans and between hu-
18

---

Published as a conference paper at ICLR 2026
Table 9: Prompt used for step-level annotation by the Llama 3.3 70B Instruct judge model on the GSM8K dataset. Placeholders for dynamic content are shown in italics.
Llama 3.3 Instruct Prompt Template
<I -lbegin.of.tee fexttt | Sttart.hesder.e payst emc end.besderr.dl
expert in mathematical word problems and quantitative reasoning. Your purpose evaluate single reasoning step to solve a multi-stpp weed problem. You must be precise, focusing only on the provided step and its relationship to the problem and previously established
<I leot, dI ><X at art attttttteeeeeteeeeete wheader. _ >users annd_header _
Your task to evalunte the prowided reusoning step for logical and mathematical correctness.
Original Math Problem: boriginal-queztion;
Correct Final Answer: (correctvvauee
Context (previous steps): {contexr)
Step
{step}
Evaluation criteria:
Does the slep correctly extract and interpret information from the 'Original Problem' or the 'Context"?
using the right numbers for right concepts?
Is the cbesen mathematical operation (e.g., audition, subiraction) the correct one to achieve the step's goal, based on the narrative of
the 'Original Prohlem"
Is the arithmetic in the performed correctly?
Is mathematical reasoning sound?
Is step logically consistent with the problem and previous sleps?
The fallowing types of steps do not contain an error and must he classified as CORRECT:
simple. factually accurate restalement of
the problem or context.
A non substantive introductory or conversational phrase (e.g. "Let'' solve this step by step", "First, we need find...")
Respond with exactly one of the following:
CORRECT: the step mathematically sound and computationally accurate INCORRECT: the step contains mathematical errors, computational mistakes, invalid reasoning
Your response should start with either "CORRECT" or "INCOREEC"" followed by a brief explanation.
<lott_i idd start header id) bassistant hnddeeer header
mans and the LLM judge, occurred on steps that followed an initial reasoning error. This highlights inherent ambiguity of labeling steps on corrupted computational path and directly motivates strict truncation policy, detailed in the following section.
Table 10: Inter-Annotator Agreement (IAA) statistics for the human validation study. The com- parison shows moderate-to-high agreement, with lower Kappa scores reflecting the extreme class imbalance.
Comparison
Cohen's 's Kappa (x)
Agreement (%
Human vs. Human (Inter-Annotator)
0.42
87.3%
Human vs. LLM-as-a-Judge
0.26
84. %
A.4 HANDLING ERROR PROPAGATION
A key challenge in labeling CoT is handling steps that occur after the first mistake. A model can make a technically correct calculation based on a previously incorrect number or expression. This creates fundamental ambiguity: should step be judged on its local correctness or its global contribution to the final answer?
To illustrate this problem, consider the flawed CoT trace shown in Table 11 The model correctly evaluates the innermost parentheses in Step 1. However, in Step 2, it makes a critical structural error by incorrectly dropping parentheses, which alters the order of operations. The correct simplification in Step 4 should have been "not (False or True)". While subsequent steps like 5 and arithmetically valid given the new, flawed expression from Step 2. their ultimate correctness is undefined as they exist on a corrupted computational path. To avoid this ambiguity and create a clean, well-defined verification task, we adopted strict truncation policy. For any given reasoning
19

---

Published as a conference paper at ICLR 2026
Table 11: example flawed CoT trace demonstrating error propagation. The first unrecover- able error in Step renders the correctness of all subsequent steps ambiguous.
Original Expression: not sa
True and sa )
True Palse
Model's Generated Chain-of-Thoughhe To evaluate expressoon. need follow the order of operations (PEMDAS):
Evaluate the innermost parentheses: (True and False) False. So. (False or (True and False)]
se or Fa se Fa
Now, expression becomes: (not
(True False)) or oz False))
3. Evaluate the next innermost parentheses: (True er False) True. So, (not (False er (True and False)
Or (True or False)) (not False or True)
expression hecomes: (not
oz Trus)
5. Evaluate the NOT operator: not False True
6. Now, the expression becomes: True or True
7. Finally, evaluate OR operator: True or True Irue Therefore, the final result of the boolean expression
chain, we only retain the steps up to and including the first incorrect step. All subsequent steps are discarded from our dataset.
A.5 DATASET STATISTICS AND MODEL PERFORMANCE
For GSM8K, we used the official testset split. Since its reasoning steps are expressed in natural language and are not always programmatically verifiable, we relied solely on validated LLM-as- a-Judge pipeline for annotation. Table 12 provides the final statistics for all three datasets, including size and label distribution. Table [3 reports the base performance of our Llama 3.1 8B Instruct model on these tasks. For our synthetic datasets, we randomly split the data into 80% training and 20% testing for the subsequent classification task.
Table 12:
Final statistics of our curated datasets, showing the number of reasoning steps and the distribution of correct/incorret labels after our full annotation and filtering process.
Dataset
Total Steps
% Correct
% Incorrect
Synthetic (Boolean)
126,624
99.8%
0.2%
Synthetic (Arithmetic)
155,434
98.8%
1.2%
GSM8K
8,737
93.4%
6.6%
Table 13: End-to-end task accuracy of our base model (Llama 3.1 8B Instruct). For the synthetic datasets, we provide fine- aneegraaned breakdown by difficulty, controlled by the number of operators (n).
Dataset
Difficulty (Operators)
Final Answer Accuracy
n=3
98.4%
n=5
93.27%
Synthetic (Boolean)
n=7
89.4%
n=10
78.43%
n=3
94.83%
n=5
86.8%
Synthetic (Arithmetic)
n=7
73.07%
n=10
52.8%
GSM8K
75.82%
20

---

Published as a conference paper at ICLR 2026
B TRANSCODER TRAINING DETAILS
Our methodology relies on high-fidelity, sparsely activating transcoders to create an interpretable model. To this end, trained a TopK-Transcodee for each target MLP module in the Llama 3.1 8B Instruct model. Our training protocol is designed for robustness and follows several best practices established in recent literature.
The transcoders were trained on a high-quality, 10B token subset of the RedPajama- V2 dataset (We- ber et al. 2024). We pre-processed the entire training corpus by concatenating and chunking all passages into uniform length, and we explicitly discarded all beginning-of-sequenee (BOS) to- ken which we found be detrimental stable transcoder training. The transcoder architecture consists of a simple autoencoder with a single hidden layer and ReLU activation. For each MLP layer in the base model, the transcoder is trained to take the residual stream before the MLP block as input and reconstruct the residual stream after the MLP's computation. The input dimension matches the Llama 3.1 8B's MLP hidden dimension (4096), and the latent feature dimension was set to overcomplete basis of 131,072. We enforeed sparsity structurally using TopK mechanism, preserving only the & = 128 largest feature activations in the forward pass.
We followed several established training techniques to improve feature quality and avoid common pitfalls (Gao et al. 2025 Yang et al. (225). The decoder weights were normalized to have unit norm, and we did not tie the encoder and decoder weights. To prevent feature collapse, implemented a dead neuron revival mechanism: if a feature neuron had not activated in 10 million tokens, its activation was forced with auxiliary loss (coefficient of 1/32).
The transcoders were trained for 4 epochs using the AdamW optimizer. The learning rate was 7c-5 with a warmup ratio of 0.5. Training was conducted on nodes, each with 8 Nvidia H200 GPUs, using total batch size of 4,096. This was achieved with per-device batch size of 32 and gradient accumulation steps. We found that the training loss generally saturated after approximately 4,000 steps, indicating efficient convergence. We show the training loss selected layers in Figure 5
Loss
topk transcoder Top 128, layers, 31 Llam 3.1 6B instruct latents131012 total, Joss/layers.23
topk tropstrroodder-oo 128, layers 31-Llama 3.1 3.1-58 s8-instruct latents131012 total. layersa
molkttranscder-Top 128-128...8 Jayers 31-Llama 3.1-8B instrnnnsuutttiii (atunssilattt1111017 total. Josy/ayers.
topl-transcode-- Top-ddrr-oo--_____e Jayers_ 31-Llama-3.1 ----instruct-ltentsllll 31072 los/yarrs
14
12
10
2k
4k
6k
8k
10k
Figure 5: Transcoder Training Loss Curves. The x-axis represents training steps. In all cases, the loss converges efficiently, generally saturating after approximately 4,000 steps.
B.1 IMPACT OF TRAINING TRANSCODERS ON INSTRUCTION- DATA
Since our base LLM used is an instruct model, natural hypothesis is that transcoders fine-tuned on mnstruction-following data might learn features more relevant CoT reasoning, thereby improving verification performance. To test this, we trained an version of our transcoders with instruction- tuning (IT) data. Starting from our pre-trained base transcoders, we continued fine-tuning for epoch on the LMSYS- Sasys--hat-II 1M dataset (Zheng et al., 2024), using the same hyperparameters for
BBS tokens are retained when generating activations but their activations are removed afterward for train- ing the transcoders.
21

---

Published as a conference paper at ICLR 2026
Table 14: Performance comparison of CRV with Base transcoders vs. transcoders further trained on Instruction-Tunie Tuning (IT) data. Arrows indicate preferred direction (1 higher is better, + lower is better).
Transcoder Training
Synthetic (Booleun)
Synthetic (Arithmelic)
GSM8K
AUROC 1 AUPR
FPR@95
AUROC
AUPR
FPR@95
AUROC f
AUPR f
FPR@95
Buse
75.87
0.97
79.17
92.47
28.92
37.09
70.17
14.3
79.61
Data
76.04
1.20
66.82
91.39
28.44
38.47
72.01
15.40
83.27
the base transcoder training. Following the methodology Lieberum et al. (2024), we prepended and appended the Llama 3.1 8B Instruct model's ITT prefixes to the user queries and model responses respectively.
However, shown in Table 14 this additional training on IT data did not yield consistent or meaningful improvement verification performance on our tasks. This finding is consistent with recent work by Kissane et al. (0044), who found that SAEs trained base model activations can also faithfully reconstruct the activations of derived models. While a deeper mechanistic investigation into how instruction-tuning affects the underlying feature space is a promising direction, we leave this for future work. For our main experiments, we therefore use the more general hase transcoders.
B.2 ATTRIBUTION GRAPH COMPUTATION
Implementation Details. We use the implementation from Hanna et al. (2025) to compute attribu- tion graphs. The primary hyperparameters were set as follows: a maximum of 4096 feature nodes, attribution traced from a maximum of 10 logit nodes (selected by a cumulative probability threshold of 0.95), and batch size of 16 for backward passes. All other parameters follow the repository defaults.
Table 15: Performance comparison of CRV using different token positions for attribution graph computation. The "After" setting computes the graph at the final token of the current step, while "Before" uses the final token of the previous step. Arrows indicate preferred direction (f higher is better, & lower is better).
Synthetic (Boolean)
Synthetic (Arithmetic)
GSMSK
Attribution Position
AUROCT 1 AUPR
FPR
AUROC
AUPR
FPR@95
AUROC1
AUPR
FPR@95
Before Afler
68.66
1.80
77.44
85.95
12.05
47.89
70.32
16.19
85,29
75.87
0.97
79.17
92.47
28.92
37.09
70.17
14.3
79.61
Ablation on Attribution Position. The attribution graph is computed with respect to a specific token position. The choice of this position is a critical methodological decision, as it determines which computational moment we analyze. We investigate two hypotheses: analyzing the state before step is generated (the "pre-computation" trace) versus the state after it is complete (the "post- computation" trace). To test this, we compare two settings: (1) Before: computing the graph at the position of the final token of the previous reasoning step. For the first step of the CoT, this corresponds to the final token of the input question. (2) After: computing the graph at the final token of the current reasoning step, which is the default setting for our main experiments.
The results, presented in Table 15 show a clear and consistent advantage for the "After" setting across nearly all metrics and domains. We hypothesize that this because the structural signatures of flawed computation are most fully consolidated in the final token's 's representation after the step has been fully executed. The compuaaatnii state may contain signals of intent or planning, but the post-computation state contains the definitive trace of the executed algorithm, including the evidence of failure. Based on these results, experiments in the main body of the paper use the "After" (current step) position.
22

---

Published as a conference paper at ICLR 2026
C ADDITIONAL CLASSIFICATION DETAILS
C.I ATTRIBUTION GRAPH FEATURES
Here we give details about the extracted features for our attribution graphs that we used for our classifier. The feature set organized into three hierarchical levels:
1. Global Graph Statistics: These features provide a high-level summary of the pruned computa- tional graph.
The total number of active transcoder features, well the count of transcoder feature nodes and residual stream nodes remaining after pruning. This captures the overall sparsity and composition of the influential subgraph.
Node Counts:
The probability of the top-ranked token and the entropy of the final logit distribution. These classic uncertainty measures serve as simple but informative baseline features.
Logit Statistics:
Node Influence and Activation Statisties: This group of features characterizes the properties of the nodes within the pruned graph, moving beyond simple counts.
Influence Scores: The mean influence of all nodes in the pruned graph, along with the total and mean influence specifically from the residual stream ("error") nodes. This helps quantify how much of the final output is attributed to specific learned features versus the model's direct pass-through states.
Activation Statistics: For the pruned transcoder feature nodes, we compute the mean, max, and standard deviation of their activation values. This captures the intensity and distribution the active, interpretable features. A high maximum activation, for instance, might signal that single, highly decisive feature was responsible for the step.
Layer-wise Feature Histogram: A histogram of active transcoder features across the model's layers. This feature vector characterizes the distribution of computational effort across the model's depth, allowing us test hypotheses such as whether errors correlate with the activation of components at specific layers.
3. Topological and Path-Based Features: To capture the structure and efficiency of the information flow, we compute rich set of topological features on the pruned, directed subgraph.
Edge and Density Statistics: Aggregate statisties on the edge weights (sum, mean, std), the total number of edges, and the graph density. We hypothesize that sparse, fragmented graph (low density, few edges) may indicate breakdown in information flow characteristic of an error.
Centrality Measures: To identify critical "hub" nodes in the computation, calculate the mean and max for both degree centrality and weighted betweenness centrality. These features assess whether influence is concentrated or diffused.
Connectivity and Path Lengths: The number of weakly connected components and the average shortest path length within the largest component. A highly fragmented graph may suggest failed computation. A particularly crucial feature is the shortest path length from any input token node to any final logit node. This directly measures how efficiently information from the prompt propagates the final decision. A long or non-existent path is hypothesized to be strong signal that the model is "ignoring" its instructions or context.
C.2 ADDITIONAL DETAILS ON BASELINES
Here provide additional implementation details for the baseline methods used in our main exper- iments, ensuring full reproducibility.
Black- Box Baselines. This category includes methods that operate solely on the output logits of the final token for each reasoning step. We use implementations from Wang et al. (025a)).
23

---

Published as a conference paper at ICLR 2026
Gray-Box Baselines. This category includes methods that leverage the model's internal hidden states. For CoE (Wang et al. 2025a) and CoT-Kinetics (Bi et al. 2025). which are training- frree we followed the official implementations and protocols described by their respective authors to compute the verification scores. We set y in CoT-Kinetics to 0.8, and use mean pooling for reasoning token aggregation.
For our supervised LR Probe baseline, the choice of which layer's hidden states to use is hyper- parameter. To determine the optimal layer for each dataset, we performed hyperparameter search, training a separate probe on the average hidden states from each of the 32 layers of Llama 3.1 8B Instruct on a small validation split. This process allowed to identify the layer that contained the most predictive signal for each distinct reasoning task. The best west-performing layers, which were subsequently used for the main results reported in Table were found to be:
Layer 0 (the token embedding layer) for the Synthetic (Boolean) dataset.
Layer 9 for the Synthetic (Arithmetic) dataset.
Layer 0 (the token embedding layer) for the GSM8K dataset.
Similar to the protocol for the LR probe, we trained separate MLP Probes the hidden states of specific layers. We searched over a hyperparameter grid similar to Zhang et al. (2025), varying the
learning rate, weight decay, hidden dimension size (d), and the source layer index. The complete search space is provided in Table I6 We selected the best hyperparameters for each dataset based on small validation set. The optimal configurations are reported in Table 17
Table 16: Hyperparameter search space for the MLP Probe baseline.
Hyperparameter
Values Searched
Source Layer Index
1, 1,.,....,31 .11]
Hidden Dimension (d)
{16. 32, 64, 128)
Learning Rate
{1e-3, 1c-4, 1e-5)
Weight Decay
{1e-1, le-2, 1e-3, 1e-4, le-5)
Table 17: Optimal hyperparameters for the MLP Probe on each dataset. The Layer column indicates the hidden state layer that yielded the highest verification performance.
Dataset
Layer
Learning Rate
Weight Decay
Hidden Dim (d)
Synthetic (Boolean)
0
1e-3
le-1
16
Synthetic (Arithmetic)
12
lc-5
lc-1
GSM8K
0
lc-4
le-5
128
C.3 ADDITIONAL DETAILS ON EVALUATION METRICS
We report three evaluation metrics: the Area Under the Receiver Operating Characteristic curve (AUROC), the False Positive Rate 95% True Positive Rate (FPR@95) and the Area Under the Precision-Recd curve (AUPR). For all metrics, we treat the verifier output as real-valued score s(x) IR, where higher values indicate higher predicted likelihood of error. The ground-truth labels are y E (0, 1). where = denotes an incorrect reasoning step (the positive class) and = denotes correct step.
AUROC. The Area Under the ROC Curve is defined as the integral of the True Positive Rate (TPR) against the False Positive Rate (FPR):
AUROC /. TPR(T) d(FP))))FF))
The rates a given threshold r are defined
Σ. 1[s(z) m
FPR(7) Σ. 1[s(z) (z) 2 A
TPR(7)
Σ 1
Σ. 1 = 0
24

---

Published as a conference paper at ICLR 2026
We compute this via the trapezoidal rule by sweeping 7 across all observed score values.
FPR@ @95. This metric captures the False Positive Rate at specific operating point where the recall (TPR) is fixed at 0.95. It is defined as:
FPR @95 F
where T0.95 is the threshold such that TPR 70.95) 0.95. practice, we select the striet threshold that yields a TPR closest to, but not less than, 0.95.
AUPR. The Area Under the Precision- Recall Curve quantifies the trade-off between precision and recall for the positive (error) class. It is defined as:
AUPR L Precision(Recall d(Recall),
where Precision at threshold r is:
E. 1[s(z) 2
Precision
1.(0(()) 2
All metrics were computed using the standard implementations provided by the scikit-learn li- brary (Pedregosa et al. 201T). applied to the held-out test sets.
C.4 ADDITIONAL CLASSIFIER AND THEIR RESULTS
To validate our choice of a Gradient Boosting classifier for the main experiments, we benchmarked its performance against several standard alternatives on curated graph feature set. We evaluated a simple baseline. linear model, and another tree-based ensemble to understand the trade-offs between model complexity and verification performance. For this analysis and main experiments in this work, we used the default hyperparameters from the scikit-learn library for each classifier, as an initial, non-exhaustive hyperparameter search did not yield any significant improvements, suggesting that the feature set itself provides a strong signal that is not overly sensitive to classifier configuration.
Table 18: Performance comparison of different diagnostic classifiers. direction (t higher is better, J lower is better).
Arrows indicate preferred
Synthetic (Boolean)
Synthetic (Arithmetic)
GSM8K
Method
AUROCT 1
AUPR
FPR
AUROC t AUPR
FPR@95 L
AUROC1
AUPR 1 FPR
Dummy
0.25
48.06
6.46
Logistic Regression
76.4
0.75
68.91
89.5
11.46
41.56
73.8
18.70
78.69
Random Forest
61.71
4.49
100
92.99
43.68
30.56
71.7
17.65
76.18
Gradient Boosting 75.87
0.97
79.17
92.47
28.92
37.09
70.17
14.3
79.61
The results are presented in Table 18 based the training set's class distribution, performs near chance level (AUROC 50). This confirms that our graph features contain a significant predictive signal that is non-trivial to learn. Interestingly, standard Logistic Regression model achieves competitive performance, yielding the best AUROC on two of the three datasets and the strongest overall results on GSMSK. This indicates that the features are highly informative even with simple linear model.
As expected, the Dummy classifier, which makes predictions
However, the tree-based ensembles often achieve superior performance on other key metrics. The Random Forest classifier, for instance, yields a substantially higher AUPR and lower FPR @95 on the complex Arithmetic dataset, suggesting its ability to capture non-linear feature interactions is critical for high-precision verification that domain. Overall, no single classifier is dominant across all domains and metrics. We chose Gradient Boosting for our main experiments as it consistently provides a strong and robust performance profile, but these results highlight that the optimal choice of diagnostic classifier may be domain-specific.
C.5 ADDITIONAL. RESULTS FOR RQS
Here we provide additional results for our research questions. We first show distributions of highly predictive features for correct versus incorrect reasoning steps synthetic datasets (Figure
25

---

Published as a conference paper at ICLR 2026
00811
Prused Cole
May Bud
Daviyy J-0.000
Plaeed Uaa
pur D20 dell
00017
100 LEA Mn LS2
027
64 52V 0.5
19
Figure 6: Topological Fingerprints of Error on Arithmetic. Distributions of five selected graph features for correct (blue) vs. incorrect (red) reasoning steps. The visual separation is statistically significant for each feature shown (independent t-test, p < 0.001) and represents medium--t-laase effect size (Cohen's 's d). This provides quantitative evidence that attribution graphs contain a clear, separable structural signal of a computation 's integrity.
Fruned Nodea Lewer
Pruned hodes moverr
Praned io Lene5
pork 838
Grad
Figure 7: Topological Fingerprints of Error Boolean. Distributions of five selected graph features for correct (blue) vs. incorrect (red) reasoning steps. The visual separation is statistically significant for each feature shown (independent t-tstt, p < 0.001) and represents medium-to-large effect size (Cohen's d). This provides quantitative evidence that attribution graphs contain a clear, separable structural signal of a computation's 's integrity.
Coteet
Cineet
cetett
Incoed
incomed
Iromd
(a) Boolean
(b) Arithmetic
(c) GSM8K
Figure 8: Distributions of features after t-SNE for correct (blue) vs. incorrect (red) reasoning steps.
The separation validates that CRV captures distinct structural fingerprint of error.
for arithmetic; Figure 7 for Boolean). Next, we display the distributions of full feature vectors after t-SNE projection Figure 8
We demonstrate another casual intervention with a concrete case study on the arithmetic task, where correct subtle mathematical parsing error not by suppressing a faulty feature, but by amplifying a correct one. We present the model with the expression (- (5+ (4+9) ) ). As shown in Table 19] the model initially misinterprets the leading unary minus, treating it as subtraction from an implicit -1 and producing the incorrect result of -42, Our CRV classifier flagged this step incorrect, and its feature importance analysis suggested that the error stemmed from the absence of a strong "multiplication" signal.
Hypothesizing that the model failed to recognize the unary minus as a multiplication by -1, we performed direct intervention. We identified a transcoder feature from the graph whose semantics corresponded to multiplication and, using a forward hook, manually amplified its activation value just before the model generated its erroneous step. The effect was immediate and successful. As shown in the "After Intervention" column, this boost was sufficient to steer the model onto the correct computational path. It now correctly multiplies by -1 and arrives at the final answer of -41.
26

---

Published as a conference paper at ICLR 2026
successful causal intervention via feature amplification. subtraction error corrected by amplifying a multiplication feature, steering the model to the correct reasoning path.
Table 19:
Before Intervention (Incorrect)
After Intervention (Correct)
Evaluate the arithmetic expression below.
Evaluate the arithmetic expression below.
(-55+ (+++))))
(- (5 5 (4*))))
To evaluate this expression, we need to follow
To evaluate this expression, we need to follow
the order of operations (PEMDAS):
the order of operations (PEMDAS):
Multiply and 9: 4 9 36
Multiply and 9: 4 9 36
2. Add 5 and 36: 5 36 41
2. Add 5 and 36: 5 36 41
Subtract 41 from -1: 41 = -42
3. Multiply 11 by 41:
= -41
The final answer is -42.
The final answer is: -41.
D USAGE OF LANGUAGE LANGUAGE MODELS
In the preparation this manuscript, we utilized LLMs as writing assistants. Their role was strictly limited to improving the clarity, conciseness, and grammatical correctness of the text. The authors take full responsibility for all content and any remaining errors.
E LIMITATIONS
Our work introduces a new methodology for the scientific analysis of reasoning. and its limitations are intrinsically linked its design as a white-box, mechanistic instrument.
Computational Intensity. A primary limitation of CRV is its computational intensity. The pro- cess, which involves training a suite of transcoders, replacing model modules, and constructing detailed attribution graph for every reasoning step, orders of magnitude more resource-intensive than black-box or gray-box verification methods. This cost is a direct consequence of our white-box approach, which prioritizes mechanistic depth over practical efficiency. As such, CRV in its cur- rent form is positioned as scientific tool for deep analysis, not scalable, real-time verifier for production systems.
Aggregative vs. Feature-Level Analysis. The feature set used by CRV is primarily aggregative; it captures statistical and topological properties of the graph, such node counts, influence scores, and density. As an early work, it does not yet fully exploit the semantic content of the individual transcoder features that constitute the graph's hsss nodes. For instance, current classifier learns statis- tical correlations over the entire feature set; it does not reason symbolically about whether a specific feature for numerical addition is appropriately activated by numerical inputs. This represents a sig- nificant opportunity. A promising future direction lies in developing more sophisticated classifiers or rule-based systems that operate directly on the semantics of these disentangled features, paving the way for new class of nymbo-sy verifiers.
Generalizability of Error Signatures. Our empirical results are based on a single model family (Llama 3.1) the 8B scale. Whether the precise structural fingerprints we identified generalize to different architectural paradigms, such as Mixture-of-Experte or across significant model scales (e.g., 70B and larger) remains an open question. Furthermore, as our cross-domain experiments revealed, the error signatures are highly domain-specific. Our work provides strong foundation and a methodology for discovering these signatures, but further studies are needed to determine more universal principles of computational failure exist.
Fidelity of Interpretability Tools. The validity of our analysis is contingent on the quality and fidelity of the underlying interpretability tools. The features identified by our transcoders, while demonstrably useful, represent possible sparse basis and are not exhaustive. Similarly, the attri- bution method provides a powerful but ultimately incomplete approximation of the true information flow within the model. Future improvements in these foundational techniques, such the develop-
27

---

Published as a conference paper at ICLR 2026
ment of more faithful sparse autoencoders or more precise attribution methods, will directly enhance the resolution and reliability of analyses like ours.
28


Reviews:
[{"review_id": "CSiNp7N5bN", "reviewer": "Reviewer_Cn2f", "summary": "The paper proposes CRV, circuit based reasoning verification, an approach for classifying whether reasoning traces are correct and where they might have possibly made an error. This is done through tracing an attribution graph of an extended retrained formulation of the original LLM. Experiments on boolean, arithmetic and GSM8K reasoning tasks show better performance compared to black- and grey-box methods.", "questions": "- It is ironic that you need Gradient Boosting, a non-NN learning method, to better understand NNs.\n- Is the proposed method also applicable to other LLMs, e.g. Qwen? I do not expect you to perform this experiment, but would like to know your intuition behind that. \n- Transcoder:\n  - When you replace the original LLM's MLPs by transcoders, how similar are their performance? For example, if the model is significantly altered through that, then whether you can detect faulty reasoning for the surrogate model might not have much to do with detecting it for the original model.\n  - Related to the above question: Is the experimental evaluation fair w.r.t. other models? I guess you evaluate other models on the original Llama, while your method is on the modified one. If there is any shift in the MLP, possibly it might become easier overall to detect faulty reasoning.\n  - Related to that topic: I have read the appendix on the transcoder training and I am aware that it is done correctly. I still would like to have your opinion on the remaining difference to the original model.\n- Could you train a model on all three tasks simultaneously and what would its performance be? From what I understand you train a model for boolean, arithmetic and GSM8K separately.\n- Please add an exact formulation for each benchmark metric and how you evaluate all these numbers exactly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Conceptual:\n- The idea is very nice and approaches the reasoning veracity problem from a much more fine-grained angle than black- and grey-box  baselines. The overall architecture, i.e. transducers, attribution graph and classifiers on activation and other features are a very nice addition to the mechanistic interpretability community, as far as I know of the literature.\n\nExperimental:\n- I like the cross-domain experiment. Even though it shows limited domain generalization, it is a highly useful information and shows the authors adherence to an exhaustive evaluation, even when results are not favourable.\n- The intervention is a nice idea. However, it is done on only one single data point (one more in the appendix?) and I wonder about whether you can really identify the reason for incorrect reasoning and do interventions more generally.\n\nWriting:\n- The writing is very well done and a pleasure to read.", "weaknesses": "Conceptual:\n- The method, as acknowledged, is computation heavy and cannot be used without expensive retraining to identify reasoning weaknesses. I think it would be an interesting follow-up question whether such a white-box method could be directly grafted on an existing LLM.\n- Some questions on the fairness and experimental setup and comparison to baselines persist, please see below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLRu1tK5hH", "reviewer": "Reviewer_9q4r", "summary": "The paper proposes Circuit-based Reasoning Verification (CRV), a method to detect errors in LLM reasoning. The core hypothesis is that reasoning failures might manifest as detectable \"structural fingerprints\" on a model's computational graph. To study this, the authors modify a transformer model by replacing each MLP with a transcoder (a sparse, interpretable autoencoder trained to imitate the MLP’s function). Using this replacement model, they construct attribution graphs that capture causal connections between tokens, features, and logits during each step of a Chain-of-Thought (CoT).\n\nFrom these graphs, CRV extracts a set of structural and topological features (e.g., node influence, connectivity, entropy) and trains a diagnostic classifier to predict whether a reasoning step is correct or incorrect. Experiments on synthetic Boolean and arithmetic tasks, as well as GSM8K, suggest that CRV outperforms black-box (e.g., logit-based) and gray-box (e.g., hidden-state probe) baselines. However, the method does not generalise well across reasoning domains, which the authors interpret as evidence that error patterns are domain-specific.\n\nThe paper also uses CRV to analyse a single reasoning failure on a simple mathematical expression and show that causal interventions (modifying a single transcoder feature) can correct the models prediction.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper's core idea of treating the model's computational graph as a debuggable execution trace is compelling. \n- The case study demonstrating the correction of a reasoning error by intervening on a single transcoder feature is interesting.", "weaknesses": "- Prior work found that models’ hidden representations often contain detectable traces of reasoning success or failure (e.g., [1, 2]), with simple probes achieving strong performance in similar settings. In contrast, the LR probe baseline reported here performs notably poorly, suggesting it may not have been well-tuned. This undermines confidence in the relative performance gains claimed for CRV.\n- CRV combines two major components: (A) replacing dense MLPs with sparse, interpretable transcoders, and (B) constructing and analyzing an attribution graph. The paper attributes its success primarily to (B), yet it never evaluates a simpler baseline that trains a classifier directly on transcoder activation statistics (A) without graph construction. The ablations in Table 3 suggest that “Node Influence & Activation Stats” make up for most of the performance, implying that much of the signal may come from the sparse representation itself. \n\n[1] A. Zhang et al., ‘Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification’, in Second Conference on Language Modeling, 2025.\n\n[2] O. Obeso, A. Arditi, J. Ferrando, J. Freeman, C. Holmes, and N. Nanda, ‘Real-Time Detection of Hallucinated Entities in Long-Form Generation’, arXiv [cs.CL]. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LMvFrtapXy", "reviewer": "Reviewer_FZ7t", "summary": "The paper demonstrates a great use case of using interpretability techniques for a practical application of detecting invalid cases of chain of thought. In particular, the authors build out an attribution graphs for valid and invalid CoT steps. These graphs in turn can be turned into features for a classifier to classify valid and invalid CoT steps, using graphical features such as global graph statistics, node statistics, or topological features. These features are used to train a Gradient Boosting Classifier. \n\nThese experiments were conducted on carefully crafted datasets, for a boolean reasoning task, arithmetics, and carefully annotated GSM8k. The authors compare to a wide range of baselines, each of which are categorized as \"black-box\" or \"gray-box\" methods, in which their approach (CSV) outperforms all baselines.\n\nI appreciate that the authors check for cross-domain generalization of their CRV classifier. It turns out that CRV does not generalize well, suggesting that CRV (and the underlying attribution graph) likely uses specific signatures per domain. It would be interesting to see how to build a CRV model that generalizes well across domains.\n\nThe authors also demonstrate that CRV's stay effective regardless of the difficulty of the problem. The authors control for task difficulty with number of steps required to solve the problem, and CRV remains effective as task difficulty is scaled. \n\nA natural question that arises is what kind of features CRV relies on. By ablating away some features from the attribution graph, the authors identify that \"local features\" (i.e., node influence) matter the most, while \"global features\" such as topological features matter less.\n\nThe authors dig deeper to look at transcoder features (as opposed to features of the attribute graph) to identify specific concepts. As expected and as seen in numerous SAE work, intervening on these features have causal power to control the CoT reasoning traces. \n\nOverall a strong contribution that leverages attribution graphs for a tangible application.", "questions": "Most of my questions have been asked in the \"weaknesses\" section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is nicely scoped and demonstrates a clear win in leveraging attribution graphs to identify valid and invalid chain-of-thought steps. This provides a practical benefit of recent interpretability methods, which (as exciting as interpretability findings are) don't always happen. I hope to see more work that connects from interpretability insights to tangible actionable applications.", "weaknesses": "Overall the paper make claims/contributions that are well-scoped and nicely backed by their experiments. If I had to list out some weaknesses....\n\n* Figure 3: How much of this visualization is an artifact of projecting high-dimensional features to 2d? The claims of \"computational near misses\" and \"zone of computational integrity\" seem a bit hand-wavy based on these visualizations.\n\n* I'm not sure what other baselines would have been possible, but IIUC some of these baselines seem a bit unfair in that some of them don't have as much access to the LM's hidden states? For instance, PPL/MaxProb/PPL are overly simplified scalar summarizations of the model's forward pass. Is that correct?\n\n* Any thoughts on how to scale the diagnoses of identifying relevant features (of the transcoder) and trying causal intervention experiments? \n\n* I appreciate that the authors tested for cross-domain generalization, although the results seem to indicate that they don't. Any thoughts on how to build a diagnostic classifier that does generalize across domains? \n\n* The fact that LR probe performed so poorly is a bit surprising. It's also very surprising to see that layer 0 worked best for the LR probe. Why do you think Layer 0 performed best? Unrelatedly, could the poor performance of LR probe be cause of averaging the hidden states across tokens? Lastly, do you think simply using mean differences of hidden states (for correct vs. incorrect cases) could work better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HQSfmrpXLm", "reviewer": "Reviewer_Mmoa", "summary": "The authors leverage recent mechanistic interpretability techniques to develop a new white-box method for assessing the internal computations performed by a model while solving simple reasoning tasks. They show that they, using their circuit-based reasoning verification (CRV) outperform considered greybox and blackbox baselines on CoT-step-correctness prediction tasks on three small synthetic and math datasets.", "questions": "My overall assessment of the paper would improve if the authors convincingly address the main concerns raised in the weaknesses section.\n\nI would also encourage the authors to provide stronger evidence supporting their claim that the proposed approach is genuinely white-box in nature.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Advancing the field of mechanistic interpretability towards the practically relevant task of model debugging is a timely and highly relevant contribution. This work makes an important first step towards a scalable \"white-box\" technique for systematically tracing back model failures (in this case reasoning errors) to models' internal states and (hypothesized) learned algorithms.\n\nThe work performs thorough evaluations with several decent baselines and including several interesting ablations to understand the strengths and weaknesses of the proposed method. E.g., OOD evaluation shows that the method does not transfer well to unseen reasoning tasks and the feature analysis shows that ablating the topological graph features hurts the performance least. While in particular the second finding is a little bit in tension with the storyline of the paper of exploiting structural properties of the models latent algorithms, it is good scientific practice. \n\nIt is interesting that a mostly straightforward adaptation of transcoder based circuits in the proposed nice framing of graph classification outperforms the considered baselines on partial-CoT-correctness prediction.\n\nThe qualitative causal experiment of building CRV-informed interventions to correct LLM outputs shows that methods like this may unlock unexpected downstream usecases.", "weaknesses": "My major concerns with this paper are:\n* In the main experiment in table 1 the probing baseline gets outperformed by most of the black-box methods. This suggests to me that the probing setup might be flawed: the datasets seem tiny, was there enough data to train the probes? Probing performance is sensitive to the layer and token position. While it is common to aggregate hidden states over multiple tokens, if token sequences get too long like in CoT this seems like a too simplistic way of probe construction. Maybe it would make sense to consider a simple 1-layer attention based aggregation of the hidden states instead of the mean or a sliding window or something like that. I would be surprised if probing really underperforms simple baselines like the black-box ones as suggested by table 1.\n* The authors emphasize that the white-box nature of their approach is important to achieve this level of performance and being able to pick up on structural graph properties were the key factor for the success of the method. However, they also show that gradient boosting based classifier is mostly robust to ablation of topological feature and achieves almost the same level of performance considering only local and global graph features. This to me also suggests that a much simpler grey-box technique, e.g., not requiring a substitute, transcoder-based attribution should achieve similar performance on the considered tasks.\n* The approach is termed white-box however its white-box aspects are under-explored. The qualitative CRV-inspired intervention goes into the right direction, however, an in-depth analysis of the white-box aspects of the work would significantly strengthen the contribution. Another thing to consider to add towards the whitebox/interpretability/understanding framing would be to leverage transcoder feature annotations based on max activating examples and so on to help the debugging process. \n\nMinor concerns: \n* Many things, such as explanation of the circuit computation and baseline methods, are deferred to the related work. It would be nice to convey the main ideas and intuitions in the main paper. If the paper gets accepted I'd like to request a short explanation of the attribution based circuit finding algorithm and some of the baselines considered in table 1.\n* Some related work on reasoning model interpretability that is closely related is missing in the related work section. E.g., [1] provide some preliminary evidence that CoT reasoning indeed involves some learned algorithms. The same goes for [2], which reverse engineers a self-verification circuit in a toy-model.\n\n[1] https://arxiv.org/abs/2510.07364v3\n[2] https://arxiv.org/abs/2504.14379", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)