Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders

## Abstract

Abstract As Large Language Models become integral to software development, with substantial portions of AI-suggested code entering production, understanding their internal correctness mechanisms becomes critical for safe deployment. We apply sparse autoencoders to decompose LLM representations, identifying directions that correspond to code correctness. We select predictor directions using t-statistics and steering directions through separation scores from base model representations, then analyze their mechanistic properties through steering, attention analysis, and weight orthogonalization. We find that code correctness directions in LLMs reliably predict incorrect code, while correction capabilities, though statistically significant, involve tradeoffs between fixing errors and preserving correct code. Mechanistically, successful code generation depends on attending to test cases rather than problem descriptions. Moreover, directions identified in base models retain their effectiveness after instruction-tuning, suggesting code correctness mechanisms learned during pre-training are repurposed during fine-tuning. Our mechanistic insights suggest three practical applications: prompting strategies should prioritize test examples over elaborate problem descriptions, predictor directions can serve as error alarms for developer review, and these same predictors can guide selective steering, intervening only when errors are anticipated to prevent the 14.66% corruption rate from constant steering.

## 1 Introduction

Large language models have achieved substantial adoption in software development, with 30% of GitHub Copilot’s suggestions across one million developers entering production (Dohmke et al., 2023). Yet these same models fail on bug-prone code, achieving only 12.27% accuracy while reproducing 44% of historical training bugs verbatim (Guo et al., 2025). This contradiction between widespread deployment and fundamental reliability issues poses critical risks in healthcare, finance, and military applications where code failures have severe consequences. The core challenge lies in our lack of mechanistic understanding of how models internally determine code validity. While current code interpretability research provides insights, these approaches cannot isolate the specific mechanisms that distinguish when models will generate correct code versus reproduce training errors, limiting our ability to ensure reliable deployment.

Understanding code correctness mechanisms requires isolating specific features within model representations, yet neural networks complicate this through superposition, compressing thousands of features into fewer dimensions (Elhage et al., 2022). This compression creates polysemantic neurons that respond to completely unrelated concepts: a single neuron might fire for Python syntax, academic citations, HTTP requests, and Korean text simultaneously (Bricken et al., 2023). Current code interpretability research, while revealing what models encode about code structure, does not address this fundamental challenge of feature entanglement (Troshin & Chirkova, 2022; Paltenghi et al., 2024; Anand et al., 2024). Sparse autoencoders provide a solution by decomposing these superposed representations into interpretable components, having already proven effective at isolating mechanisms for entity recognition (Ferrando et al., 2024) and safety-relevant behaviors (Templeton et al., 2024) in natural language. Applying this decomposition to code generation represents an unexplored opportunity to determine whether models possess code correctness mechanisms.

Inspired by Ferrando et al. (2024)’s success in identifying entity recognition and uncertainty directions through SAEs, we adapt their framework to uncover how models internally represent code correctness. We apply their methodology to Gemma-2 using MBPP problems, analyzing residual streams at final prompt tokens where semantic information concentrates across layers (Geva et al., 2023; Lieberum et al., 2024). This suggests models employ two distinct mechanisms for code validity, with detection directions (identified via t-statistics) serving as error alarms and steering directions (via separation scores) enabling targeted corrections.

Overall, our contributions are as follows:

- •
Using sparse autoencoders, we discover detection directions predict errors reliably (F1: 0.821) but fail as confidence indicators for correct code (F1: 0.504), revealing asymmetry in how models represent code correctness.
- •
Through steering interventions, we show steering interventions produce significant corrections while introducing tradeoffs, fixing 4.04% of errors but corrupting 14.66% of correct code, necessitating selective rather than universal application.
- •
Attention analysis reveals test cases matter more than problem descriptions for both mechanisms, with correct-steering increasing test attention (+14.60) and incorrect-steering suppressing it (-12.69), while both ignore problem text.
- •
Weight orthogonalization proves correct directions are necessary for generation, their removal causing 83.62% corruption versus 18.97% for control.
- •
We demonstrate code correctness features persist from base to chat
models, with incorrect-predicting and correct-steering directions from base Gemma-2 retaining their effectiveness after instruction-tuning, despite SAEs being trained only on base model representations.

## 2 Sparse Autoencoders

Sparse autoencoders are motivated by the Linear Representation Hypothesis, which posits that neural networks encode meaningful concepts as directions in activation space (Park et al., 2023; Mikolov et al., 2013). However, models face a fundamental challenge: they must compress thousands of features in lower-dimensional spaces, creating superposition where individual neurons respond to multiple unrelated concepts (Elhage et al., 2022). A single neuron might simultaneously activate for Python syntax, academic citations, HTTP requests, and Korean text (Bricken et al., 2023). This polysemantic behavior makes it difficult to isolate specific mechanisms like code correctness detection from the entangled representations.

Dictionary learning provides a principled solution to decompose these superposed representations (Olshausen & Field, 1997). We employ pre-trained JumpReLU SAEs from GemmaScope (Lieberum et al., 2024), which expand model representations $\mathbf{x}\in\mathbb{R}^{d}$ into an 8× larger sparse space $a(\mathbf{x})\in\mathbb{R}^{d_{\text{SAE}}}$, enabling fine-grained feature separation. The encoding process applies:

$$ $a(\mathbf{x})=\text{JumpReLU}_{\theta}(\mathbf{x}\mathbf{W}_{\text{enc}}+\mathbf{b}_{\text{enc}})$ (1) $$

where JumpReLU implements threshold activation:

$$ $\text{JumpReLU}_{\theta}(\mathbf{x})=\mathbf{x}\odot H(\mathbf{x}-\theta)$ (2) $$

Here, $H$ represents the Heaviside step function and $\theta$ is a learnable threshold vector. The decoder reconstructs through

$$ $\text{SAE}(\mathbf{x})=a(\mathbf{x})\mathbf{W}_{\text{dec}}+\mathbf{b}_{\text{dec}}$ (3) $$

Training optimizes a combined loss that balances reconstruction accuracy with sparsity:

$$ $\mathcal{L}(\mathbf{x})=\underbrace{\|\mathbf{x}-\text{SAE}(\mathbf{x})\|_{2}^{2}}_{\mathcal{L}_{\text{reconstruction}}}+\underbrace{\lambda\|a(\mathbf{x})\|_{0}}_{\mathcal{L}_{\text{sparsity}}}$ (4) $$

This encourages monosemantic features where each dimension captures a single interpretable feature (Bricken et al., 2023; Templeton et al., 2024). We refer to $a_{i}(\mathbf{x})$ as latent activations (feature presence strength) and $\mathbf{W}_{\text{dec}}[:,i]$ as latent directions (learned feature vectors) throughout our analysis.

## 3 Related Work

Recent advances in mechanistic interpretability have mapped how language models process factual information, including entity recognition mechanisms (Ferrando et al., 2024), specialized extraction heads that route attributes to final positions (Geva et al., 2023), and structured circuits for factual recall (Nanda et al., 2023). These discoveries reveal interpretable, causal mechanisms underlying natural language processing. However, equivalent mechanistic understanding for code generation remains limited.

Current code interpretability research has provided valuable insights through probing classifiers, attention analysis, and model representation studies (Troshin & Chirkova, 2022; Paltenghi et al., 2024; Anand et al., 2024). While these approaches reveal what models encode about code structure and how they process information, they do not address the fundamental challenge of superposition—where models compress multiple features into the same neurons, making individual features difficult to isolate and interpret (Elhage et al., 2022). This superposition phenomenon means that analyzing raw activations provides entangled, polysemantic signals rather than clean, interpretable features.

Sparse autoencoders offer a solution by decomposing these entangled representations into interpretable components. SAEs have identified safety-relevant features (Templeton et al., 2024) and demonstrated their ability to extract meaningful computational structures from complex representations. Our work applies this methodology to code generation, employing validation through activation steering (Turner et al., 2023), weight orthogonalization (Arditi et al., 2024), and attention analysis (Nanda et al., 2023; Geva et al., 2023). This represents the first application of sparse autoencoders to address superposition in code representations, extending entity recognition methodologies to uncover how models internally represent program validity.

## 4 Methodology

We identify code correctness mechanisms in the Gemma-2-2b base model (Team et al., 2024) by extending Ferrando et al. (2024)’s entity recognition framework. Our approach discovers complementary mechanisms: detection directions that signal errors through confidence gradients, and steering directions that enable corrections through categorical activation patterns.

Using Mostly Basic Python Problems (MBPP) (Austin et al., 2021), a benchmark of 1,000 Python problems with test-based evaluation, we generate binary-labeled samples via the pass@1 criterion with temperature 0 for deterministic outputs. Each prompt is formatted using a standardized template (Figure [1](https://arxiv.org/html/2510.02917v1#S4.F1)) containing three components: problem description, test cases, and a code initiator. We add the code initiator to guide the base model to start generating code. Passing all three test cases yields correct labels; any failure produces incorrect labels. Data splits allocate 50% for direction selection, 10% for threshold & coefficient calibration, and 40% for mechanistic analysis.

Figure: Figure 1: Standardized prompt template for MBPP problems containing problem description, test cases with function signatures, and code initiator.
Refer to caption: x1.png

We extract residual stream activations at the final token before the model’s answer begins, following Marks & Tegmark (2023) who demonstrated that end-of-instruction tokens aggregate information about the entire question. This approach, also employed by Ferrando et al. (2024) for uncertainty directions, captures the model’s complete understanding of the problem specification before generation commences. Pre-trained GemmaScope autoencoders (Lieberum et al., 2024) decompose these activations into interpretable latents $a_{l,j}(\mathbf{x}_{l})$ at each layer $l$.

We exclude features activating $>$2% on the pile-10k dataset to filter out general language patterns. From this filtered set, we apply complementary metrics suited to each mechanism’s computational role. Throughout our analysis, $N^{\text{correct}}$ and $N^{\text{incorrect}}$ denote the total number of correct and incorrect code samples, respectively.

Prediction directions require sensitivity to confidence gradients. Code correctness manifests not as binary presence but as activation intensity differences between correct and incorrect samples. The t-statistic captures these graded signals while accounting for variance:

$$ $t_{l,j}^{\text{correct}}=\frac{\mu(a_{l,j}(\mathbf{x}_{i}^{\text{correct}}))-\mu(a_{l,j}(\mathbf{x}_{i}^{\text{incorrect}}))}{\sqrt{\frac{\sigma(a_{l,j}(\mathbf{x}_{i}^{\text{correct}}))^{2}}{N^{\text{correct}}}+\frac{\sigma(a_{l,j}(\mathbf{x}_{i}^{\text{incorrect}}))^{2}}{N^{\text{incorrect}}}}},\quad t_{l,j}^{\text{incorrect}}=\frac{\mu(a_{l,j}(\mathbf{x}_{i}^{\text{incorrect}}))-\mu(a_{l,j}(\mathbf{x}_{i}^{\text{correct}}))}{\sqrt{\frac{\sigma(a_{l,j}(\mathbf{x}_{i}^{\text{correct}}))^{2}}{N^{\text{correct}}}+\frac{\sigma(a_{l,j}(\mathbf{x}_{i}^{\text{incorrect}}))^{2}}{N^{\text{incorrect}}}}}$ (5) $$

where $\mu$ and $\sigma$ denote the mean and standard deviation of non-zero activations. This identifies features encoding model confidence about code correctness.

Steering directions demand categorical exclusivity for clean intervention. We first compute how frequently each feature activates:

$$ $f_{l,j}^{\text{correct}}=\frac{1}{N^{\text{correct}}}\sum_{i=1}^{N^{\text{correct}}}\mathbf{1}[a_{l,j}(\mathbf{x}_{l,i}^{\text{correct}})>0],\quad f_{l,j}^{\text{incorrect}}=\frac{1}{N^{\text{incorrect}}}\sum_{i=1}^{N^{\text{incorrect}}}\mathbf{1}[a_{l,j}(\mathbf{x}_{l,i}^{\text{incorrect}})>0]$ (6) $$

Separation scores then measure exclusivity:

$$ $s_{l,j}^{\text{correct}}=f_{l,j}^{\text{correct}}-f_{l,j}^{\text{incorrect}},\quad s_{l,j}^{\text{incorrect}}=f_{l,j}^{\text{incorrect}}-f_{l,j}^{\text{correct}}$ (7) $$

High separation indicates switch-like features firing predominantly for one code type, enabling targeted corrections without affecting the opposite category.

Searching across all 26 model layers, we select features with maximum t-statistics for prediction and maximum separation scores for steering.

**Table 1: Key features identified for mechanistic analysis with their usage across sections.**
| Feature | Layer | Index | Metric | Used in |
| --- | --- | --- | --- | --- |
| Correct Predicting | 16 | 14439 | t-stat: 5.086 | [5.1](https://arxiv.org/html/2510.02917v1#S5.SS1), [5.5](https://arxiv.org/html/2510.02917v1#S5.SS5) |
| Incorrect Predicting | 19 | 5441 | t-stat: 5.680 | [5.1](https://arxiv.org/html/2510.02917v1#S5.SS1), [5.5](https://arxiv.org/html/2510.02917v1#S5.SS5) |
| Correct Steering | 16 | 11225 | sep: 0.221 | [5.2](https://arxiv.org/html/2510.02917v1#S5.SS2), [5.3](https://arxiv.org/html/2510.02917v1#S5.SS3), [5.4](https://arxiv.org/html/2510.02917v1#S5.SS4), [5.5](https://arxiv.org/html/2510.02917v1#S5.SS5) |
| Incorrect Steering | 25 | 2853 | sep: 0.201 | [5.2](https://arxiv.org/html/2510.02917v1#S5.SS2), [5.3](https://arxiv.org/html/2510.02917v1#S5.SS3), [5.4](https://arxiv.org/html/2510.02917v1#S5.SS4), [5.5](https://arxiv.org/html/2510.02917v1#S5.SS5) |

## 5 Mechanistic Analysis

### 5.1 Detection Directions Predict Errors Reliably

Predictor directions reveal that models develop anomaly detectors rather than validity assessors. Despite similar AUROC scores ($\sim$0.6), incorrect-preferring features achieve F1=0.821 while correct-preferring features reach only 0.504.

Figure: Figure 2: Top 10 tokens with the highest logit increases from predictor features. Left: Correct-predicting feature (L16-14439). Right: Incorrect-predicting feature (L19-5441).
Refer to caption: figures/correct-predicting.png

Inspecting the top positive logits uncovers what these features detect. The incorrect-predicting feature activates on anomalous patterns such as null indicators, achieving 0.985 recall and 0.703 precision by detecting irregularities characteristic of errors. Unexpectedly, it also responds to foreign language tokens, providing empirical evidence that while SAEs decompose into sparse features, they do not fully solve polysemanticity. Correct-preferring features show worse specificity, activating on formatting tokens rather than semantic patterns. This produces extensive false positives as the feature mistakes well-formatted incorrect code for correct implementations. The metrics confirm this failure: while recall reaches 0.828 from detecting structured code, precision drops to 0.362 due to false positives, resulting in an F1 of 0.504 that shows the limitations of surface-level detection.

Figure: Figure 3: Temperature robustness analysis from T=0.0 to T=1.4. Left: AUROC scores. Right: F1 scores.
Refer to caption: figures/AUROC-temperature.png

This anomaly-detection architecture remains stable. Temperature variations (0.0-1.4) leave error detection intact or improved (F1: 0.821→0.986), as anomalies remain detectable regardless of sampling randomness. Meanwhile, formatting-based correct detection degrades, confirming its superficial nature.

These findings reveal an asymmetry: models encode incorrect code as detectable anomalies but lack corresponding representations for correctness. While this prevents using these features as general confidence indicators, the reliable error detection (F1: 0.821) suggests practical utility as an alarm system, flagging generations that require review.

### 5.2 Steering Directions Achieve Modest Corrections

Transitioning from correlational identification to causal validation, we employ activation steering (Turner et al., 2023) to test whether separation-score directions influence code generation. Our intervention modifies the model’s residual stream:

$$ $\mathbf{x}^{\text{steered}}=\mathbf{x}+\alpha\cdot\mathbf{W}_{\text{dec}}[j,:]$ (8) $$

where $\mathbf{W}_{\text{dec}}[j,:]$ is the latent direction and $\alpha$ controls steering strength. We optimize steering coefficients through a two-phase search, where grid search (intervals of 10) identifies active ranges before the golden section search pinpoints optimal values. For correct steering, we maximize correction rate ($\alpha=29$), while for incorrect steering we maximize $\frac{1}{2}(C_{r}+S)$ where $C_{r}$ denotes corruption rate and $S$ the mean Python token similarity percentage ($\alpha=287$), preserving code structure while breaking functionality.

Figure: (a)
Refer to caption: figures/correction-rates.png

We employ three setups: baseline (no steering, hence 0% correction/corruption as code maintains initial state), control (feature L1-4801 with zero discrimination, coefficients $\alpha=29$ for correct steering comparison and $\alpha=287$ for incorrect steering comparison), and steering directions. One-tailed (greater) binomial testing validates two hypotheses: code correctness directions versus baseline tests if steering has any effect, while code correctness directions versus control tests if our feature selection matters beyond random perturbation.

Figure: Figure 5: Left: Correct-steering example. Right: Incorrect-steering example.

Steering interventions validate causal influence while revealing inherent tradeoffs. Correct-steering achieves 4.04% correction rate on initially incorrect code (p$<$0.001), with Figure [5](https://arxiv.org/html/2510.02917v1#S5.F5) providing a concrete example. Yet this same intervention corrupts 14.66% of initially correct code. This degradation rate exceeds the correction rate nearly fourfold, suggesting selected steering rather than constant steering. Logit analysis reveals that correct-steering amplifies formatting tokens (spaces, tabs, comments), yet corrected code samples demonstrate semantic improvements, including bug fixes and algorithm implementations. The gap between formatting-related logits and semantic corrections illustrates why steering experiments are more informative than logit analysis alone.

Figure: Figure 6: Top 10 tokens with the highest logit increases from steering features. Left: Correct-steering feature (L16-11225). Right: Incorrect-steering feature (L25-2853).
Refer to caption: figures/correct-steering.png

Incorrect-steering’s failure is more straightforward, with ’8’ token repetition in steered code (Figure [5](https://arxiv.org/html/2510.02917v1#S5.F5)), confirming that separation scores are ineffective at identifying incorrect features. This is further supported by Figure [6](https://arxiv.org/html/2510.02917v1#S5.F6), where the tokens are predominantly variations of ’eight’. This failure pattern explains why the steering coefficient search algorithm settles on the comparatively large value of 287, nearly 10-fold larger than the correct steering coefficient of 29. Such large magnitudes mean that even control features produce a substantial impact when steered at this coefficient, explaining their complete degeneracy.

### 5.3 Test Cases Matter More Than Problem Descriptions

Attention weight analysis reveals how steering interventions redistribute focus across prompt components. We extract attention weights from all heads at the steering layer (L16 for correct, L25 for incorrect) at the final instruction token, sum attention within three prompt sections, normalize each section to percentages of total attention, and then compute percentage point changes between baseline and steered conditions.

Figure: Figure 7: Percentage point changes in attention to problem descriptions, test cases, and code initiator under steering interventions.
Refer to caption: figures/attention_delta_plots.png

Test cases exhibit the largest differential at 27.29 points between correct (+14.60) and incorrect (-12.69) steering. Problem descriptions decrease regardless of steering direction, never exceeding an 8-point reduction. The code initiator, a prompt artifact less relevant than problem descriptions and test cases, receives increased attention (+17.54) under incorrect-steering, confirming these features disrupt information processing. These patterns demonstrate that successful code generation depends on attending to test cases rather than problem descriptions. This mechanism suggests that prompting strategies should prioritize concrete test examples over detailed problem descriptions.

### 5.4 Correct Directions Prove Necessary for Generation

Weight orthogonalization permanently modifies every matrix writing to the residual stream:

$$ $\mathbf{W}_{\text{out}}^{\text{new}}\leftarrow\mathbf{W}_{\text{out}}-\mathbf{W}_{\text{out}}\mathbf{d}^{T}\mathbf{d}$ (9) $$

following Arditi et al. (2024), this prevents the model from writing the specified direction $\mathbf{d}$. We employ three setups: baseline (no orthogonalization, hence 0% correction/corruption as code maintains initial state), control (feature L1-4801 with zero discrimination), and steering directions.

Correct orthogonalization corrupts 83.6% of initially correct solutions, compared to only 19.0% for control features (p$<$0.001), a 4.4-fold difference. As shown in Figure [8](https://arxiv.org/html/2510.02917v1#S5.F8)(a), functional code degrades to comments or empty strings when these features are removed. The model retains task knowledge (evidenced by relevant comments) but cannot produce executable code. This demonstrates that correct-steering features are necessary for code generation.

Incorrect orthogonalization shows contrasting results. We expected removing incorrect steering directions would reduce errors, but achieved only a 2.2% correction rate, below control features at 5.5%. This indicates our failure to identify effective incorrect-preferring directions, consistent with Section [5.2](https://arxiv.org/html/2510.02917v1#S5.SS2), where separation scores proved ineffective for finding incorrect features.

Figure: (a)
Refer to caption: figures/incorrect_orthogonalization.png

### 5.5 Mechanisms Persist from Base to Chat Models

GemmaScope SAEs were trained exclusively on the base model using pre-training data, yet SAE-derived directions retain their effectiveness in the instruction-tuned model. This persistence occurs despite instruction-tuning improving baseline performance from 29.9% to 38.4% pass rate on MBPP.

Error detection remains reliable across both models. The incorrect-preferring feature (L19-5441) achieves F1=0.821 in the base model and F1=0.772 after instruction-tuning. Both models maintain F1$>$0.75 for incorrect prediction, preserving the reliability threshold established in Section [5.1](https://arxiv.org/html/2510.02917v1#S5.SS1).

Steering interventions retain statistical significance. Correct-steering achieves 4.04% correction rate in the base model (p$<$0.001) and 2.93% in the instruction-tuned model (p$<$0.001). While the rate decreases, both models demonstrate statistically significant correction ability using identical features and coefficients.

These results indicate code correctness mechanisms learned during pre-training persist through instruction-tuning. Rather than developing new mechanisms, fine-tuning appears to refine existing representations while maintaining their fundamental structure. This persistence enables SAEs trained on base models to identify causally relevant features in their instruction-tuned counterparts.

## 6 Conclusions

Using sparse autoencoders, we identified and characterized code correctness directions in LLM representations, finding that predictor directions reliably detect incorrect code (F1: 0.821) while steering directions achieve corrections with inherent tradeoffs (4.04% fixed, 14.66% corrupted). Mechanistically, we demonstrated that successful code generation depends on attending to test cases rather than problem descriptions. Notably, incorrect-predicting and correct-steering directions identified in base models retain their effectiveness after instruction-tuning, suggesting code correctness mechanisms learned during pre-training are repurposed during fine-tuning. These mechanistic insights suggest practical applications: prompting strategies should prioritize test examples over elaborate problem descriptions, predictor directions can serve as error alarms for developer review, and these same predictors can guide selective steering, intervening only when errors are anticipated to prevent the 14.66% corruption rate from constant steering. This work advances the mechanistic interpretability of code processing in LLMs, revealing how models internally represent and process code correctness.

## 7 LLM Usage

We used Claude (Anthropic) to assist with multiple aspects of this work. For implementation, we used Claude Code to generate analysis code for SAE decomposition, steering interventions, and attention analysis based on our specifications, with all code being manually reviewed, tested, and validated. For manuscript preparation, Claude assisted with initial draft generation, which we subsequently revised and refined. For the literature review, we used Claude to identify potentially relevant papers, though all citations were independently verified for relevance and accuracy. All experimental design, analysis, interpretation of results, and scientific conclusions are our own. We take full responsibility for all content in this paper.

## References

- Anand et al. (2024)
Abhinav Anand, Shweta Verma, Krishna Narasimhan, and Mira Mezini.
A critical study of what code-llms (do not) learn.
*arXiv preprint arXiv:2406.11930*, 2024.
- Arditi et al. (2024)
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda.
Refusal in language models is mediated by a single direction.
*arXiv preprint arXiv:2406.11717*, 2024.
- Austin et al. (2021)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
Program synthesis with large language models.
*arXiv preprint arXiv:2108.07732*, 2021.
- Bricken et al. (2023)
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah.
Towards monosemanticity: Decomposing language models with dictionary learning.
*Transformer Circuits Thread*, 2023.
https://transformer-circuits.pub/2023/monosemantic-features/index.html.
- Dohmke et al. (2023)
Thomas Dohmke, Marco Iansiti, and Greg Richards.
Sea change in software development: Economic and productivity analysis of the ai-powered developer lifecycle.
*arXiv preprint arXiv:2306.15033*, 2023.
- Elhage et al. (2022)
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.
Toy models of superposition.
*Transformer Circuits Thread*, 2022.
https://transformer-circuits.pub/2022/toy_model/index.html.
- Ferrando et al. (2024)
Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda.
Do i know this entity? knowledge awareness and hallucinations in language models.
*arXiv preprint arXiv:2411.14257*, 2024.
- Geva et al. (2023)
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson.
Dissecting recall of factual associations in auto-regressive language models.
*arXiv preprint arXiv:2304.14767*, 2023.
- Guo et al. (2025)
Liwei Guo, Sixiang Ye, Zeyu Sun, Xiang Chen, Yuxia Zhang, Bo Wang, Jie M Zhang, Zheng Li, and Yong Liu.
Llms are bug replicators: An empirical study on llms’ capability in completing bug-prone code.
*arXiv preprint arXiv:2503.11082*, 2025.
- Lieberum et al. (2024)
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda.
Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2.
*arXiv preprint arXiv:2408.05147*, 2024.
- Marks & Tegmark (2023)
Samuel Marks and Max Tegmark.
The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.
*arXiv preprint arXiv:2310.06824*, 2023.
- Mikolov et al. (2013)
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their compositionality.
*Advances in neural information processing systems*, 26, 2013.
- Nanda et al. (2023)
Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Rohin Shah.
Fact finding: Attempting to reverse-engineer factual recall on the neuron level.
AI Alignment Forum, 2023.
URL [https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall](https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall).
- Olshausen & Field (1997)
Bruno A Olshausen and David J Field.
Sparse coding with an overcomplete basis set: A strategy employed by v1?
*Vision research*, 37(23):3311–3325, 1997.
- Paltenghi et al. (2024)
Matteo Paltenghi, Rahul Pandita, Austin Z Henley, and Albert Ziegler.
Follow-up attention: An empirical study of developer and neural model code exploration.
*IEEE Transactions on Software Engineering*, 2024.
- Park et al. (2023)
Kiho Park, Yo Joong Choe, and Victor Veitch.
The linear representation hypothesis and the geometry of large language models.
*arXiv preprint arXiv:2311.03658*, 2023.
- Team et al. (2024)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
*arXiv preprint arXiv:2408.00118*, 2024.
- Templeton et al. (2024)
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan.
Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.
*Transformer Circuits Thread*, 2024.
URL [https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html](https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html).
- Troshin & Chirkova (2022)
Sergey Troshin and Nadezhda Chirkova.
Probing pretrained models of source code.
*arXiv preprint arXiv:2202.08975*, 2022.
- Turner et al. (2023)
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid.
Activation addition: Steering language models without optimization.
*arXiv e-prints*, pp. arXiv–2308, 2023.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 2
presentation: 3
questions: - Refer to weaknesses
- You exclude features activating >2% on pile-10k. How sensitive are results to this threshold?
- Do detection and steering directions work differently for different error types (syntax errors, logic errors)? Breaking down performance by error category would be valuable.
soundness: 3
strengths: - First use of SAEs to decompose code correctness representations, extending entity recognition frameworks to a new domain with appropriate adaptations.
- The paper employs multiple validation techniques (steering, attention analysis, weight orthogonalization) that provide converging evidence for causal mechanisms rather than mere correlations.
- The discovery that models encode incorrect code as detectable anomalies but lack corresponding representations for correctness reveals fundamental insights about how LLMs represent code validity.
- Demonstrating that code correctness mechanisms persist from base to chat models (F1: 0.821 → 0.772 for error detection) suggests interpretability methods can generalize across training stages.
summary: This paper applies sparse autoencoders (SAEs) to decompose LLM representations and identify directions corresponding to code correctness in the Gemma-2-2b model. The authors discover two distinct mechanisms: detection directions that reliably predict incorrect code (F1: 0.821) but fail as confidence indicators for correct code (F1: 0.504), and steering directions that achieve modest corrections (4.04% of errors fixed) while corrupting 14.66% of initially correct code. Through mechanistic analysis including activation steering, attention weight analysis, and weight orthogonalization, the authors demonstrate that successful code generation depends primarily on attending to test cases rather than problem descriptions, and that correct-steering directions are causally necessary for code generation. The work represents the first application of SAEs to address superposition in code representations and suggests practical applications including using predictor directions as error alarms and applying selective rather than constant steering interventions.
weaknesses: - The entire analysis is conducted on a single model (Gemma-2-2b, 2B parameters) and single benchmark (MBPP). This raises serious questions about whether findings generalize to other model families, sizes, or programming tasks (e.g., HumanEval, APPS, more complex algorithms). The authors should at minimum validate key findings on one additional model and benchmark.
- The 4.04% correction rate is overshadowed by the 14.66% corruption rate which is nearly 4-fold difference. While the authors acknowledge this necessitates "selective steering," they don't demonstrate or evaluate such a selective approach. The practical utility remains unclear without showing that combining detection and steering actually improves overall performance.
- The incorrect steering direction identified via separation scores produces only repetitive '8' tokens and achieves merely 2.2% corrections (below control at 5.5%). This represents a fundamental failure of the separation score methodology for identifying incorrect features.
- The discovery that the incorrect-predicting feature activates on both anomalous code patterns AND foreign language tokens directly contradicts the SAE promise of monosemantic features. The paper dismisses this as "empirical evidence that SAEs do not fully solve polysemanticity" but doesn't adequately discuss implications for the reliability of their findings or alternative approaches.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - Can the features decomposed by a sparse autoencoder be generalized to other coding tasks (such as multi-language, multi-file projects)?

- How can the manipulation direction be optimized to reduce the rate of breaking correct code and make it more suitable for practical deployment?

- In actual development processes, how can the detection direction be combined with existing static analysis tools to improve code quality?

- The paper failed to effectively identify the "erroneous code" direction, are there alternative methods or future plans?
soundness: 2
strengths: - This study elucidates the mechanism of using sparse autoencoders in code generation models, overcoming the interpretability challenge caused by feature superposition.

- It covers multiple aspects, including detection, manipulation, attention distribution, and weight orthogonalization, with rigorous experimental design and thorough statistical testing.

- It proposes superior code suggestion strategies, error alerting mechanisms, and targeted model intervention recommendations, demonstrating practical engineering value.

- It proves that the correctness mechanism in the pre-trained model remains effective after fine-tuning, possessing both theoretical significance and practical applicability.
summary: This paper investigates the "code correctness" mechanism of Large Language Models (LLMs) in code generation tasks, motivated by the widespread application of AI code generation but with inherent reliability risks, especially in high-risk domains. 
The authors decompose the internal representation of LLMs using a Sparse Autoencoder (SAE) to identify directions relevant to code correctness: detected directions are used to predict errors, while manipulated directions can be used to correct them. 
The paper systematically analyzes the performance, attention distribution, and persistence of these mechanisms after model fine-tuning, finding that test cases are more critical than problem descriptions, and offers implications for practical development processes, such as improved prompt design and automatic error alerts.
weaknesses: - The correction rate was only 4.04%, while erroneous intervention resulted in 14.66% of correct code being corrupted, indicating a high risk in the practical application of the manipulation direction and insufficient exploration of how to reduce side effects.

- Although key experiments were conducted, the experiments were not comprehensive enough: (1) All experiments were based on Gemma-2 and MBPP, failing to cover other models or more complex code scenarios, and the generalizability of the results needs further verification. (2) Negative mechanism identification failed: the "erroneous code" direction was not effectively identified, and the related experimental results and analysis were relatively weak.

- The pipelines relied on closed-source large models: Although manually reviewed, some experiments and literature searches relied on Claude, which may affect independence.

- Missing some references:
  - Sparse Autoencoders Find Highly Interpretable Features in Language Models
  - A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  - Sparse Autoencoders Find Highly Interpretable Features in Language Models

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Overall while the authors did a good job it misses the core question of how does the issue of entanglement be solved.

2. Methodological Gaps
	•	No clear guarantee of SAE uniqueness - how do authors ensure SAEs avoid the entanglement problem they claim to solve?
	•	Mean deviation process in prediction direction needs better motivation and explanation
	•	Missing details on how different values in Lines 261-265 are obtained

3. Conceptual and Terminological Issues
	•	"Causal influence" (L306) would be more accurately termed "intervention"
	•	Introduction could better connect SAE methodology symbols to the actual methodology section
	•	Some claims about superposition and feature compression need stronger theoretical grounding

4. Incomplete Analysis
	•	Limited discussion of potential failure modes or limitations of the SAE approach
	•	Insufficient exploration of how findings generalize across different code types and complexity levels
soundness: 3
strengths: a. Important and Timely Research Question
	•	Addresses critical need for understanding LLM code generation reliability as these models enter industrial deployment
	•	Focuses on mechanistic interpretability, which is essential for trustworthy AI systems

b. Novel Methodological Approach
	•	Creative application of SAEs to decompose code correctness representations into interpretable features
	•	Addresses the superposition problem that compresses high-dimensional features into lower dimensions
	•	Uses multiple complementary analysis techniques (steering, attention, weight orthogonalization)

c. Well-Structured Experimental Design
	•	Systematic approach using t-statistics and separation scores
	•	Demonstrates both predictive capabilities and correction mechanisms
	•	Acknowledges trade-offs in the findings (error fixing vs. correct code preservation)
summary: The authors apply previously developed methods to analyze LLM performance in code generation tasks. This is a field that has seen widespread adoption as in both academia and industry the share LLM generated code is increasing.  The authors applied sparse autoencoders on the residual streams of each layer to disentangle super-positioned representations and then detect the most impactful latent features. These were analyzed and intervened upon through a steering method. The authors confirmed previous results, that LLMs display anomaly detection mechanisms and not validity assessments. Furthermore, they found that applying steering directions to identified latent features leads to a trade-off between accurate code corrections and corrupting previously well generated code.
weaknesses: a. All of the analysis was performed on the latent features with highest metric as displayed in Table 1. Do the conclusions hold for say the top-5 to -10 latent features? Are the metric values for these features outliers compared to other features? Some further details in this aspect would significantly enhance the robustness of results. 

b. There is some lack of clarity when reading the paper, particularly if unfamiliar with the methods developed in Ferrando et al., 2024 and Marks & Tegmark, 2023. There was some effort to mention details of the methods, although not enough was included to fully grasp the methods (In some cases it could better to simply guide the reader to the original paper). Some aspects that could be addressed include: 

1. Clarify indexing of equations 5-7 

2. Clarify the use of the identified latent feature for classification through thresholding, as understanding the appearance of AUROC and F1 scores was not clear. 

3. Personally, the usage of “feature” for the encoded position in the SAE can be error inducing as the word is more commonly used for input “features”. Possibly using “latents” (as in Ferrando et al., 2024) could be better. 

4. Why was feature L1-4801 used as control? 

5. Some further clarification on how the steering strength parameter was obtained could be added as supplementary material. 

c. The values plotted in Figure 3 generally do not seem to match the values mentioned in the text. The F1 value for incorrect-predicting at T=0 is below 0.8 while the reference value in the text is 0.821, and in line 246 the improvement 0.821->0.986 is mentioned whereas the values seem to lie in the [0.76, 0.83] interval. 

d. The key suggestion given by the authors from the obtained results, is to include more test cases when prompting. While this is easily applicable in the used dataset for simple methods, it is often not so simple when developing methods in real scenarios. It would have a larger impact on SOTA if suggestions on improving LLMs to achieve validity assessment mechanisms. 


Minor suggestion:

a. MBPP is not defined in line 56, later defined in line 140 

b. The link of reference Elhage et al. 2022 is incomplete, while the text is present the link only goes to “/toy”.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: Small Comment:
It would be great to reference Figure 7 in the text.
soundness: 2
strengths: Novelty: This is the first mechanistic interpretability study specifically aimed at code correctness in LLMs, offering fresh insights into a relatively unexplored question that prompts opportunities for follow-up directions in the research community.

Transfer Signal: Some of the discovered directions carry over from base to chat-tuned model, hinting these are not pure overfits to one checkpoint.
summary: This paper uses layer-specific sparse autoencoders on Gemma-2-2B’s residual stream to identify “predictor” directions that distinguish correct vs incorrect code generations and “steering” directions that, when added at inference, bias the model’s generation. The authors produce key findings on coding benchmark MBPP, the best “incorrect-predicting” feature serves as a modest error alarm (F1 = 0.82), while the “correct-predicting” feature is weak (F1 = 0.504). Steering with the “correctness” direction repairs ~4.0% of initially wrong solutions but also corrupts ~14.7% of initially correct ones. The authors also perform an attention analysis showing steering shifts focus towards the unit tests. The authors also suggest that the correct directions are causal for code correctness behaviour as orthogonalizing a “correct” direction degrades ~83.6% of previously correct solutions vs. ~19% for a neutral control. The authors found that these directions largely transfer from the base model to a chat-tuned variant, indicating some robustness of the discovered mechanisms.
weaknesses: **Methodology Ambiguity:**
The paper’s high-level recipe is clear, but many operational details are missing or scattered, which makes replication and interpretation difficult. In particular, several core steps are not specified precisely enough to rule out confounds or to let others reproduce the results end-to-end:
- Definition of S (similarity): “Mean Python token similarity %” is undefined (tokenizer, reference, surface vs AST notion of similarity, averaging window). Clarifying this would strengthen the understanding.
- Evaluation bookkeeping: How “initially correct” is labeled, accuracy, and sample counts per condition. Clarifying this would strengthen the claims.
- Control condition: The neutral control feature has not been explored and a single control is weak to support specificity claims. 

**Feature-Selection Landscape:**
The current search/selection feels under-characterized. Please add a compact Feature-Selection Landscape to show the chosen directions are genuine outliers rather than hand-picked. These additions make selection auditable at a glance, demonstrate how exceptional the chosen features are, and strengthen this paper’s evidence.

**Single Benchmark Scope:**
All results are on MBPP (short Python tasks). To support this paper
claims about code correctness more broadly, it would help to include at least one additional benchmark (HumanEval, APPS) and, ideally, also non-Python. Showing that the key findings (e.g. stronger error alarm, test-focused attention, and orthogonalization effects) transfer across datasets/languages will materially improve generality and impact.


Decision: Reject