Failure by Interference: Language Models Make Balanced Parentheses Errors  When Faulty Mechanisms Overshadow Sound Ones
Authors: Daking Rai
Samuel Miller
Kevin Moran
Ziyu Yao
Venue: NeurIPS 2025 poster ()
Keywords: Language Models
Mechanistic Interpretability
LM Generation Steering
Syntactic Code Completion
Primary area: deep_learning

Abstract: Despite remarkable advances in coding capabilities, language models (LMs) still struggle with simple syntactic tasks such as generating balanced parentheses. In this study, we investigate the underlying mechanisms behind the persistence of these errors across LMs of varying sizes (124M–7B) to both understand and mitigate the errors. Our study reveals that LMs rely on a number of components (attention heads and FF neurons) that independently make their own predictions. While some components reliably promote correct answers across a generalized range of inputs (i.e., implementing "sound mechanisms''), others are less reliable and introduce noise by promoting incorrect tokens (i.e., implementing "faulty mechanisms''). Errors occur when the faulty mechanisms overshadow the sound ones and dominantly affect the predictions. Motivated by this insight, we introduce RASteer, a steering method to systematically identify and increase the contribution of reliable components for improving model performance. RASteer substantially improves performance on balanced parentheses tasks, boosting accuracy of some models from $0$\% to around $100$\% without impairing the models' general coding ability. We further demonstrate its broader applicability in arithmetic reasoning tasks, achieving performance gains of up to around $20$\%.

Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones
Daking Rai George Mason University drai20gmu. edu
Samuel Miller George Mason University smille200gmu. edu
Kevin Moran
University of Central Florida kpmoran@ucf edu
Ziyu Yao George Mason University zi ziyuyao@gmu edu
Abstract
Despite remarkable advances in coding capabilities, language models (LMs) still struggle with simple syntactic tasks such as generating balanced parentheses. this study, investigate the underlying mechanisms behind the persistence of these errors across LMs of varying sizes (124M-7B) to both understand and mit- igate the errors. Our study reveals that LMs rely number of components (attention heads and FF neurons) that independently make their own predictions. While some components reliably predict correct answers across generalized range of inputs (i.e., implementing "sound mechanisms"). others are less reliable and introduce noise by promoting incorrect tokens (i.e., implementing "faulty mecha- nisms"). Errors occur when the faulty mechanisms overshadow the sound ones and dominantly affect the predictions. Motivated by this insight, we introduce RAS- TEER, steering method to systematically identify and increase the contribution of reliable components for improving model performance. RASTEER substantially improves performance on balanced parentheses tasks, boosting accuracy of some models from 0% to around 100%, without impairing the models' general coding ability. We further demonstrate its broader applicability in arithmetic reasoning tasks, achieving performance gains up to around 20%
Introduction
Recent years have seen remarkable progress the code generation capabilities of language models (LMs), driven by an increase in model size, training data, and improvements in overall training 40 Yet, despite these advances, LMs continue to struggle with basic syntactic tasks such as generating balanced parentheses and correct indentation [12 45). The failure LMs to perform these seemingly simple tasks stands in stark contrast to their improved performance more complex coding benchmarks [21] 9 4. raising important interpretabil- ity question: How do LMs internally compute predictions for syntactic tasks and why do these computations sometimes fail?
methodologies 17 27 24 15 36
In this work, we seek understand this failure by investigating the internal mechanisms of seven LMs, ranging in size from 124M to 7B parameters, while they perform the balanced parentheses
'The source code and dataset for the paper is available at hwtp:::////gtttbcomooooo fai failure-a by- interference
NL---NLP--- Lab/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

task, the task of predicting the correct number of closing parentheses in code statements. Recent work has attempted to reverse-engineer the full mechanisms of LMs 30 33 5]; however, even
simple tasks were found to demand a bag of rather complicated computations 28 25 which makes the bottom-up reverse engineering challenging. In our work, we instead understand the failure of an LM in top-down manner, where we look for LM components, including attention heads and feed-forward (FF) neurons, that directly contribute to the final logit calculation of an LMM. Our study shows that LMs employ set of components, each with varying generalizability and reliability, to perform the balanced parentheses task. While most such components demonstrate high accuracy only within narrow range of inputs and add noise in others, we still identify rare set of highly effective and generalizable components. For instance, we find that a single attention head (L30HO) in CodeLlama-7b [36] outperforms the full model on our synthetic balanced parentheses dataset, highlighting the presence of strong, underleveraged mechanisms within the model. Key insights we derive from these findings are: (1) LM doesn't I rely a single mechanism to make prediction but many mechanisms with varying levels of reliability, and (2) LMs do not fail due to the absence of sound mechanisms, but rather due to the presence of too many faulty mechanisms that introduce noise and overshadow the sound ones.
Building on these interpretability insights, we propose RASTEER, approach that RAnks LM components based on their reliability and STEERS generation by increasing the contribution of more reliable components to the final logits, for improving model performance. Despite the existing work in LM steering [35] [2], none has explored steering for tasks that are multi-class, position- sensitive, and have clear "steering directions", as the balanced parentheses task. Applying RASTEER led to dramatic performance improvements across all seven models we studied, boosting accuracy on the balanced parentheses task from 0% to 100% for some models. To assess the broader impact of RASTEER, we evaluate whether steering for the balanced parentheses task affects the general code generation capabilities of models. On the HumanEval benchmark [9], we find that RASTEER preserves performance and even yields a modest improvement of 5.49% for Llama2-7b [17]. Finally, we also show the effectiveness of RASTEER beyond balanced parentheses tasks by applying an arithmetic reasoning task, where it achieves a performance gain of up to 20.25% for Pythia-6.9b (6)
2 Preliminaries
2.1 Background: Transformer-basee
Language Models
Transformer-based LM 4 maps an input sequence of tokens X = (.11..., (.,) to probability distribution over the vocabulary V and predicts the next token In+1, typically by sampling selecting the token with the highest probability. Initially, each input token æ is mapped to an embedding vector using a learned embedding matrix WE E R [ where d is the model dimension, combined with positional encoding. The resulting representation r initializes the model's ss residual stream, which is refined sequentially across layers. At each layer E (1,..., I.). the residual stream representation updated sequentially by two sub-layers: multi-head self-attention (MHSA) sub-layer followed by feedfforwrrd (FF) sub-layer:
e r + MHSA' (LayerNorm(r")), r+1 = pd + FF (LayerNorm(f"))
Multi-head Self-Attention (MHSA) The MHSA sub-layer consists of multiple attention heads, indexed by head h and operating in parallel. Each attention head performs distinct computation that contributes additively to the residual stream. Specifically, the h-th attention head at layer E computes:
H (t. K K& (r..).W....
(2)
where Quht- WWEh Kch w-eh Key and Le r' W
Vatue are the query, key, and value W.S.
Query'
matrices computed from the input re using learned projection matrices W& Query: W&A Key: Value E
dhedd and w E Ihchaaxd are learned projection matrices specific to head (e, h), and dhead is the dimensionality per head.
Ferd-forward (FF) Sub-layer The FF sub-layer at each layer E consists of a two-layer ford-forward network:
dg Σ
dp
(F'( = oFF
o k$)
1=1
2

---

where be E R is a column of the input projection matrix W E Rdxde and of E R is a column of the output projection matrix W& E Rdxx The bias term is omitted for simplicity. The activation function (-)) is typically non-linearity such as GcLU or ReLU. We follow Geva et al. [ to decompose the computation, where of is an input-independen parameter, which is referred as an FF neuron in this work, and m a (F* is an iaput-dependent coefficient representing the activation strength of the FF neuron.
After the final layer L, the model computes unnormalized logits over the vocabulary using the last-token residual stream output: logits = r We. where Wt E [Rdx[V] is the unembedding matrix.
2.2 LMs Failed in Naive Syntactic Code Completion
Recent empirical studies [2 45 have found that subset of LM code generation errors stem from failures to accurately complete basic syntactic structures like the balanced parentheses task, an issue that persists even in state-of-the-art models such as GPT-4 [2] and Phi-3 [I] To systematically study this LM behavior, we decompose the task of balanced parentheses as a collection of sub-tasks, determined by how an LM tokenizer processes sequences of N closing parentheses. P Specifically, every LM tokenizer in our study represents one, two, three, and four closing parentheses as single tokens, while sequences with N > 4 are split into multiple tokens of these one to four closing parentheses tokens. As a result, we define four sub-tasks within the broader balanced parentheses task, corresponding to N = 21, 3, 4], and synthesize a separate dataset for each. Specifically, we synthesize dataset using the following template for each sub-task:
One-Paren: #print the string ChummHnpring
{num}>> +
Two-Paren: #print the string (nuu))lpriiin (str ([uum] + ))
Three-Paren: #print the string (nemm))nrrine (nprinn (st (nnmm) }))
Four-Paren: #print the string (num))nprinn (str (str (str (esttr((1un (fnum) - )))
For each sub-task, we generate 350 training, 150 dev, and 150 test examples, each consisting of the input prompts created by randomly selecting a numeric value (num) } from the range 100 to 999 and
ground-truth output tokens. We select seven Moodels-GPT-5 Small [3. GPT-2 Medium B1. GPT-2 Large B1, GPT-2 XL B1), 7odeLlama-7b [36). Llama-2 7b 40), and Pythia-6.99 6 based on combination of factors: model size (ranging from 117M to 7B parameters), model family (including
both general-purpose purpose base models and Code LMs), and performance diversity (capturing broad spectrum of accuracy from 0% to 100% on our synthetic dataset). Specifically, all models had 100% accuracy for the one and two-paren task. However, most models exhibit low accuracy on the three-paren and four-paren sub-tasks, particularly all GPT-2 models had 0% accuracy on the four-paren sub-task. Full results can be found in Appendix A
3 Understanding LMs in Making Balanced Parentheses Errors
3.1 Overview
To understand why an LM makes (in)correct predictions, we investigate LM components (i.e., attention heads and FF neurons) that directly contribute the final logit of the model from the last-token position. By focusing on these components, we abstract away the need to analyze the full underlying mechanisms, which can be highly labor-intensive and complex. Crucially, since all internal computations must ultimately influence the model's output through these final components, examining them can still provide necessary insights to understand when and why a prediction goes wrong. Specifically, we will apply Algorithm to identify components that selectively promote the correct token over the distractors, and Algorithm 2 to identify components that promote the correct token with a thresholded strength. We consider LM components that selectively promote correct tokens with reasonable strength across generalized contexts as reliable contributors that implement sound mechanisms, and others as unreliable contributors implementing faulty mechanisms.
The two algorithms both utilize the logit lens technique (29), which projects the corresponding component activation to the vocabulary space. For attention heads, the component activation h, is
2 Empirically, when an LM does not generate parentheses following the way how its tokenizer works (e.g.. predicting "))" n first and expecting another "))", rather than predicting "))))", it can hardly succeed.
3

---

Algorithm 1 Measuring Task Correctness for LM Components
Algorithm 2 Labeling Token Promotion for LM Components
Require:
Component activation he E RR unem- bedding matrix Wo E Rdx ground-truth token t V, and distractor tokens Theg C
Require:
Component activation h, E R unem- bedding matrix Wu E Rdx[v), target token
e V, and promotion threshold r [0,
1: Compute logits: 1. h; We ER" ER
1: Compute logits: 1. b, Wo RIV)
2: if I.[4] max(lc[Tneg] then
2: if le[t] 2 7 max(le) then
return True
Marked as correct
3:
return True
b token labeled
4: clse
4: else
5:
refurn False
5:
return False
6: end
6: end if
H.S:;: for FF neurons, mfu.. Given the large number FF neurons each model (c.g., 131,072 in CodeLlama-7b)) perform static pre-filtering step to exclude neurons that unlikely to affect the target tokens. Specifically, we follow prior work [16] to interpret an FF neuron by projecting its parameter the vocabulary space using the unembedding matrix, i.e., of Wy. We then retain only neurons whose projections include at least one of the four closing- sessingparentheiss tokens among their top-50 or bottom-50 logit-ranked tokens.
3.2 LMs Developed Mechanisms of Varying Levels of Generalizability
We start with measuring the task accuracy of each LM component on each sub-task. Specifically, for every input prompt on sub-task training set, we follow Algorithm[]] to measure the component's logit projection yields the highest value the ground-truth token among all four token choices. In other words, we check the component can correctly promote the correct token more than the incorrect ones. Based on the correctness counts, calculate the accuracy of the component for each sub-task. We visualize the accuracy distributions of the attention heads of Codel in Appendix B.I] We observe that most models consistently have more high-accuracy (e.g., greater than 0.7) components for the one-paren sub-task, with the count gradually decreasing for the two-paren, three-paren, and four-paren sub-tasks This trend closely mirrors the overall performance of the models on each respective sub-task.
Based on the observed accuracy distributions, decide to group LM components based the number sub-tasks in which they achieve high accuracy, using 0.7 as a threshold. Specifically, if a component achieves least 0.7 accuracy in than sub-task, (i.e., serving as the prediction head of) a "sound mechanism" that generalizes across sub-tasks. Table reports the number of attention heads and FF neurons that generalize to different numbers of sub-tasks. Our analysis reveals that most components are specialized, attaining high accuracy on only single sub-task, while a smaller subset generalizes effectively across multiple sub-tasks. These results indicate that LMs implement a diverse set of mechanisms to solve each sub-task, with varying degrees of generalizability.
interpret
implementing
I
Table 1: Number of LM components (attention heads and FF neurons) that have high accuracy (2 70%) across different numbers sub-tasks.
Attention Heads
FF Neurons
Model (# heads, # neurons)
task
2 tasks
tasks
4 tasks
task
2 tasks
3 tasks
4 tasks
GPT-2 Small (144, 9,216)
11
I
0
15.5
0
GPT-2 Medium (384, 24.576)
10
2
0
235
0
0
GPT-2 Large (720, 46, 080)
18
4
0
365
1
0
0
GPT-2 XL (1,200, 76,800)
38
3
0
626
3
0
lama-7b (1,024, 131,072)
27
6
1
852
6
Llamu2-7b (1.024. [31,072)
18
2
0
391
5
0
0
Pythia-6.9b (1,024, (1,022 131,072)
23
4
0
1893
2
0
0
'The accuracy threshold of 0.7 was selected based on empirical observations of accuracy distributions across tab-tasks (Figure 4. We found that this value effectively separates components with high accuracy in at least sub sub-task from others.

---

Dadm--.044
Castebi73
Caste1
Copled
Average Logit
Lage e12
Laga c1
One Paren Input.
Two Paren Inpul
Three Paren Ingul
Four Paren Inpul.
Figure 1: Average logit values and coefficients of FF neuron L19N11 of Code when the input prompts demand one-, two-, three-, and four-paren closing tokens.
Generalization Capability of Attention Heads We further look into attention heads that generalize across multiple sub-tasks. These heads are listed in Table 4 We observe that attention heads with stronger generalization tend to emerge in deeper layers the model. Notably, for CodeLlama- 7b, attention head 0 at layer 30 (L30HO) achieved almost 100% accuracy across all sub-tasks, which was even better than the full model (96.00% on average). In Table we see that the generalizable attention head primarily attends to the first function name token (L30HO) open parenthesis (L.0HH66 that has yet to be closed across all the sub-tasks. comparison, the attention scores of a non-genenn non-generalized attention head are largely spread across different prompt tokens, with the major attention being placed on the hegin-of-sentent token (omitted in the visualization).
Generalization Capability of FF Neurons Prior work looking at only the top-k tokens sorted by their logit scores under the input-nndependent projection of of WW. However, contrary this conjecture, our analysis reveals that LMs do not rely solely on the top-k loghts-rather they also utilize bottom-k logits through a dual-sign mechanism. Specifically, neurons apply positive coefficients to promote top-k tokens and negative coefficients suppress them or, equivalently. to promote bottom-k tokens. This mechanism allows single neuron to support two distinct sub-tasks when its input- indeeendeeedntt projection contains relevant tokens at both extremes. For example, neuron 11 in layer 19 (L19N1 1) CodeLlama has ")" among its bottom- 10 tokens, and ")))", and "))))" among its top-50 tokens, after the input-independent projection. As shown in Figure D the neuron promotes the bottom token ")" for one-paren inputs by assigning it a negative coefficient (on average, 11004), but promotes the top token ") ) for four-paren inputs by assigning it positive coefficient (on average, 1.29). Being able to "flip" the coefficient depending on the input prompts allows this neuron to promote the correct tokens generalizably across both the one-paren and the four-paren sub-tasks. However, because both ")))" " and "))))" are ranked as top tokens for L19N11, when the neuron promotes "))))", ". inevitably also promotes ")))". We term this phenomenon noisy promotion, meaning that the FF neuron has to promote the ground-trute -tuth and the distractor tokens at the same time. This observation highlights both an architectural constraint and an adaptation developed with FF neurons because FF neurons can only rank tokens on either the top side or the bottom side of its parametric memory, can generalize to at most two sub-tasks; however, it attempts to overcome this limitation by developing noisy promotion strategy.
32 interpreted an FF neuron mainly by
3.3 LMs Predict via Noisy Promotion and Low Selectivity
To further understand the "noisy promotion" effect of LM components, we conduct the second analysis to look at the recall and precision of LM component's promotion. Unlike previous analysis, which focuses on whether an LM component comparatively ranks the correct token with a higher logit than the other three distractors (Tneg), this analysis checks the absolute logit value projected by LM component to each answer token. Specifically, recall measures whether component promotes the correct answer token for the associated inputs (irrespective of whether the distractors are promoted), and precision measures whether the component promotes a token only when the token is the true answer.
We use Algorithm 2 to identify which tokens each LM component promotes, based on a fixed promotion threshold т, set to 0.5 in our analysis (see Appendix B.2 for details on setting the promotion threshold to 0.5). To compute recall and precision for each sub-task, we construct balanced dataset containing equal numbers positive and negative examples, using the same training set introduced in Section For instance, in the one-paren sub-task, half of the examples require ")"
5

---

as the correct token sampled from the one-paren train set, while the remaining examples are sampled evenly from the other three sub-tasks.
Figure 2 presents the precision-recall scatter plots, averaged across all sub-tasks, for all attention heads and FF neurons in CameL-ama-7h and GPT-2 Small; results for other models shown in Appendix B.3 Our analysis reveals that most components exhibit both low precision and low recall. small subset achieves high recall, but notably, we observe an absence of components with high precision-highlighnnn a widespread lack of selectivity across models. This observation suggests that LMs rely on large number of components and make heavy use of noisy promotion, where components often activate for both correct and incorrect tokens boosting the correct token more strongly, but not exclusively. Consequently, predictions emerge from the aggregate effect of many low-stlectivity components, rather than from a small number of highly precise ones.
Procision
Recall
Recall
Recall
Recall
(a) CodeLlama-7b
(b) GPT-2 Small
(c) CodeLlama-7b
(d) GPT-2 Small
Figure 2: Scatter plots of average precision vs. recall across all sub-tasks for attention heads (left two) and FF neurons (right two) of CodeL and GPT-2 Small.
Ranking and Steering LM Components for Performance Enhancement
Our analysis in Section 3 indicates that LMs developed both sound and faulty mechanisms for the balanced parentheses tasks, and that they make predictions via a noisy promotion strategy. We hypothesize that the errors of balanced parentheses do not stem from the absence of sound mechanisms, but rather from their influence being overshadowed by the faulty ones that introduce noise into the model's 's computation. Inspired by the hypothesis, we propose RASTEER, an approach aiming to improve an LM's performance on the balanced parentheses task by first RAnking LM components based on the soundness of their mechanisms and then STEERing these components to augment their effect, so as to get rid of the mechanism overshadowing.
4.1 RASTEER: Ranking and Steering LM Components
We rank the LM components in the following order. First of all, we group LM components based on their generalizability (Section 3.2) and sort them based the number of sub-tasks they generalize to in descending order. Then, within each generalizability group, we further sort components based on their promotion effect averaged over all the sub-tasks (Section 3.3) In experiments, we consider recall, precision, and FI as the metrics and decide the most effective one based on the experimental results. Like in our analysis, the sorting process utilized training set for each sub-task.
Given a sorted list of LM components, we perform LM steering to increase the impact the top-k components on the final prediction. Specifically, for each selected component c, we scale its activation hg by multiplier a [1.1, 2.0] before adding it to the residual stream.
4.2 A Strong Baseline: Ranking LM Components from Circuit Discovery
In contrast to our top-down approach of only finding LM components that have strong contributions to model's prediction, recent research Mechanistic Interpretability (MT) 33
5
14 takes an
bottom-up approach to identify a complete mechanism (particularly, circuits
28
) for LM
behaviors Given its promise, we add a circuit baseline. Specifically, we discover one circuit for each sub-task following the activation patching of Nikankin et al, (28). which focuses on localizing causally important attention heads while retaining all FF layers We then rank each attention head
based on its patching effect averaged over sub-tasks. Steering is only performed to attention heads. We leave details of our circuit discovery in Appendix C
We do not localize both attention heads and FF neurons due the prohibitively high computational cost.

---

Accurs
C77 orn
GF7 cord
gr2id
Number of Attention Heads
Number of FF Neurons
Number of FF Neurons
(a) CASTEER-Attention Heads
(b) RASTEER-FF FF Neurons
(c) Circuit Baseline
Arcurary
ga24irge
pyttiadd5 55
Number of FF Neurons
Number of FF Neurons
Number of FF Neurons
(d) RASTHER-AAtention Heads
(e) RRASTEER-FF Neurons
(f) Circuit Baseline
Figure 3: Performance of RASTEER (steering either attention heads or FF neurons) and the circuit baseline on the three- three-paren (top) and the four-paren (bottom) sub-tasks. When zero heads or neurons steered, it shows each modell's 's raw performance without steering. Results of RASTEER one- paren and two-paren sub-tasks and when steering both attention heads and FF neurons are provided in Appendix D.T
5
Experiments
5.1 Experimental Setup
Dataset We mainly evaluate our approach on the test set of balanced parentheses for each sub-task, which consists of 150 examples drawn from the same synthetic configuration but disjoint from the training set, as described in Section [ In addition, to verify that the steering intervention does not degrade the model's general coding performance,
also evaluate our approach on IuumanEval (9), a standard benchmark dataset for evaluating the coding capability of an LM. For both datasets, we report the accuracy of the model before and after steering.
Approaches and Configuration We experiment with three variants when applying RASTEER to steer (I) only attention heads, (2) only FF neurons, and (3) both attention heads and FF neurons. Notably, we promote the same set of components for all sub-tasks as we rank LM components based on an averaged effect over all sub-tasks; because, in practice, we would not foresee the category of sub-task that an upcoming input falls into. Similarly, steer attention heads across all token positions, since do not assume prior knowledge of the position at which the model must generate a closing parenthesis. In each variant, we vary the number of top-ranked components steer and observe impact on the model performance. For the circuit baseline, since we only calculate effect scores for attention heads, the steering experiments conducted attention heads only. For all steering experiments, the optimal scaling multiplier (a) for each model. Model inference is performed via greedy decoding for stability.
expect steering enhance all sub-tasks. The demand is also
the dev set determine
5.2 Main Experimental Results
RASTEER provides dramatic improvement on three-paren and four-paren sub-tasks In Figure 3 we present the effect of RASTEER on the three-paren and four-paren sub-tasks, when steering the top attention heads the top FF neurons. On the one-paren and two-paren sub-tasks, all models achieved almost perfect accuracy pre-steering and were not impacted by steering. Our approach, by promoting more reliable components at the top, leads dramatic performance gains on the balanced parentheses tasks. For instance, GPT-2 XL. initially achieved 0% accuracy on three and four-paren sub-tasks, yet promoting only the top-10 10 attention heads results an improvement to ~ 100% accuracy. Similar trends were observed across all models: promoting the top-60 attention heads leads to 100% accuracy of models on all sub-tasks, with the exception GPT-2 Small on the four-paren sub-task (see Appendix D.2 for analysis of why RASTEER fails to improve GPT-2 Small in the four-paren
7

---

sub-task). Interestingly, we observe that larger models generally require fewer component promotions to achieve strong performance. For example, in the four- paren sub-task, only GPT-2 Small and Medium required more than the top-10 10 attention heads for substantial improvements, while all other models achieve ~ 100% accuracy by promoting just the top-10 10 attention heads F
Attention heads promotion yielded better performance than promoting FF neurons We observed that promoting only the attention heads achieved performance comparable to joint promotion of both the attention heads and the FF neurons, while promoting only FF neurons resulted in little to no improvement, particularly for smaller models. For example, in GPT-2 XL. promoting sixty FF neurons had no effect on the three-pane paren sub-task, whereas promoting just five attention heads improved model performance from 0% to 100%. A similar trend was observed across all GPT-2 models on the purr- sub-task. We posit that attention heads are more effective for steering because they are not structurally constrained in the same way as FF neurons, which appear to generalize to only a limited number of sub-tasks, as discussed in Section 3.2 Additionally, FF neurons may require coordination with other components influence predictions, rather than contributing meaningfully in isolation. Based on these findings, we restrict our subsequent experiments to the attention head steering,
RASTEER with components ranked by Fl-score has the best performance We ranked components using three reliability metrics: recall, precision, and Fl-score. We show the results based on Fl-score in Figure and others in Appendix D.3] Among these, Fi-score consistently yielded the best overall performance. Between recall and precision, recall outperforms precision aligning with our earlier analysis in Section 3 which suggests that LMs tend rely more on noisy promotion rather than highly selective, precise components.
RASTEER outperforms the circuit baseline As shown in Figure 3 RASTEER consistently out- performs the circuit baseline across models and tasks. While the circuit haseline yields comparable improvements in models with over billion parameters, it proves ineffective for smaller models. For example, in the four-paren sub-task, all GPT-2 pondels-excepp GPT-2 XL-failed to benefit from promoting the top-60 attention heads ranked by circuit analysis. Furthermore, performance even declined as more heads were promoted, suggesting that component rankings derived from circuit analysis do not reliably translate into effective steering strategies for improving model performance. To understand the discrepancy, we further analyze the reasons why the circuit baseline is effective for larger models (over iB parameters) but unstable for smaller ones. For larger models, we first examine the differences in attention heads selected by the circuit baseline and RASTEER. Our analysis shows that the degree of overalp vary between 20-100% as shown in Figure 9 and steering these overlapping heads yields mixed eesulte-effective for Llama-2 7b and lama--7, but not in Pythia-6.9b. indicating redundancy in useful mechanisms. For smaller models, we find that the instability mainly arises from steering attention heads that do not directly contribute to the final logits but are instead involved in intermediate computations, such as information transfer and feature extraction. Steering these additional heads drastically reduces performance, demonstrating that not all functionally relevant components are beneficial for steering. We include further discussion in Appendix D.4
5.3 HumanEval Results
To ensure that our LM steering for the balanced parentheses task does not adversely impact broader code generation capabilities, we evaluate the post-steering performance on the HumanEval bench- mark [9] using Jada-- Llama2-7b, and Pythia-6.9h. We did not experiment with GPT-2 models because they had 0% accuracy on the HumanEval benchmark. Steering the top-20 attention selected from the parentheses task and scaled with a multiplier range of 2.0). did not degrade HumanEval performance and even led to a 5,49% improvement for Llama2-7b: CodeL- lama (30.48% + 29.87%). Llama2 (1 (11588% + 17.07%). Pythia (10.36% 10.97%). However, extending steering beyond the top-20 heads resulted in performance degradation: CodeLlama-7b and Pythia-6.99 experienced modest decline of 1-1.5% when up 60 heads were promoted, whereas Llama2-77 7b maintained a slight gain with best performance while promoting just top-20 attention heads. These results suggest that targeted promotion of a small set of reliable components
*We have also experimented with the more recent Qwen2.5-3b model 38 and shown that RASTEER (applied to top-20 attention heads) can improve its accuracy on Three- and Four-Paren subtasks from 92.66% to 98.00% and from 46.66% 77.33% respectively.
8

---

can improve task-specific performance without compromising general capabilities, while broader interventions may lead to diminishing returns or adverse effects.
5.3.1 Does RASTEER Generalize to Arithmetic Reasoning?
We further assess RASTEER Pythia-12b, and Llama3-8h [3 We consider a two-operand arithmetic reasoning task, for which we construct dataset comprising 750 training. 350 dev, and 350 test set examples. Each input prompt consists of four tokens; the first operand, an operator (+, -, x, +), the second operand, and an equals sign. To construct the dataset, we randomly sample integers in the range [0, 500] for both operands and ensure that the resulting answer also falls within this range to ensure single-integer tokenization. Because the task does not have sub-task categorization similar the balanced parenthese task, when applying RASTEER, we only ranked LM components based on their promotion effect on the training set, following Algorithm 2 We chose recall as the promotion effect metric as we observed a similar noisy promotion strategy in this task.
arithmetic task using three models: GPT-2 XI., Pythia-6.9b,
As shown in Table 2 RASTEER yields performance improvements across most arithmetic operations for all three models, with the largest gain of 20.25% % observed in Pythia-6.9b for multiplication. These results show that our method can potentially generalize beyond the balanced parentheses task and also indirectly support our central insights from Section 3 LMs rely on a large number of components with varying reliability, generating predictions through the noisy loken promotion of these components.
Table 2: Performance of RASTEER when steering top-30 attention heads sorted by recall on two- operand arithmetic tasks
Model
Addition
Subtraction
Multiplication
Division
GPT-2 XL
0.00%
0.00%
0.00% + 0.00%
17.50% + 17.50%
19.66% + 23.66%
Pythia-6.96
25.41% %.
31.66%
0.00% + 2.50%
14.41% + 34.66%
19.91% + 33.755
Pythia-12b
17.33%
24.66%
6.00% + 7.33%
8.00% + 21.00%
27.00% + -36.33%
Llama3-8b
82.33%
84.33%
87.91% - 88.00%
74.08% - 80.08%
78.16%
+ -79.91%
Related Work
LMs for Code Generation and Syntactic Failures Recent advances in LMs have led to significant improvements in their code generation capability 17 2 24 (3] However, LMs are still prone to range of semantic and syntactic errors, including balanced parentheses, and indentations [ 45 To investigate these findings, our work focuses on the balanced parentheses task as a representative task to study the internal mechanisms of LMs for the syntactic task. Furthermore, we also explore whether we can leverage understanding from our interpretability study to mitigate these failures without harming overall model performance on broader code generation tasks.
Mechanistic Interpretability (MI) MI is subfield of interpretability that aims to reverse-engineer
the algorithms learned by model 25 7 14 30 32). In study, employ techniques and insights from MI to identify and study LM components that implement the balanced parentheses task. However, in work, we do not aim to fully reconstruct the underlying algorithm or circuit responsible for this behavior. This is primarily because our analysis in Section 3 iidiccated that the model does not rely impractical to interpret each mechanism in detail, in line with recent work findings that LMs often employ multiple mechanisms for tasks such as arithmetic reasoning (28) and factual recall [10].
single mechanism, but rather a large number of components, making it
LM Generation Steering Steering LM generation has recently emerged as a popular lightweight approach for controlling model behavior at inference time 135
26 Most approaches use
steering vectors to clicit high-level traits like honesty [47), truthfulness 22 sycophancy (35). or sentiment [39], and have been applied to enhance capabilities [46]
26 42 and support red- teaming 3 34 In our work, we do not use the steering vectors approach, as it is not clear what vectors to look for a multi-class, position-sensitive task like balancing parentheses. Instead,
"Other GPT-2 models are excluded due to their inability to perform the task, while CodeLlama-70 7b and 1lbma2-7b are omitted because they tokenize complete integer into multiple tokens of digits, making the token prediction-basee based analysis difficult. Following Nikankin et al. (88), avoid such models.

---

focus on identifying and promoting specific model components that contribute reliably to correct predictions. While activation steering has been extensively studied, steering generation through the promotion of individual model components remains relatively underexplored [66 23
7 Limitations and Conclusion
In this work, we study the mechanisms for why LMs make errors in simplistic balancing parentheses tasks and also propose steering approach to improve them. While RASTEER substantially improves LM performance on both balanced parentheses and arithmetic reasoning tasks, our study conducted using synthetic data may have exaggerated its effectiveness. Additionally, analyze LM failures the paannteessa task, which can be cleanly categorized into four distinct sub-tasks depending on how the LM tokenizes different numbers of closing parentheses. However, this decomposition may not generalize to all tasks, especially those with a much larger set of possible output tokens. Although we demonstrate that RASTEER remains effective when the output space expands, as in arithmetic reasoning, there may exist other tasks-suchs as general reasoning maasonin--whree the output space is even larger or less well-defined, potentially limiting the approach's 's applicability. Besides the task setup. our approach assumes that LMs follow a simple additive motif, where the final logit is formed by simply adding the contributions from individual components. While this assumption also supported by several prior findings [TO] [4] and further reinforced by our results with RASTEER, may overlook non-additive mechanisms such as those that actively suppress noise. Investigating such dynamics remains important direction for future work. Our method also relies on simple heuristics: components are ranked using recall, precision, and Fl-score, and promoted via fixed scalar multipliers. Future work could explore more sophisticated techniques for both ranking and promotion. Lastly, rather than reconstructing full mechanisms or circuits, we focus on steering components that directly influence the final logit. We believe that this form of functional interpretability, centered on understanding high-impact LM components for practical applications, remains underexplored.
Acknowledgments and Disclosure of Funding
This project was sponsored by the National Science Foundation (Award Number 23 1468/2423813) The project was also supported by GPU resources provided by the Office of Research Computing George Mason University (URL: https://tt .omoor...ggeeee. gm.. edu) and funded in part by grants from the National Science Foundation (Award Number 2018631).
References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint 2rXiv:2404.142) 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[3] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models mediated by single direction. arXiv preprint arXiv:2406.1 17717, 2024.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. 2021.
[5] Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety re- view. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL hntps //openreiiew.o patpenrrvieeennet/fruumiiiiaeeae htt.://ooororiiireeeeeeeeee Survey Certification, Expert Certification.
[6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward
10

---

Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning. pages 2397-2430. PMLR, 2023.
[7] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-oddss Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transsormer-- http///222///ooommmmmm.
[8] Lawrence Chan, Adria Garriga-Alonso, Nicholas Goldowsky-Dill Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: A method for rigorously testing interpretability hypotheses. In AI Alignment Forum, volume 2, 2022.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman. Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[10] Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in Ilms. arXiv preprint arXiv:2402.07322 2024.
[11] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209. 02202222990055,, 2022.
[12] Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, al. What's wrong with your code generated by large language models? an extensive study. arXiv preprint arXiv:2407.06153 2024.
[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[14] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transsfomer-ciint cirruutsseerrrrrtsr.eireaie httpcuuitsss.p//222222 pub/2021 //tra/ewrk/nndex.mmmm
[15] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.0555
2022.
[16] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer forw-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
[18] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute theat:r-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76060, 2023.

---

[19] Michael Hanna. Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going be- yond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URI. http:p/:/pppaaaeeeeeeeeeer/rrrra
[20] Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404. 2024.
[21] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, 2024.
[22] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451-41555 2023.
[23] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL
net/
forum?id=alLuyppaa
[24]
Raymond Li, Loubna Ben allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltomozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URIhhttpr httpttttt///eeeeie http://pppeeeee.eeeeee net/ forum?ormmm idd= KoFUg41haE Reproducibility Certification.
[25] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson. Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https //trans wwwee-
pub/2025/
211/222222///// /tt//tatribiiionn attribbtton-grapps// biology.hhhi .ttll
[26] Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, pages 32287-32307, 2024.
(27] Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Code- gen2: Lessons for training Ilms on programming and natural languages. arXiv preprint arXiv:2305.02339 2023.
[28] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algo- rithms:
Language models solve math with bag of heuristics. In The Thirteenth International Conference on Learning Representations, 2025. URL httpps:/ nwt.
id=09YTt:2rrP
[29] nostalgebraist. Interpreting gpt: the logit lens. Al Aligument Forum, 2020. http://gg..oooooooe//////oaaaaas
https::/ /ww
the- 1oo-toogittt lens
[30] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to cireuits. Distill, 5(3)e00022 001, 2020.
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
12

---

[32] Daking Rai and Ziyu Yao. An investigation of neuron activation as unified lens to explain oo-in--f-thhghhh eliciting arithmetic reasoning of llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7174-7193, 2024.
[33] Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mech- anistic interpretability for transformer-based language models. arXiv preprinl arXiv:2407.0244 200.226677. 2024.
[34] Nina Rimsky. Red-teaming language models via activation engineering. Accessed: October, 13:2023, 2023.
[35] Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering Ilama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 15504-15522, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.11653//1/2224.all acl.oo...cl-.o...2...
URL hatop:://adaattaaloooooo//////// acw--//0.......1ooooo......... 828/
[36]
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[37] Aaquib Syed, Can Rager, and Arthur Conmy. Autribution patching outperforms automated cireuit discovery. In Proceedings of the 7th BlackboxNLPP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407-416, 2024.
[38] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm. qwenlm.github..
wwt.inigiuuuu....oooooo.o.. ht/blog///////oo hwwpl//gggg...//////o/////. qwen2.5) 5/
[39] Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. CoRR, 2023.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[41] Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. CoRR, 2023.
[42] Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours. arXiv preprint arXiv:2403.0576: 2024.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https
7/ //openreview www//eeruuntit Mett:formm7ddNNpssVNNoo11
[45] Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. Where do large language models fail when generating code? arXiv preprint arXiv:2406.087. 2024.
[46] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems,
2024.
[47] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Knn-Kathrin Dombrowski, al. Representation engineering: top-down approach to ai transparency. CoRR, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make two central claims: (1) LMs rely set of mechanisms, rather than a single unified mechanism, to make predictions, and these mechanisms vary in their reliability; (2) Model errors often occur when the LM disproportionately relies on less reliable mechanisms. In Section 3 we support these claims by measuring accuracy, recall, and precision of individual components, and in Section 5 we show that promoting the top-k most reliable components via RASTEER leads to substantial performance gains.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of our work in Section 7 The main limitation of our paper is that we experiment with a synthetic dataset, and the effectiveness of our approach may be exaggerated. We also discuss several other limitations and future work, as our finding leads several interesting future directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not have any theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the information required to reproduce the main experimental results, including the data synthesis process, metrics calculation, and other hyperparameters used for the experiment the paper itself in Section 3. Section 4 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide open access the data and code, along with detailed instructions enable faithful reproduction of the main experimental results, available at
com/
Eiyu-Yao-ae WWW-NNL----aaalnnae -atba//aaalueeb---mmeeeeeeeee interference
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr cc/
Segllic/uides/Codeeee (odeeSSumissiiooool11..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ptaiss.cc/uublicguuiddes SeweeeeeeeeooeSeeeeeeeeeeeeee setpsubmiisssioiiii swioiilicoyyycccooocooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, we provide all the training and test details, including the data synthesis process, number of train, test, and dev set splits, and hyperparameters used in the experiments
in Section Section 3 Section 4 and Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We do not report any results with statistical significance.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the information about the compute resource on Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httttp////eeuriie..cc//////////a
Answer: [Yes]
Justification: Yes, the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
17

---

Justification: This paper aims to understand the inner workings of LMs by examining how they perform the balanced parentheses task and why they sometimes fail. We view this as foundational interpretability research, with its impact expected to emerge through future applications interpretability, such as diagnosing unexpected model behaviors and enabling more effective control and steering of LMS to better serve user needs. As automatic code generation has become one of the core applications of LMs, we expect our approach, RASTEER, will benefit practitioners such software engineers who may use LMs along with our approach for more reliable code generation. We do not perceive any potential negative impact from this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose any such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification:
Code is implemented by us and we clarify all model licenses in Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce synthetic dataset for balanced parentheses completion. We include details of the dataset generation in Section
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: We do not perform crowdsoureing experiment and research with human subjects in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper doesn't 't involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not LLM for anything besides writing aid tool.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy attt:::/reaarips.cc//offeeeeeeee for what should or should not be described.
20

---

A
Model Performance on Balanced Parentheses Task
Table 3:
Accuracy LMs on the balanced parenthesis task, LM when the ground-trun truth token includes
one, two, three, and four closing parentheses, respectively.
Model
One-Paren
Two-Paren
Three-Paren
Four-Paren
GPT-2 Small
100.00%
100.00%
49.00%
0.00%
GPT-2 Medium
100.00%
100.00%
93.00%
0.00%
GPT-2 Large
100.00%
100.00%
88.00%
0.00%
GPT-2 XL
100.00%
100.00%
0.00%
0.00%
Llama2-7b
100.00%
100.00%
100.00%
1.00%
CodeLlama-7b
99.00%
100.00%
98.00%
87.00%
Pythia-6.9b
100.00%
100.00%
100.00%
83.00%
B Additional Results of LM Components Analysis
B.1 Accuracy Distributions of Attention Heads across Sub-tasks
The accuracy distribution of the attention heads for GPT-2 Small, GPT-2 Medium, GPT-2 Large, Llama2-7b, and Pythia-6.9b shown Figure
B.2 Selection of Promotion Threshold
To determine an appropriate promotion threshold, we initially selected 0.5 as a reasonable starting point for the analysis in Section 3 and later validated this choice through tuning on the development set for RASTEER. Specifically, we evaluated thresholds in the range [0.4, 0.5, 0.6, 0.7, 0.8, 0.9) and found that 0.5 consistently achieved strong performance. For example, on the four-paren sub-task other thresholds,
with GPT-2 Small (top-60) 60 heads), 0.5 yielded 16.8% caeeracy--ouuerfoommii which ranged between 5.3% and 14.0%. Notably, we found that the promotion threshold had less impact on larger models (2 IB parameters), where promoting just small number of reliable components often sufficient for performance gains. In summary, we posit that the promotion threshold balances between false positives and false negatives: thresholds below 0.5 risk including too many components that not meaningfully promote the target token, while thresholds above 0.5 risk excluding components that genuinely helpful.
B.3 Precision vs. Recall Analysis Across Models
Figure the precision recall analysis across all sub-tasks for both attention head and neurons. For all models, most components exhibit both low precision and low recall. A small subset achieves high recall, but notably, we observe an absence components with high precision-hhihliiieeeee highlighting widespread lack of selectivity across models.
B.4 Analysis of Generalizable Attention Heads
In Table 4
list the attention heads that generalize or sub-tasks and FF neurons that generalize to more than one sub-task for each model. In Table 5 we present the attention visualization of three heads, generalizable and one not.
21

---

Table 4: List of attention heads that generalize to one or more sub-tasks and FF neurons that generalize to more than one sub-task.
Model
Generalizable Attention Heads
Generalizable FF Neurons
GPT-2 Small
LIH1 (1-paren), L.H4 (2-paren). 1.H66 (I-paren), L4H5 (2-paren),
N/A
I.5H3 (2-paren), 16H3 (3-paren). 1,7H6 (4-paren), 18H5 (1-paren).
LI0HB (2-paren), L10H5 (3-paren), L11I1 (l-paren), 191110 191110(I-paeen) (I-paren,
2-paren)
GPT-2 Medium
LIH7 (1-paren), L9H8 (3-paren). L13H10 (2-paren). L14H7 (2-puren), N/A
LI6H0 (2-paren), L16H3 (4-paren). L19H3 (2-paren). L19H9 (3-paren).
L2IHI (1-paren). 1.23H8 (3-paren), L.7H5 (I-paren, 3-paren), L18H6
(1-paren, 2-paren)
GPT-2 Large
LiHB (1-paren), L2H5 (1-puren), L31111 (1-paren), L4I117 (1-paren),
L29N1354 (I-paren, paren)
LSH117 (1-puren), L6H119 (1-paren), L8H11 (2-puren), L10H1 (3-paren),
LITH3 (4-puren). L12H8 (4-paren). LI2HI5 (4-paren), L13H10 (4-
paren). L14H3 (4-paren), LISH14 (2-paren). L18H11 (2-puren), L23H6
(2-paren). L.7H3 (2-paren), L30H9 (4- (4-paren). L22H15 (I-paren,
paren), 1.24H3 (1-paren, 4-paren). 1.25H4 (1-paren, 2-paren), 1.26H8
(2-puren, 3-puren)
GPT-2 XL
L11124 (1-paren), L2H113 (1-paren), L3110 (1-paren), L4H12 (1-paren),
L36N1149 (1-paren, 3-
(2 L.H22 (4-paren). L14H3 (4-paren). L14H18 (1- (1-paren).
paren). L36N5870 (1-paren. 3-paren)
LI5H9 (3-paren). L16H22 (4-paren), LISH13 (2- (2-paren) L18H23 (4-
paren). L19H4 (2-paren), L19H6 (4-paren), L21HH (4-paren), L21H5 (3-
paren), 1.21HI1 (4-paren), 1.22H13 (4-paren), 1,22H18 (4-paren), 1.23H3
(2-paren), L23H1 (4-par))) L24H15 (4-puren) L241124 (2-paren).
L25110 (2-paren), L261116 (3-paren). L261118 (4-parem). L27117 (2-
paren), L27H10 (2-paren), L29H3 (4-paren), L29H12 (29 L:001 L30H14
(2-paren). L31H18 (4-paren), L32HO (1-paren), L33H11 (2-paren).
L.3H14 (2-paren), L35H22 (2-paren). L41H19 (2-paren). L42H5 (1-
L13H17 (1-paren, 4-paren). L32H1 (2-paren, 3-paren). 142H16
3-parenn L30H16 (1 (I-paren. 2-paren, 3-paren)
CodeLlama-7b
LOHO (1-paren), H.IH6 (1-paren), LIH7 (4-paren), L8H29 (I-paren).
LI9NII (I-paren, 4-paren), 1.20N3998 (1-paren, 4- paren), L22N8326 (1-paren, 2-paren). L27N9695 (2- paren, 3-paren), L29N8515 (I-paren, 2-paren)
I.1HI6 (1-paren), 1.1HH1 (4-paren), L18H200 (1 (1-paren). 1,18221 (1-paren), L21H22 (2-paren). L22H4 (1 (1-paren). L241118 (1 (I-puren).
(1-puren, L25H5 (1-paren), L26H (1 parren), L27H14 (3-paren).
L28H22 (4-paren), L28H29 (3-paren). L29H7 (1-puren). L29H12 (1-
paren), L29H29 (1-paren), L30H11 (1-paren). L30H13 (1-paren), L31H4
(I-paren). L31H8 (4-paren), L31H10 (1-paren), L31H14 (1 (1-paren).
131H18 (I-paren). LI7H17 (1-paren, 2-paren), 1 1,27115 (1-paren, 2- paren), L271124 (1-paren, 2-paren), L28H11 (1 (I-paren, 2-parem), L29H0
(1-paren. 2-puren), L31H22 (3-puren. 4-paren), L30110 (1-paren. 2-paren, 3-paren, 4-puren) LIH30 (1-paren). L3H3 (1-paren). L17H17 (1 (1-paren). L18H21 (1-
Llama2-7b
L15N6063 (I-paren, 3-
paren), L.0022 (3-paren), L22H4 (1-paren), L24H18 (1 (1-paren), L25H5
paren). LI (1 (I-paren, 4- paren). L20N3398 (1 (1-paren,
(2-paren). 1.25H24 (4-paren). 1.27H24 (2-paren), 1,28H13 (1 (l-paren).
L281129 L29H10 (1 (1-paren), L31114 (1-paren), L31HS (4. (4-pare)),
4-paren), L27N5474 (2-
L31118 (2-puren). L31H21 (1-paren), L31H22 (4-pure)) L27H11 (1-
paren, 4-puren), L30N9014 (1-paren, 2-paren) L2IN13821 (I-paren, 4-
paren, 2-paren), L30HO (2-puren, 3-paren)
Pythia-6.9b
L4H21 (1-paren). L9H20 (3-paren). L10H14 (1 (1-paren). L10H21 (1-
paren). L.12H12 (3-paren). L13H7 (1-paren). L13H26 (2-paren), L14H9
paren). L25N2012 (1-paren, 4-paren)
(I-paren), LI4HH1 (2-paren), L15H9 (1-paren), L16H12 (1 (1-paren).
LI7H10 (1-paren). L18H6 (4-paren). L19H22 (1-paren), L22H1O (1- paren). L23118 (4-puren), L261110 (2-puren), L261113 (1-paren). L27HII (1-paren). L27H3 (3-paren). L27H18 (1-paren). L28H17 (I-paren). L29H15 (1-paren). L10HD (1-paren. 2-paren). L10H7 (I-paren, 2-paren). L29H17 (1-paren, 2-paren), L30H22 (2-paren, 3-paren)
Sub-Task
L30HO (CodeLlamma)
L33H16 (GPT-2 XL.
L.3H2 (Codel.lama)
One-Paren
#printthestring 8119 pmin
aprint the string 2311 prinf123)
#printthestrngg 8
1rrintt((8 199
de
Two-Paren
#printihesttinng@891 | print (6
#prent string san prrnti pon
Three-Paren
#printhhesttnng4444piititrr co (554 #print the string 245 print((as (str 2445
Table 5: Attention visualization of attention heads. L30HO of CodeLlama generalizes to all four sub-tasks. L30HI6 of GPT-2 XL generalizes to one-, two-, and three-paren. L30H2 of CodeLlama does not generalize to any sub-task.
22

---

Frequency
1.9
1.25
13
1,00
1.0
0.75
8.0
0.5
200.0
Accuracy
Accuracy
Accuracy
Accuracy
34
3.0
Frequency
1.0
0.5
0.5
600
0.2
04
06
3,5
100
3.0
0.2
04
06
0.8
1.0
20
0.0
0.2
0.4
0.6
08
1.5
0.0
2.2
0.4
0.
0.5
10
Accuracy
Accuracy
Accuracy
Accuracy
4.0
Frequency So
3.5
1.0
0.5
1.0
0.0
0.6
1.0
02
04
0.0
6.6
6.8
Accuracy
Accuracy
Accuracy
Accuracy
12:
Frequency
9.00
Accuracy
Accuracy
Accuracy
Accuracy
2.00
4.0
Frequency
1.50
3.35
355
3.0
1:25
3.25
0.5
9
02
06 Accuracy
0A4
08
1.0
0.80
0.25
3.00
0.55
1.00
2.00
0.0
0.
0A
0.6
08
1.5
0.0
0.0
0.2
0.4
0.5
100
Accuracy
Accuracy
Accuracy
Frequency
25
000
0.6 Accuracy
1.0
0.0
0.4
0.3
90.9
Accuracy
Accuracy
Accuracy
17.5
Frequency
12.5
3.0
30.0
2.5
50
00:00
Pgo
Accuracy
Accuracy
Accuracy
Accuracy
(a) One-Paren
(b) Two-Paren
(c) Three-Pares
(d) Four-Paren
Figure 4: The plots illustrate how attention head accuracy varies across sub-tasks across six models. Each row corresponds to a different model: top to bottom-GPTS Small, GPT-2 Medium, GPT-2 Large, GPT-2 XL, Llama2-7b. CodeLlama, and Pythia-6. .bb Attention heads with accuracy below 0.01 are excluded to avoid distortion of the distribution due to their high frequency.
23

---

1.4
10
we-
965
Recall
Recall
Recall
Recall
(a) GPT-2 Medium
(b) GPT-2 Large
(c) GPT-2 XL
(d) Llama2-77 7b
1.
03
Recall
Recall
Recall
Recall
(0) GPT-2 Medium
(f) GPT-2 Large
(g) GPT-2 XL
(h) Llama2-70 7b
Figure: 5: Scatter plots of average precision recall across all sub-tasks for attention heads (Top) and FF neurons (Bottom)
24

---

C Details and Results of Circuit Discovery
We follow the same activation patching approach of Nikankin al. 28 discover circuits for each sub-task. We only localize important attention heads while retaining all FF layers in the circuit our preliminary exploration,
to mitigate the high computational cost of the circuit experiments, computationally efficient approaches that try to approximate activation patching, such as edge attribution patching [3 19]. did not yield circuits with reasonable faithfulness scores.
Methodology Activation patching requires three forward passes a clean run, a corrupt run, and a patched run, to evaluate an LMM component's effect on the performance of sub-task, where the
predicted next-token of the clean run is the sub-task's s correct target label and the corrupt run results in the prediction of a token that isn't the correct target label. In noising- based approach [33 20), the patched run takes the input of the clean run, i.e., the clean prompt, its input and "patches" in the associated cached activation from the corrupted run for an LM component to determine how important that component is for the successful performance of the sub-task. Following Nikankin et al. (28], we define the effect score of an LM component as in Equation 4 where Patean and Ppotched represent the probability distribution of the next-token predictions for the clean and patched runs, respectively, and r and r' represent the target token of the clean prompt and the counterfactual prompt, respectively. This metric assigns high effect score components whose "patched" runs result in a large decrease of the correct token label's probability and/or a large increase of a counterfactual closing parentheses token's 's probability.
E(r,r') = Ppatchea(T') Petean (r')
Pelean (r) - Ppatchea(T)) Ppatchea (r)
+
Petean(r"
Counterfactual Prompts in Corrupt Runs The corruption strategy used for the corrupt runs was resampling ablation [28 [). This corruption strategy requires the construction of counterfactual prompts to be utilized in corrupt runs. We constructed these counterfactual prompts for each sub-task by taking its associated clean prompts and increasing its number of open parentheses by one. This was done by replacing single open-parenthesit token contained in the clean prompt with token containing two open parentheses for each possible open-parenthesis token position. For example, for the clean prompt under the two-paren sub-task, "#print the string 1601nprint (str (160" - "))", we create its corresponding counterfactual prompt as: "#print the string 160mpprint (str ((160" ")) For the four-paren sub-task, this token replacement strategy resulted in counterfactual prompts that were unable to be completed within a single-token prediction. In this case, we consider the subsequent token as r'. For example, for CodeLlama-7b., the counterfactual prompt for the four-paren sub-task can be "#print the string 61 615tmprint ( (str (str (str (615" - "))"
Dataset Construction We utilized the same training set as RASTEER for each sub-task when finding circuits (also described in Section Specifically, we collected positive prompts for the sub-task (e.g., prompts demanding ")" as the true token in the one-paren sub-task) as the clean prompts, and additionally filtered out prompts where the model cannot successfully predict the true token. This additional constraint of a clean prompt resulting imposed to reduce the amount of unintended noise contained in the LM components' effect scores; the same strategy was also adopted by Nikankin et al. (28). To ensure equal training dataset sizes between RASTEER and the circuit discovery baseline, additional clean prompts that fulfilled this constraint were sampled using the respective sub-task's prompt template, on the condition that the three-digit integer contained in these prompts did not overlap with the three-digit integers contained in the sub-task's ' test-set prompts. This additional sampling of clean prompts was exclusive to the sub-task's 's training datasets and was not performed on the sub-task's 's test datasets to maintain fair comparison between the methods.
correct target token prediction was
Subsequently, the respective clean prompts were used to generate counterfactual prompts, where we replaced a single open-paree penentarntseess token in the clean prompt with a two-parenthesis token, varying the position of this replacement and optionally replacing the integer number to increase the number of candidate counterfactual prompts. These candidate prompts were then filtered on the constraint that the logit of the clean prompt's target token (e.g., "))" in two-paren) is less than the logit of the corrupt token (e.g., ")))" for corrupt prompts two-paren) under the counterfactual prompt. The resulting counterfactual prompts for each sub-task were used to form the final dataset for activation
25

---

patching. This filtering step was relaxed for model/sub-taiee combinations, which resulted in an empty filtered counterfactual-promee set, allow for a baseline comparison for all model/sub-task combinations where the model could successfully perform the sub-task.
Finally, in order to understand the quality of each discovered circuit, applied the same dataset construction strategy to sample clean prompts and generate counterfactual prompts for examples on the test set of each sub-task. Note that we created this modified test set only for the purpose of evaluating circuits, whereas in our main evaluation (Section 5.2), all approaches were evaluated on exactly the same test sets.
Results of Circuit Discovery We define the faithfulness of a circuit as how much of the model's performance on each sub-task. more specifically, the logit value of the clean prompt's 's correct target token, can be accounted for by the circuit 44 28). To evaluate the faithfulness of prospective circuit, a circuit run, a model run, and a corrupted run are required for every clean and counterfactual prompt pair in the test dataset. For the circuit and model runs, the clean prompt is used as the input, while for the corrupted run, the counterfactual prompt is used as the input. In a circuit run, the output activations of all non-circuit LM components are "patched" with their associated activations from a cached corrupted run. No interventions are required for the model and corrupted runs. Following Nikankin et al. 28 we measure faithfulness using the metric in Equation 5 where rirrcui ( (correct), . modee (correct) (correct),
and N gormpp (correct) represent the logit value of the correct target token normalized by the maximal logit. This metric has an upper bound of 1.0 for all model/sub- task combinations and lower bound of -1.0 when the counterfactual prompts for model/sub- combination meet the mootenmen filtering criteria. We consider circuit faithful if it achieves an average faithfulness score of 0.9 across the test dataset.
N N.ciiru (correct) - N Lecrupp (correct)
F(circuit) =
(5)
N Lmodal (correct) N Leorrpp (correct)
Table
6 presents the faithfulness scores of circuits when only top-K attention heads (sorted by their effect scores) are retained. We only present the result of K when the model achieves a high faithfulness score.
Table 6: Faithfulness scores of circuits found for model/sub-task combinations, where the circuit consists of all FF components and the top-K position-dependern attention heads, in terms of their effect scores. Note that for sub-tasks that model is unable achieve non-zero accuracy, activation patching cannot be applied; as a result, we skip finding circuits for them.
Model
One-Paren
Two-Paren
Three-Paren
Four-Paren
GPT-2 Small
0.96 (K=1)
0.91 (K=1)
0.90 (K=1264)
GPT-2 Medium
0.98 (K=15)
0.91 (K=11)
0.96 (K=3454)
GPT-2 Large
0.96 (K=18)
0.91 (K=4902)
0.95 (K=6748)
GPT-2 XL
0.97 (K=10)
0.90 (K=8179)
Llama2-7b
0.92 (K=9)
0.96 (K=24)
0.92 (K=50)
CameL_
0.94 (K=31)
0.95 (K=23)
0.97 (K=5)
0.91 (K=152)
Pythia-6.9b
0.95 (K=6)
0.90 (K=5)
0.93 (K=146)
0.90 (K=157)
26

---

D Additional Results of RASTEER
D.1 Results of RASTEER When Steering Both Attention Heads and FF Neurons using F1 Ranking Metric
As shown in Figure 6 steering both attention heads and FF neurons yields better performance than steering either component alone, indicating a complementary effect.
Accuracy
Accuracy
Fiol
Number FF Neurons
Number Heads and Neurons
(a) Three-Paree
(b) Four-Paren
Figure 6: Performance of the model after steering both attention heads and FF neurons using RASTEER on the three-paren sub-task (left) and the four-paren sub-task (right).
D.2
Why Does RASTEER-Attention Heads Fail on the Four-paren for GPT-2 Small?
1.0
Bree
Surgarin
For
F1-Score
F1-score
10
25
50
16
20 20
Attention Head Sorted by F1-Score
Attention Head Sorted by F1-Score
(a) GPT-2 Small
(b) GPT-2 Medium
Figure 7: Fl-score distributions of the top-60 RASTEER attention heads sorted by sub-task F1-Score for GPT-2 Small and GPT-2 Medium.
We attempt answer this question by first examining whether GPT-2 Small lacks attention heads to steer that are accurate for the four-pare sub-task compared to models where RASTEER was successful. In Table 4 we find that while some of the models, i.e., GPT-2 Large, GPT-2 XL. CodeLlama-70 -bb and 1lama2-7b, have a comparatively larger number of attention heads that are accurate on the four-paren sub-task to GPT-2 Small's one four-paren accurate attention head, some models, i.e., GPT-2 Medium and Pythia-6.9b, have a similar or equal number of four-paren accurate attention heads. Thus, appears that GPT-2 Small's low number of four-paren accurate attention heads is not the sole reason for RASTEER'S 's failure.
Following, we examine if GPT-2 Small has a dramatic weakness in terms noisy promotion, such that the ground-truth token is under-promen promoted or other candidate tokens are over- werr-roooted for the four-paren sub-tasks' prompts, compared to models with a similar number of four-paren accurate attention heads. We chose to directly compare the noisy promotion of GPT-2 Small with GPT-2 Medium, instead of Pythia-6.9b, due to these models having the same number of four four-paren accurate attention heads and similar performance on the four sub-task before applying RASTEER. We examine the sub-task level F1 -cores of the top-60 RASTEER attention heads of each model for the three-paren and four-pan paren sub-tasks. In Figure 7 we observe similar F1-score distributions for each model/sub-task combination with closely matching counts of attention heads with low/high F1-scores. Indicating that there is similar number of attention heads across GPT-2 Small and GPT-2 Medium
27

---

that will correctly promote the ground-truth token/incorrectly promote other candidate tokens for the four-paren sub-tasks' prompts. Thus, it appears that GPT-2 Small's lack of substantial performance increase after applying RASTEER is not due to a weakness in its noisy promotion, but rather an unexplored factor.
D.3 Results of RASTEER using Recall and Precision for Component Ranking
As shown in Figure X while both RASTEER with precision- or recall-based ranking metries showed improved performance, the recall-based metric shows slightly better performance on the four-pares sub-task for GPT-2 Large.
Accuracy
Accuracy
Number of Attention Heads
Number Attention Heads
(a) RASTEER on Three-Paren
(b) RASTEER on Four-Paren
Accuracy
Accuracy
Number of Attention Heads
Number Attention Heads
(c) RASTEER on Three-Paren
(d) RASTEER on Four-Paren
Figure 8: Comparison of RASTEER performance using recall-based (top) and precision-base -based
(bottom) ranking metrics when steering attention heads on the three-paren and four-paren sub-tasks.
D.4 Why Does the Circuit Baseline Underperform RASTEER?
As shown in Figure 3. the circuit baseline performs comparably to RASTEER for larger LMs (over 1B parameters), but fails to show consistent correlation between steering important attention heads and performance improvement in smaller LMs. To better understand this performance gap, we examine the differences in the attention heads selected by cach method-boll in settings where the circuit baseline is effective and where underperforms. We first investigate whether the performance gains in the cireuit baseline come from steering the same set of attention heads RASTEER for the larger LMs. Specifically, we compare the top heads identified by the circuit baseline with the top-k heads identified by RASTEER, with ls selected to be the minimal number of heads that raise model's accuracy to ~ 100% when steered (referred to as "minimal effective heads" onwards). As shown in Figure 9 the degree of overlap varies between 20- 100%, which shows that the two approaches can identify different sets of important heads. To further isolate the contribution of shared heads on performance improvement, we perform a minimal steering experiment, where for CodeLlama, Pythia, and Llama2, we steer the shared minimal effective heads between the two approaches. For example, for CodeLlama, both approaches need only top-5 heads for stecring the model achieve 100% accuracy, and we steer the 3 heads overlapped between them (i.e., 0.6 percentage when = 5 in Figure 99. We skip GPT2-XL because of the high overlap. We observe that steering only this minimal shared effective heads led to a similar 100% accuracy for Llama2 and CodeLlama but no improvement for Pythia. This indicates that the overlapping heads are generally a smaller subset of necessary heads for steering, but not always. This is especially evident in Pythia-6.9b, where steering two disjoint sets of four heads (excluding the shared one) from each method independently achieves ~ 100% accuracy, highlighting the redundancy of useful mechanisms within the model.
28

---

3.8
Overlap
gpt2 xi tap5
5ythie-699--ap5
Liema 7b top10
30
Top Circuit Attention Heads
Figure 9: Percentage overlap of the top-k RASTEER attention heads, where k=5 for GPT-2 XL, CodeLlama-7b5 7b, and Pythia-6.9b and k=10 for GPT-2 Medium and Llama2-7b, with the top-60 attention heads from the respective circuit baseline, for models where the steering of small number of RASTEER attention heads resulted in most of the performance improvement on the four- cour-paren sub-task. For example, the 0.2 overlap percentage for top- 10 eircuit attention heads indicates shared head with RASTEER'S 's top-5 heads.
Next, we investigate why the circuit baseline is not effective for smaller LMs. Since the circuit analysis consists of heads that will not be discovered by RASTEER and responsible for intermediate computations such as inter-token information transfer and feature extraction, we posit that these additional heads may introduce instability in steering. To the effect of steering these additional components, we steer the top-10 10 RASTEER heads in GPT-2 Medium, which has performance of 100%, along with 5 eircuit-identified heads from non-final token positions (which cannot directly affect the output). This led to decline in accuracy from 100% to 4.00%, and adding 5 more (10 total) such heads collapses performance near 0.00%. This indicates that all task-relevant components are good for steering. Specifically, components that are functionally important for the task but do not directly affect the final logit may introduce instability when promoted.
29

---

E Experiments compute resources
We conduct all experiments on an NVIDIA A100 GPU with 40GB GPU memory and up to 50GB of CPU memory.
F Licenses for existing assets
We use the following open-weight models for our experiments.
F.1
Models
GPT-2 Models [3] (Modified MIT License at utttp:e//gtthbbb.cmmmoppaaaa/// gpt-2/
blob/nasterr/LIcENSE
Llama2-7b 40
Special Llama-2 License at (ttps://ww.looma.coo//eeeesss
Llama-3 8b
17
Special Llama-3 License at https://laanaa.mtttccco//aaaaaa
license/
7odel.lama-7b
36
Special Llama-2 License http:://githubb.co//eeeaaaaaa
Pythia-6.9b and Pythia-12b
6
Apache
License 2.0 at https://github.ccom
Qwen-2.5-3h (38 Apache License at httph:tt//// http::///ggggggg......m http:/acccccooeeeneeeeeeeee
5o 5
blob/main/LICEKSS
30


Reviews:
[{"review_id": "9AkQAlEC8M", "reviewer": "Reviewer_6Wg5", "summary": "The stated goal of this submission is to analyze the internal mechanisms used by LMs to generate balanced parentheses. Specifically, the submission makes the assumption that all opened parentheses should be closed by generating a single token. In practice, this means that the LM is prompted with code having 1 to 4 open parentheses and now should generate exactly one of the following tokens: \")\", \"))\", \")))\", or \"))))\".\n\nThe analysis of internal mechanisms consists in ranking what the authors term \"components\", i.e., individual attention heads and individual neurons in feed-forward layers. The authors rank components according to wether they promote or suppress the correct output token. Concretely, components are scored via the LogitLens, i.e., each component's output is projected onto the vocabulary using the unembedding matrix, resulting in a logit score which is taken to quantify the component's contribution to the model generating the output token or not. By comparing if a component's logits of the correct token are larger than the logits all wrong token choices, the authors also compute the task accuracy of that component.\n\nAnalyzing these scores, the authors find that some components promote the correct output tokens while others promote wrong ones. The former are termed \"sound mechanisms\" and the latter \"faulty mechanisms\".\nThresholding on the scores, the authors identify task-relevant subgraphs of the entire computation graph, i.e., sets consisting of attention heads and FF neurons. The authors show that scaling the activations of all task-relevant subgraphs by a factor >1 improves task parentheses balancing without adverse effects on general code generation capabilities.\n\nIn addition to parentheses balancing, the authors also apply this method to a basic arithmetic task in the range of 0-999 and show that activation scaling of the identified components improves performance.", "questions": "Q1. Regarding weakness W1 above, my concern is that since, according to my current understanding, the proposed method increases the output probabilities of all valid tokens and decreases the output probabilities of all non-valid tokens. So a simple baseline would be to restrict the original models to generating only the four closing parentheses tokens in the parentheses balancing task and to generating only integer tokens in the arithmetic task. How would the proposed method compare to this baseline? (or if this\n\nQ2. Regarding W2, One counter-argument could be that one should post-process LM output with a linter or that all code in HumanEval is somehow processed into a standard format (e.g., removing line breaks where possible, thereby increasing the number of adjacent parentheses). I'm happy to remove this weakness if this done.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nS1. The paper successfully localizes a small set of components that are causally relevant for the task of interest on two tasks across several models, suggesting that the proposed method is generally useful\n\nS2. The paper is written well and is easy to follow\n\nWeaknesses:\n\nW1. The analysis and identification of components is limited to promotion and suppression of output tokens. It is unclear if the identified components are specific to the task of interest, or if they are simply general biases that always increase or decrease the output token probabilities regardless of context. For example, if one were to apply the logitlens to a randomly-initialized model and rank attention heads and FF neurons according to their logit contributions to specific tokens, one would also find \"sound mechanisms\" that promote these tokens and \"faulty mechanisms\" that suppress target tokens and/or promote non-target tokens.\n\nW2. The proposed method is only applicable to tasks in which the desired output is a single token from a small set of tokens that are known in advance\n\nW3. Related to W2, the balanced parentheses task is analyzed only in a very specific setting with 1 to 4 parentheses to be closed in a single step on a single line. My intuition is that a lot of naturally occurring code would split closing parentheses across multiple lines with various levels of indentation, making the considered setting somewhat artificial. This might be relevant for the author's argument that the interventions on the identified components do not affect general coding capabilities, since the prevalence of single-token closed parentheses might be relatively small in the evaluation data, and hence even randomly replacing \")))\" or \"))))\" with another set of closing parenthesis might not lead to a large performance drop, especially since the benchmark (HumanEval) is evaluating Python code, which generally does not have too many parentheses.", "was_revised": true, "final_justification": "While the author response addressed two of the three weaknesses I identified (W1, W3), one weakness (W2, limited applicability) remains and prevents me from giving a higher score.", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and constructive feedback. We appreciate the reviewer’s recognition that the study was able to localize causally relevant components across multiple tasks and models, as well as their feedback on the clarity and readability of our writing. \n\n**Response to W1 and Q1:**  \n\n**1. Our method selects task-specific and context-sensitive components, not components with general biases to a set of valid tokens.** \n\n- **Clarification:** First, we would like to clarify a potential misunderstanding. Our approach does not work by “increasing the output probabilities of all valid tokens (e.g., parentheses tokens) and decreasing the output probabilities of all non-valid tokens”. Instead, our approach works by promoting the output of LM components that can increase the output probabilities of the correct valid token out of all valid ones. This is achieved by analyzing the task-specific performance of each LM component at the prediction position (Section 3).  \n\n- **Metric Design Reinforces Precision:** Our component ranking uses F1-score, which penalizes components that promote correct tokens in the wrong contexts. Components with general bias would have poor precision and hence receive low scores. Consequently, our approach is unlikely to promote components that are simply general biases to a set of tokens.  \n\n- **Evidence from Subtask Accuracy:** In Section 3.2, we discover several components with high accuracy across multiple subtasks. For instance, attention head L30H0 consistently achieved nearly 100% accuracy across all subtasks with different inputs. This indicates a clear context-dependent behavior of discovered components. \n\n- **Shared Component Set Across Subtasks in Steering:** We do not promote different sets of components for different subtasks (see Line 234); instead, we steer the same set of components across all subtasks. This further reduces the risk of selecting components that only help with a specific subtask and instead ensures broader, context-sensitive behavior. \n\n- **Hypothetical randomly-initialized model:** We agree that a randomly initialized model may consist of components that exhibit general biases. However, it is unlikely to contain components with the high, context-sensitive accuracy we observe (e.g., L30H0). We will clarify this distinction in the final version. \n\n \n\n**2. Would a restricted-output baseline (valid tokens only) match RaSTEER?** \n\nWe believe the suggested baseline is equivalent to comparing with the full model because models rarely produce invalid tokens for the task input. In our study, we found **all models already generate valid tokens even when they make mistakes**, with only 1 exception in GPT-2 Small and 3 in CodeLlama-7b. This suggests the model understands the task (producing balanced parentheses) and is **already biased towards producing valid tokens**, but struggles with selecting the correct one among the valid tokens. Under this situation, the suggested baseline would not lead to different results. \n\n**Response to W2:** \n\nWhile we primarily focused on the balanced parentheses task that consisted of only 4 valid output tokens, we also apply our approach to a two-operand arithmetic task (Section 5.4) where the set of valid target tokens includes all the integers from [0, 1000] that tokenizes to single tokens. Despite this significantly larger set of valid tokens, our method still achieved meaningful performance gains—up to 20.25% improvement in Pythia-6.9b—suggesting that RaSTEER is effective even when the number of valid output tokens is large. In addition, as the reviewer pointed out, our experiment on HumanEval further shows that our approach works well when there is a large set of valid tokens. \n\n**Response to Q2:** We may have misunderstood the reviewer’s suggestion because of an incomplete sentence. We would like to clarify that when we applied our approach to the HumanEval benchmark, we did not use any syntax checker; we steered the model over all decoding steps because when a model requires to generate closing parentheses cannot be trivially \"known in advance\". However, if the suggestion was to use Linter to detect syntactic errors and apply our method only when needed, we agree that this is a reasonable and practical integration. Currently, our approach is designed to proactively steer model behavior during the entire generation, rather than relying on post-hoc correction. However, we acknowledge that when the downstream effects of steering on overall code generation task are uncertain or untested, it may be preferable to apply conditionally—e.g., only when a linter detects failure.  \n\n**Response to W3:** We acknowledge that our synthetic dataset focuses on a simplified setting with 1–4 closing parentheses on a single line, which may not capture the full variety of naturally occurring code. To assess how artificial this setup is, we analyzed the generation of Llama2-7b, CodeLlama-7b, and GPT-2-XL in HumanEval test set. We did not find any instances where closing parentheses were split across multiple lines. While this does not negate the reviewer’s broader point—especially for other programming languages or codebases with different formatting conventions—it suggests that **our setting does occur naturally** and is especially common in Python, where multi-line closing parentheses are less frequent.  \n\nIn regard to the concern that RaSTEER may have little impact on general coding performance due to the infrequency of closing parentheses, we analyzed HumanEval generation from Llama2-7b and found that the model is tasked with producing one of the closing parentheses tokens in at least **69.63%** of the generation steps needed to produce a valid line of code. This indicates that parentheses tokens are indeed frequent for HumanEval benchmark. In addition, we would like to clarify that RaSTEER is applied at every generation step, not just when predicting closing parentheses (because our goal is to steer the general model behavior in deciding balanced parentheses, and we cannot foresee when a model will predict parentheses). Therefore, the fact that RaSTEER does not hurt a model's generic coding performance even when they are activated for every generation step indicates that it selectively encourages better syntactic completion without degrading semantic fluency."}, {"review_id": "NR6fTBGhU8", "reviewer": "Reviewer_kKr8", "summary": "The paper investigates the difficulty in generating balanced parantheses by different LLMs. The authors identify the LLM components those are responsible for introducing inaccuracies in balanced parantheses generation. In order to mitigate this, the paper proposes a steering method called RASTEER to rank the components based on their reliability and contribution in final logits computation. The authors report substantial improvements in the LLMs for the task and also demonstrate the overall efficacy of their approach in mathematical reasoning.", "questions": "1. The authors should mention the selection criteria of the LLMs.\n2. It is not clear why larger LLMs (> 7B parameters) are not explored.\n3. The authors can add a detailed section on different prompts describing each undertaken strategy in the appendix.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors analyze components such as attention heads, feed forward neural networks that contribute majorly in the final logit computations. They also propose algorithms to identify the correct tokens over distractors and promote correct tokens with predefined thresholds. When the NLP community is trying out various complex tasks on LLMs, the authors attempt to unearth the reason behind errors encountered for a simple task of balancing parantheses.\n\nThe authors should mention the selection criteria of the LLMs. It is not clear why larger LLMs (> 7B parameters) are not explored. Different prompts detailing the strategies would have helped to understand the paper better.", "was_revised": true, "final_justification": "The authors have given satisfactory responses to the queries regarding the prompting strategies and the experiments with models > 7B parameters.", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and positive assessment of our work. We appreciate the reviewer’s recognition of our proposed algorithms to identify components that promote the correct tokens over distractors. \n\n**Response to W1, Q1, and Q2:** To demonstrate that insights obtained from our study and proposed approach generalize across various model families and sizes, we considered the models that varied in the following aspects. \n\n- Model Size (117M to 7b models): We only considered model sizes below 7b because of computational budget constraints. We would also like to mention that many prior interpretability studies [1, 2, 3, 4] also consider models of similar sizes for similar studies.  \n\n- Model families: \n\n   - Base models (GPT-2, Llama2-7b, Pythia) \n\n    - Code-specific LM (CodeLlama-7b) \n\n- Performance diversity: We selected models with different accuracy on our synthetic dataset (0% to 100%) and on HumanEval (0% to 30.48%).\n\nAlthough we face computational constraints, we agree that extending the analysis to larger models is an important direction for future work and will clarify this limitation in the revised version. To this end, we conducted **additional experiments on Pythia-12b for the arithmetic task**. RaSTEER improved accuracy across operations—addition, subtraction, multiplication, and division—from 17.33%, 6.00%, 8.00%, 27.00% to 24.66%, 7.33%, 21.00%, 36.33%, respectively. These results are consistent with improvements seen in smaller models and demonstrate RaSTEER’s potential applicability to larger-scale LMs. We will include the selection criteria of the LLMs and new results from Pythia-12b in the final version of the paper. \n\n**Response to w2 and Q3:** Thank you for the suggestion. We describe our prompt design strategy for the balanced parentheses task in Section 2.2, and provide a brief discussion of the arithmetic reasoning prompt in Section 5.4. For HumanEval, we use the original dataset-provided questions without additional prompting. We will include a more detailed description of all prompt strategies in the Appendix of the final version to improve clarity and reproducibility.  \n\n**References:**\n\n[1] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. In The Thirteenth International Conference on Learning Representations, 2025.  \n\n[2] Stolfo, Alessandro, Yonatan Belinkov, and Mrinmaya Sachan. \"A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. \n\n[3] Nanda, Neel, et al. \"Progress measures for grokking via mechanistic interpretability.\" The Eleventh International Conference on Learning Representations. 2023.\n\n[4] Wang, Kevin Ro, et al. \"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.\" The Eleventh International Conference on Learning Representations. 2023."}, {"review_id": "qmBNG2bsnG", "reviewer": "Reviewer_K9m9", "summary": "This study explores why language models (LMs) continue to struggle with fundamental syntactic tasks, such as generating balanced parentheses, despite significant progress in coding capabilities. The research uncovers that LMs employ a diverse array of components—including attention heads and feedforward (FF) neurons—to address sub-tasks, with most components specialized for specific functions and only a few demonstrating broader generalizability. However, these components often exhibit \"noisy promotion,\" simultaneously supporting both correct and incorrect tokens, which results in low precision and recall. To address this, the authors introduce RASTEER, a method that ranks components based on their reliability and amplifies the activations of the most dependable ones. Experiments show that selectively promoting these top-ranked components significantly enhances performance on balanced parentheses tasks, as well as on synthetic datasets, the HumanEval benchmark, and arithmetic reasoning tasks, all while maintaining general coding proficiency.", "questions": "[1] Does RASTEER improve model performance on more general benchmarks, as well as on mathematical reasoning benchmarks such as GPQA, MMLU, and GSM8K? The Arithmetic Reasoning tasks presented in this paper are constructed by the authors and involve only two operands, making them relatively simple and limited in scope.\n[2] RASTEER selects the reliable components of language models through grouping and sorting, which involves two hyperparameters: a threshold set to 0.7 and a promotion threshold set to 0.5. How did the authors determine these values?\n[3] The experiments in this paper are mainly conducted using models such as GPT-2, CodeLlama, and Llama2, which are already outdated and exhibit relatively weak performance. This likely explains the frequent errors observed in tasks like balanced parentheses. Therefore, it remains unclear whether RASTEER would still be effective when applied to more advanced models, such as the Qwen2.5 series.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n[1] The paper demonstrates a deep understanding of language model (LM) architectures and mechanisms. It provides a detailed analysis of how attention heads and feed-forward (FF) neurons contribute to syntactic tasks like balanced parentheses generation.\n[2] The paper is well-structured, progressing logically from problem formulation to methodology, experiments, and conclusions. Key insights (e.g., noisy promotion, mechanism overshadowing) are clearly explained.\nWeakness：\n[1] The study mainly focuses on synthetic datasets for balanced parentheses, which is very simple and naive. Generalizability to real-world coding scenarios or more complex syntactic structures is not fully explored.\n[2] This paper primarily conducts experiments on models such as GPT-2, CodeLlama, and Llama2, which are relatively outdated and have weaker performance. Therefore, it is difficult to determine whether RASTEER would remain effective on more advanced models, such as the Qwen2.5 series.", "was_revised": true, "final_justification": "This paper solves reasoning tasks from a fundamental perspective, which I believe would be contributory in this field.", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and constructive feedback. We appreciate the reviewer’s recognition of the detailed analysis of how attention heads and feed-forward neurons contribute to syntactic tasks, and the clear, logical structure of the paper. We are also glad that key insights—such as noisy promotion and mechanism overshadowing—were found to be clearly explained.\n\n**Response to W1:** We would like to clarify that the simplified setting was an intentional design choice to enable precise analysis of failure modes in a controlled setting. In more realistic scenarios, errors are often challenging to disentangle, potentially stemming from reasoning failures, knowledge gaps, formatting issues, or task ambiguity. Our controlled setup, by contrast, allowed us to confidently attribute failures to the specific syntactic error under investigation. In addition, synthetic setting also offers a low-cost and tractable way to synthesize dataset, which would be significantly more challenging to annotate in realistic codebases. For these reasons, this approach, where simplified settings distilled from realistic applications is **widely adopted in interpretability research [1, 2, 3, 4]**, and we follow this vein when designing our research. Finally, we also applied RaSTEER to the HumanEval benchmark to check unintended effects, and surprisingly, we find our approach could lead to a small improvement in general coding tasks as well. We believe this provides positive preliminary results for the broader applicability of our approach beyond this initial setting.  \n\n**Response (W2 and Q3):** We appreciate the reviewer’s concern and agree that evaluating on more recent models is important. To address this, we conducted additional experiments on **Qwen2.5-3B**, as suggested by the reviewer. We found that Qwen2.5-3B has 100% accuracy in the one-paren and two-paren subtasks. While for the more challenging three-paren and four-paren subtasks, the model achieved 92.66% and 46.66% accuracy, respectively. Next, we applied RaSTEER by promoting the top 20 attention heads, which improved the accuracy on the three-paren subtask from 92.66% to 98.00%, and on the four-paren subtask from 46.66% to 77.33%. We observed no further gains when promoting additional heads. We also verified that the RaSTEER did not harm general coding performance on HumanEval; instead accuracy improved the accuracy slightly from 33.53% to 35.36%. Additionally, we have also applied RaSTEER to **Llama3-8B** (misnamed as GPT-3 8B in the current version; we will correct this in the final version), a relatively new model, for the arithmetic task in Section 5.4 and observed improvement (up to 6.00%) in accuracy. These results suggest that RaSTEER is effective when applied to more recent and advanced models such as Qwen2.5 and Llama3 series. \n\n**Response to Q1:** In this study, we focused on implicit reasoning tasks, where the model is required to produce the final answer in a single decoding step without generating intermediate reasoning. In contrast, benchmarks such as MMLU and GSM8K involve explicit chain-of-thought (CoT) reasoning, which requires multi-step generation and was considered outside the scope of this work. We believe that understanding the implicit reasoning of the model is a critical step toward interpreting and improving more complex explicit reasoning behaviors. Following common practice in prior work [1, 2, 3], we used a synthetic dataset to study the mechanisms involved in implicit reasoning. That said, we agree with the reviewer and acknowledge that evaluating RaSTEER on more challenging benchmarks like MMLU, GSM8K, and GPQA is an important direction for future work.\n\n**Response to Q2:** The **accuracy threshold** of 0.7 for grouping components was selected based on empirical observations of accuracy distributions across sub-tasks (Figures 1 and 5). We found that this value effectively separates components with high accuracy in at least one sub-task from others.  For the **promotion threshold**, we initially used 0.5 as a reasonable threshold for the analysis in Section 3, then validated it later through tuning on the development set for RaSTEER. Specifically, we evaluated thresholds in the range [0.4,0.5,0.6,0.7,0.8,0.9] and found that 0.5 consistently achieved strong performance. For example, on the four-paren sub-task with GPT-2 Small (top-60 heads), 0.5 yielded 16.8% accuracy—outperforming other thresholds, which ranged between 5.3% and 14.0%. Notably, we found that the promotion threshold had less impact on larger models (≥1B parameters), where promoting just a small number of reliable components is often sufficient for performance gains. In summary, we posit that the promotion threshold balances between false positives and false negatives: thresholds below 0.5 risk including too many components that do not meaningfully promote the target token, while thresholds above 0.5 risk excluding components that are genuinely helpful. We will include these empirical details in the updated version of the paper. \n\n**References:** \n\n[1] Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. In The Thirteenth International Conference on Learning Representations, 2025.  \n\n[2] Stolfo, Alessandro, Yonatan Belinkov, and Mrinmaya Sachan. \"A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. \n\n[3] Nanda, Neel, et al. \"Progress measures for grokking via mechanistic interpretability.\" The Eleventh International Conference on Learning Representations. 2023. \n\n[4] Wang, Kevin Ro, et al. \"Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.\" The Eleventh International Conference on Learning Representations. 2023."}, {"review_id": "c2rPWtMctT", "reviewer": "Reviewer_Wfy8", "summary": "This paper researches the failures of simple synthetic tasks of LLMs. Authors investigate a parenthesese matching problem, which is simple for humans but hard for LLMs. Authors delve into the mechanism of the phenomenon by examining the computation flow of Transformers. By utilizing an unembedding matrix, the authors can identify neurons that promote correct answers to a certain configurable degree. Authors then discover that LMs develop different mechanisms to solve tasks with different generalization level. For example, an FF neuron at most generalizes to two sub-tasks; the failures are caused by noisy signals. To this end, authors propose sorting components by the degree to which they promote correct responses and scaling the signals of these components accordingly. Results show the proposed method perform better than the circuit baseline.", "questions": "* How do authors obtain the unembedding matrix?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* Interesting problem which is valuable to investigate.\n* The mechanism research method is sound and clear\n* The proposed solution is reasonable\n\n### Weakness\n\n* It is unclear how well the algorithms 1 and 2 make sense because they ignore the long-horizon effect of the neuron. For example, a single neuron may not promote the correct response, but it may be useful in a circuit. Nevertheless, I understand this is hard to quantify.\n\n* The proposed method requires a training dataset, which limits its general application.\n\nRegardless, I believe this is a good analysis paper.", "was_revised": true, "final_justification": "The authors address my concerns on the circuit method, the training set (only 350 examples), and clarification questions. Therefore, I maintain my score.", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and positive assessment of our work. We appreciate the reviewer’s recognition of the importance of the problem, the soundness and clarity of our analysis, and the proposed solution. \n\n**Response to W1:** We have a circuit baseline that should capture components that have these long-horizon indirect effects on the upstream components. However, as seen in Figure 4, when we steer with components found through the circuit analysis, it doesn’t have a strong correlation with the accuracy. This indicates that not all task-relevant components are good for steering. Specifically, components that are functionally important for the task but do not directly affect the final logit may introduce instability when promoted.  \n\n**Response to W2:** While our method does require a training set, it is lightweight compared to fine-tuning-based approaches. We use only 350 training examples for the balanced parentheses task, which can be easily synthesized. Additionally, we would also like to note that most steering methods, such as CAA [1], also require training data. That said, we agree with the reviewer that reducing data requirements further is an important direction for future work. \n\n**Response to Q1:** We obtain the unembedding matrix directly from the final layer of the model, which maps the hidden representations of the final layer to vocabulary logits. We will clarify this in the final version of the paper. \n\n**References:**\n\n[1] Rimsky, Nina, et al. \"Steering Llama 2 via Contrastive Activation Addition.\" 2024."}]

Author rebuttal:

Meta-review: The paper does a mechanistic analysis of the failure of LMs of varying sizes (124M–7B) on the simple task of generating balanced parentheses. The authors identify several components of the trained LMs that make their own predictions for the completion task, where some are sound and some are faulty. They argue that errors occur when there are more faulty vs sound mechanisms and to address this propose a steering mechanism that improves performance drastically on this task. They show some gains from this idea on a more general arithmetic reasoning task (still idealized).

The paper steers away from doing a full mechanistic analysis but rather identifying a set of components that affect performance, and boost the correct ones to improve performance. This to my knowledge is a novel steering approach beyond directional steering. While this is computationally very expensive, requires a training set, and has been evaluated only for simple tasks with a single output token answer, I (and the reviewers) believe that it is a useful contribution to the literature. Additionally, during the rebuttal phase, the authors have provided additional experiments to alleviate most of the concerns of the reviewers.

I encourage the authors to add the additional proposed experiments to the camera-ready and a discussion about how to extend beyond single token answers, and the impact of their steering approach on the general performance of the models.
Decision: Accept (poster)