Improving Reasoning Performance in Large Language Models via Representation Engineering
Authors: Bertram Højer
Oliver Simon Jarvis
Stefan Heinrich
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Reasoning
Intelligence
Representation Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We derive vectors from the internal state of large language models that can be used to control model behavior and improve performance on simple reasoning tasks with no additional training.

Abstract: Recent advancements in large language models (LLMs) have resulted in increasingly anthropomorphic language concerning the ability of LLMs to reason. Whether \textit{reasoning} in LLMs should be understood to be inherently different is, however, widely debated. We propose utilizing a representation engineering approach wherein model activations are read from the residual stream of an LLM when processing a reasoning task. The activations are used to derive a control vector that is applied to the model as an inference-time intervention, modulating the representational space of the model, to improve performance on the specified task. We publish the code for deriving control vectors and analyzing model representations. The method allows us to improve performance on reasoning benchmarks and assess how control vectors influence the final logit distribution of a model via metrics such as KL divergence and entropy. We apply control vectors to Mistral-7B-Instruct and a range of Pythia models on an inductive, a deductive and mathematical reasoning task. We show that an LLM can, to a certain degree, be controlled to improve its perceived reasoning ability by modulating activations. The intervention is dependent upon the ability to reliably extract the model's typical state when correctly solving a task. Our results suggest that reasoning performance can be modulated in the same manner as other information-processing tasks performed by LLMs and demonstrate that we are capable of improving performance on specific tasks via a simple intervention on the residual stream with no additional training.

Published as a conference paper at ICLR 2025
IMPROVING REASONING PERFORMANCE IN LARGE LANGUAGE MODELS VIA REPRESENTATION ENGI- NEERING
Bertram Højer;" Oliver Jarvis; Stefan Heinrich Department of Computer Science, IT University of Copenhagen, Denmark [berh, ojar, stehe]@itu.de
ABSTRACT
Recent advancements in large language models (LLMs) have resulted in in- creasingly anthropomorphic language concerning the ability of LLMs to reason. Whether reasoning in LLMs should be understood to be inherently different is, however, widely debated. We propose utilizing representation engineering ap- proach wherein model activations are read from the residual stream an LLM used derive control
when processing reasoning task. The activations vector that is applied to the model as an inference-time intervention, modulating the representational space of the model, improve performance on the speci- fied task. We publish the code for deriving control vectors and analyzing model representations
[1 The method allows us to improve performance on reasoning benchmarks and assess how control vectors influence the final logit distribution of a model via metrics such as KL divergence and entropy. We apply control vectors to Mistral-7B-Instteee and mathematical reasoning task. We show that an L.M can, certain degree, be controlled to improve its perceived reasoning ability by modulating activations. The intervention is dependent upon the ability to reliably extract the model's typ- ical state when correctly solving task. Our results suggest that reasoning per- formance can be modulated in the same manner as other information information-processing tasks performed by LLMs and demonstrate that we capable of improving per- formance on specific tasks via simple intervention on the residual stream with additional training.
and range of Pythia models on an inductive, a deductive
INTRODUCTION
Many recent developments in the study artificial intelligence and more specifically large language models (LLMs) have focused on improving their ability to solve reasoning tasks. The notion of reasoning is, however, notoriously hard to ground and define (Chollet 2019 Pavlick 2023). While various forms of reasoning such inductive, deductive and abductive reasoning well-defined, they typically refer to high-level processes often associated with System 2 thinking as opposed to fundamental computational mechanisms (Johnson-Laird, (008). The increasing scale of LLMs and better data-curation have yielded better results on standard benchmark datasets leading to research into the reasoning strategies employed by LLMs; this research generally focuses on model outputs as opposed to internal states (e.g. Mondorf & Plank 2024). We thus cannot conclude anything with regards to the processes or representational learning dynamics related to the ability of an LLM to solve reasoning tasks. We propose modeling the typical representations of a simple "reasoning process" and using those representations to improve reasoning performance on three
tasks.
Recent work has shown that it is possible to induce specific types of behavior by manually modulating the internal state of a model. Researchers have assessed editing LLM knowledge in MLP layers of transformers (Meng et al. 2023a b) and have mapped entire computational circuits
"Equal contributions. code: http:///eee.t.t...aaaaaaaaaaaaar

---

Published as a conference paper at ICLR 2025
within models (Wang et al. 2022) More interestingly. it has been shown that do not need to investigate specific MLP or attention layers, but can instead look at the residual stream of an LLM. By modulating the residual stream in meaningful way it is possible to induce different types of "behavioral" traits such as honesty, truthfulness and emotional valence. These types of behavior be conceived directions in the representational space of an L.M (Liu al., 2023: Hendell et al. 20233 Todd et al.) 2024). Additionally, modulating the residual stream has also been used improve the ability of transformer models trained to play the board games chess and othello (Karvonen 2024; Nanda et al.) 2023). A similar approach was utilized by Anthropic for their Golden Gate Claude model in which they steered an LLM towards "acting" as the Golden Gate bridge (Templeton et al., 2024). The above research highlights the efficacy of the approach well as the broad potential applications.
As a novel contribution, we directly analyze the ability to improve "reasoning" using a rep- resentation engineering approach. We specifically assess whether the residual stream of LLMs contains valuable (and actionable) information as to a model's reasoning ability and whether it can be used to improve it, has been done for other types of model "behavior". Concretely, we extract activations from the hidden dimension of LLMs. We then create a control vector on the basis of these activations and assess whether inference-time intervention can improve reasoning ability on and across the specific set of tasks we analyze.
The main contribution of this paper thus an analysis of the effectiveness of perturbing the residual stream in order to improve model's performance on a set of reasoning-relater tasks. We particularly address whether the method allows find a direction in the representational space of LLMs related to reasoning, and how the intervention affects the representational space of a model.
2 APPROACH
We employ a representation engineering approach, deriving control vectors based on typical model representations when performing the specified reasoning task. This allows to assess whether one can induce better reasoning performance similarly to how one can adjust e.g. the emotional valence of model outputs (Zou al. 2023).
4-3211 y MLP(y))
Add Norm
Add & Norm
Feed-forward layer
Feed-forward layer
y
Resisdual Stream
1x Block
Block
Add & Norm
4.
AH((x))
Add & Norm
Multi-headed self-attention
Multi-headed self-attention
(a) Traditional representation of transformer chitecture as introduced by Vassaan el al. (2017)
Conceptual reframing of the transformer archi- tecture as discussed by Elhage et al. (202T)
Figure 1:
A conceptual reframing of transformer information processing emphasizing the impor- tance the residual stream. As opposed to focusing on the operations done by each computational block, this framing emphasizes each computational block reading from and writing residual stream that represented in the hidden dimension of a transformer model.
In most LLMs, the transformer architecture comprises an embedding layer followed by n computa- tional blocks finally followed by output layer of token logits from which new token is generated
2

---

Published as a conference paper at ICLR 2025
via a softmax function over the logits (Vaswani et al., 2017). Following arguments by Elhage et al. (2221) as well as recent related research (Templeton et al. 2024), we highlight the residual stream as key component of the transformer, and emphasize that the fundamental operation of the trans- former is representation transformation. This conceptual reframing illustrated in Figure The operations done by single layer in any given transformer be described in lequation w the term C₁ + a.
YE LayerNorm (xe + Att(xx))
X(+1 = LayerNorm (ye + MLP(yc)) + a
X; is the hidden dimension activation vector at the l'th layer, Ye the activations after the scaled dot-product mechhnnism and X(+1 the activations after the MLP transformation.
Recent work has illustrated the efficacy of analyzing and manipulating the residual stream by e.g. using sparse autoencoders extract features from residual stream activations (Huben et al. 2023). ) This work suggests that the residual stream is a better level of analysis than single neurons (or cohorts of neurons) due partly the issue of polysemanticity. Targeting the residual stream has also been proposed as a solution to cross-layer superposition: the notion that features are smeared across many hidden layers of deep neural network (Bricken et al. [023 Templeton et al. 2024). We extract the representation after each layer at the final token of task example. From these extracted activations
we derive layer-specific control vectors. Applying the control vector is the simple addition of adding the term to the standard transformer operation as illustrated in Eqqution
2.1 CONTROL VECTORS
control vector can be trained in multiple ways based on the extracted model activations. The sim- plest way is by creating a reading vector, which is merely the average over the extracted activations. Given a set of prompts P, a model with L layers, and He(P:) as the hidden state of the residual stream for the ('th layer and the i'th prompt, the reading vector is described as follows:
PI
C₁ P
H((P)
(2)
Related work indicates that for extracting a desired behavior from an L.M it best to use contrastive pairs. The use of contrastive pairs entails looking the representations for positive and negative prompts. For the case of say, emotional valence, it could entail examples of 'happy' prompts and examples of 'sad' prompts. For the case of reasoning we look at examples of successful reasoning and unsuccessful reasoning see subsection 3.1 This ensures that the control vectors are based on a difference representations (Zou et (003).. The control vector can thus be scaled to induce the desired behavior or its opposite. Given set of contrastive pairs of positive and negative prompts p= =(+, P-), we define the control vector as:
1 p
Σ
(He - He (P:T)
C₁
Another method for training control vectors involves applying Principal Component Analysis (PCA) the contrastive activations (Zou et al.] 2023). PCA finds directions in the data that account for the most variance, with the first component indicating the direction in the data explaining maxi- mum variance. The intuition is that the difference between these contrasting representations will contribute significantly the variance in the activations, and the first principal component should thus approximate the direction in the activation space that most effectively distinguishes between the desired outcome and its contrast. This direction then be used as the control vector. Similar to reading vectors using contrastive pairs the PCA control vector is defined in the following manner.
C, = PCA (HH, - H, (P..) Vie [I...., p 1)
(4)
A key consideration is how much one wishes to modulate representations (controlled by a). as too strong modulation could result in non-sensical outputs. When creating control vectors of the reading
3

---

Published as a conference paper at ICLR 2025
type, a = is equivalent to adding "full" activation vector to the signal ||ce|| ~
||Ha|]
which follows from Equation 2] However, when applying PCA we get ||c||| = 1. To account for the difference we implement up-scaling of PCA-based control vectors which is dependent on the actual norm of the extracted residual stream activations (see Subsection A.I)
The derivation of control vectors is thus a relatively simple process that does not induce a large computational overhead. We simply need to extract activations while performing inference on given task to control and improve the outputs of LLMs without any further training.
3
EVALUATION & ANALYSIS
3.1
EXPERIMENT
For IOI we create 2, 000 examples for each condition of the following form:
Mary |A| and John[a] went to the store. John gave the groceries to Mary[a]-
This task is inductive as multiple answers could be correct, we are however interested in controlling the model to generate desired name and pose this inductive inference problem. We study four different conditions: ABBA (A), BABA (B), ABBA-Long (AL) and BABA-Long (BL).
bAbl comprises various reasoning tasks, one related to deductive reasoning of which there are 2, 000 examples. The questions have the following format
Passage: Mice are afraid of wolves. Gertrude is a mouse. Cats are afraid of sheep. Question: What is Gertrude afraid of? Answer: Wolf
GSM8K consists of high quality grade school math problems on which relatively capable LLMs still struggle. The questions are of the following format:
Question: "Natalia sold clips to 48 of her friends in April, and then she sold half many clips in May. How many clips did Natalia sell altogether in April and May?" Answer: "Natalia sold (48/2 48/2 = 24) 24 clips in May. Natalia sold (48 24
48 + 24 72) 72 clips altogether in April and May. #### 72"
We provide fully detailed example of a question from the GSMSK dataset in subsection A.3 For GSM8K we use total sample 400 prompts to derive the control vectors. For each dataset create train and test splits with stratified labels. We then derive the control vector based on model representations when it generates outputs on examples from the train split and test model performance with control vector applied on the test set.
In order to produce the contrastive pairs we need positive and negative prompt examples. For the positive examples, obvious method might be to select all cases where the model successfully solves a task, making sure that the class labels are balanced so as to ensure that the model is not biased towards specific answer. A prompt for unsuccessful reasoning is less clear however. We propose multiple schemes to elicit representations typical of poor reasoning. 1) A naive approach entails asking a model to produce an incorrect answer. This was found during testing to be a poor scheme since producing the wrong answer when prompted to might be considered good reasoning. 2) Taking examples where the model answers question incorrectly, this approach does however have pitfalls. In some cases answer is not actually wrong, but simply not the correct token we were envisioning. an example in the IOI task "Mary[a) and John went to the store. John1s) gave the groceries to Carlec Carl is not necessarily wrong, it is just not the token we expected. 3) The third scheme consists of simply creating negative prompts that are 75 random character strings sampled from A to While this scheme does not introduce natural contrast it provides "a point of reference" " in terms of model representations and empirically works better than merely using reading vectors. We test scheme 2) and 3) in the following experiments.
We carry out multiple experiments applying the intervention to the residual stream of various LLMs at the middle layer at varying scales of a and assess how this influences performance the three
Questions are generally more complicated than this example. See subsectioo A.2 for more details.
4

---

Published as a conference paper at ICLR 2025
datasets. When deriving control vectors get control vector for each layer, previous work how- ever indicates that only applying the vectors to the middle layer is enough to induce strong changes to model outputs (Templeton et al. 2024). We generally assess the impact of the intervention at E [-1, 1] at increments of 0.1, but look at a range of [-3,3] for Mistral-7B-Instruee
3.2 EVALUATION METRICS
We benchmark performance by investigating the output logits of constrained set of potential re- sponses, where the potential responses are the collection of all answers in the dataset, similarly to what done in the Im-eval framework (Sutawika et al. 2024). We do not use an exact-match met- ric as it is particularly error-prone in O-shot prompt scenarios for smaller LLMss, where the desired structure of the output is unknown to the model. This results verbose and prosaic language gen- erated by the model to potentially be labeled as inaccurate despite providing correct answers. This issue is less pronounced few-shot prompting.
We thus use few-shot examples of the task when extracting the activations, in part alleviate the above-stated issues, but more importantly because LLMs have been shown to be in-context learners (Brown al., 2020). We furthermore cannot expect to be able improve model cannot adequately solve when the method of improving a model is derived directly from it's own hidden states. For the GSMSK task the model generates a full response with reasoning trace and we use a decoding scheme to extract the final answer.
task it
Beyond benchmarking the model outputs, we aim to understand the distribution changes as control vectors are applied. We analyze this by examining summary statistics of the resulting distri- butions and comparing them to the original model. We specifically focus on Kullback-Leibe (KL. divergence between the logit distribution of model with and without the intervention applied at various levels, providing metric for distribution change magnitude:
P(x) Pa(x)
DKL. (PIIPa) = P(x)
(5)
DEN
where P is the original logit distribution and Pa is the modified logit distribution. We addition- ally look at model entropy function of a, revealing whether the model becomes more less confident with increasing a:
= - Po(t SEX
(6)
Finally, we assess the average probability for correct versus incorrect answers as function of a. This is done to assess how the intervention concretely influences the token probability distribution.
1 N
P
(oll))
pincorrect
P
correct
(8)
where N is the number samples, ý the predicted answer to the input æ4, and where ý con- strained to the list of potential answers.
These metrics yield insights into how the control vector intervention affects model states. If certain level of control creates disproportionate spike in the KL divergence it would suggest a certain threshold in terms of bringing the model out of its usual state. We expect a linear increase in KL divergence increases. Entropy indicates the uncertuinty contained in probability distribution; we expect that the control vector improves task accuracy the entropy should drop as the probability mass concentrates on the correct answer token. Finally, assessing the probability mass concentrated at the correct token tells us concretely whether we are generally (across many samples) improving the ability to correctly solve the task.
we observe decreasing entropy and increased probability on correct tokens as accuracy increases this suggests that the control vector intervention is working intended. However, if the metrics do not align and we observe for instance decreased entropy but no increase in accuracy it might indicate that we affecting the model in unintended ways.
5

---

Published as a conference paper at ICLR 2025
We do not train any models, but extract control vectors based on hidden states of the mod- els. We evaluate model performance with the intervention on the test set and derive the control vectors from activations based on the training set. We do this to avoid data contamination although cannot be certain that the training data of the models indirectly.
have not been contaminated
3.3 MODELS
We work with models from the Pythia suite developed by FleutherAl (Biderman al. 2023). We specifically look two models, namely Pythia-I.4B and Pythia-2.8B. We analyze these models based on initial benchmarking studies and because this range allows us to analyze the efficacy of the representation engineering approach across different scales. We additionally test the approach on Mistral-7B IB-Intructt to scale the approach to a larger more capable model and to assess how the control vectors affect an instruction- fine-tunedd model.
3.4 RESULTS
In this section we report results for Pythia-I.44 Pythia-2.88 and Mistral-7B IB-Inttruct with the PCA derived control vectors applied. We generally see improved performance across models and tasks, although there are differences in the optimal value of as well as whether to scale the control vector negatively or positively.
PYTHIA
We train control vectors on the signal extracted from the A condition and apply them across ex- perimental conditions. We observe slight generalization across task variations for both Pythia- and Pythia-2.8B as seen in 2a and 9a subsection A.4 for Pythia-1.49 Pythia-2.8B improves accuracy slightly when the control vector is applied across conditions, with some variation in and KL divergence increases quadratically as a is increased both directions, see 2b The intervention mostly effective for B and BL suggesting that the control vectors encoding position-sensitive information of the indirect-object to be generated. These findings are corroborated by the entropy analysis which indicates slight decrease accuracy increases, however, decreases KI. divergence increases. We furthermore see that the probability mass on average accu- mulates slightly around the correct answer with an increase in a in 2d 2d although the average change in probability on the correct token is minute. We report only on the IOI task for Pythia models as they were not capable of adequately solving neither the bAbl or GSM8K tasks.
observe that entropy
MISTRAL
We similar results for Mistral as for Pythia-2.8B on bAbl, see Figure 3 based accuracy and KL divergence increases quadratically. We do, however, observe slight upward trend in entropy against our expectations of how the intervention should affect the final logit distribu- tion when it successfully improves a model's accuracy. We assess performance on the bAbl dataset as IOI is too simple relative to the performance of the model.
We improve the logit-
We additionally find that the control vector intervention improves model performance on the GSM8K task; the approach successfully improves the model's ability to solve the task with nega- tive a as indicated by Figure 4 While model representations are not very robust to the intervention (based on the jagged trend line), we do find that the intervention works. The KL divergence and entropy measures look quite different for GSM8K, although there does seem to be positive corre- lation between the metrics.
Finally we assess how control vectors derived from the bAbl task influences performance on GSM8K evaluation and vice-versa. In Figure 5 we report the results of of this experiment. We find an almost identical increase in accuracy on the GSM8K test set when bAbl control vec- tor is applied, and see similar trends in the KL divergence and entropy plots. We also find that a GSM8K control vector improves performance that the representation from which we derive the control vector captures aspects of the information- processing model performs when solving reasoning benchmark task.
the bAbl task, see subsection A.5) This suggests

---

Published as a conference paper at ICLR 2025
0.85
0.06
0.05
Divergennce
AL
0.80
Accuracy
0.04
DL.
0.75
0.03
0.70
0.00
1.00
0.70
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
(a) Logit based accuracy on the IOI task
(b) KL. Divergence by a
1.0
6.5
0.8
6.0
Proh. Mass
Entxapy
3.0
4.5
AL
3.5
0.0
-0.50
1.00
(c) Entropy by
(d) Probability mass by
Figure 2: Pythia-2.8B: results from PCA derived control vectors on the IOI task. We observe slight improvement across most conditions and more stable model representations than for the smaller model.
0.70
0.6
0.05
K1. Divergence
0.5
Accuracy
0.4
0.1
0.0
-2
Q
(a) Logit based accuracy the bAbl task
(b) KL Divergence by
1.0
Correct
3.5
Prob. Mass
Eatropy
3.0
0.4
0.2
2.0
-3
-2
-1.09
-0.50
0.25
0.75
1.00
(c) Entropy by a
(d) Probability mass by a
Figure 3: Mistral-7B-Insteeeee results from contrast-based control vectors the deductive bAbl dataset.

---

Published as a conference paper at ICLR 2025
0.46
0.42
Aonuracy
0.40
0.36
0.34
0.30
-3
(a) Accuracy by a on the GSM8K task
0.4
Divergencee
0.3
Entropp
0.1
6.0
-2
-3
(b) KL Divergence by a
(c) Entropy by
Figure 4: Mistral-7B-Instruu we report the changes in model performance on the GSM8K task when we apply control vectors derived from the same task.
0.46
Accuracy
0.40
0.38
(a) Accuracy by a on the GSM8K task
3.5
Divergence
Batroger
8.10
2.5
0.05
10-
0.00
(b) KI. Divergence by
(c) Entropy by
Figure 5: -instrct-B-Instruett
Instruct: we report changes in model performance on the GSM8K task when apply control vectors derived from the bAbl task varying degrees of a. Control vectors derived from bAbl can improve the model's accuracy on the GSM8K task.
8

---

Published as a conference paper at ICLR 2025
Question A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
Question
A candle meits by 2 centimeters every hour that it burns. How many centimeters shorter will candle be after burning from 1:00 PM to 5:00 PM?
-2.6
-2.6
6.) know white
takes thet takes half that much white Elbet, which is of white
We
(..) to catrulate total amount time the candle burns and then muttiply that by the rate which it melts-onuni. The candle hurns for S hours (from 1:00 PS to 5:00 FM) Durlng this time, it melts 2
C.J to calculate the tocal number of hours the candle burns and than
that takes
multiphy that by the rate at which melts unvsl. The candle burns from
number
bolts required, we can and the blue and white fiber together: Thoal bolls Blur fiter White fitter: Total holts 2
To find the totail number of bolts recuired, we simply add the amount bise and whitn fiber: Theal holts - Blue fiber White tiber. Total bolts 2 1, Total bolts 3. Therefare, it takes bolts fiber in totall make the robe.
1:00 PM 5:50 PMi which is total of
hours "The candle melts by
centimett hourlesS, So, the total amount meiting hours x contimetarsffot 10 centimeters. 'atufiiterelione, candlle
centimeters
(1/2) Tocal bolts 2.5 Therefoce, the robe takes total of 2.5 bolts.
will melt by
centimetess herefoce,
oentimeters shorter burning from 1:00 PM 5:00 PM
centimeters shorter after burning from
PM.
Figure 6: Full responses created by Distral-B-IInstruee at 0 and -2.6. The model seemingly changes its reasoning trace to reach the correct answer.
We additionally analyze how model responses change when the intervention is applied with the optimal a for a specific model and task. We report two examples in Figure 6 that showcase how Mistral-7B InInttutt generates a different reasoning trace that results in the correct answer being generated as opposed to when no control is applied (see also Figure 8 in |subsection A.3] for an incorrect example). More generally, we observe that differently sized models are affected differently by different levels of a. Whether this is because of larger models being more robust to changes in their representations or merely the fact that only modifying the middle layer representations is a smaller part of larger model is an open question.
In summary we find that control vectors can successfully be used to modify the representation of LLMs to improve performance The finding that the intervention works for a task as complicated as GSM8K is especially affirming, and even more so that it works across tasks. The qualitative examples showcase that applying the interventions retains a model's 's ability to generate coherent text and that it is not merely inducing the generation of specific subset of tokens; an important result for the intervention to be applicable outside of research, which is largely focused on improving performance on benchmark datasets.
the specified tasks for all models, although to varying degrees.
4 DISCUSSION
In this paper, we study the ability of LLMs to solve reasoming tasks by modeling 'reasoning" as a direction in the residual stream. Fundamentally, training control vectors relies on the representations learned by an LLM and the ability to modify these in a way that preserves a model's ability to gener- ate semantically meaningful text. Similar approaches have been applied to modify the "sentiment" of model outputs while maintaining semantic coherency. This research has especially focused on inducing feelings of positivity and negativity effectively creating control vectors equivalent to pre- pending prompt with "Pretend you are fceling [X]" (Liu et al. 2023 Hendel et al. 2023). We have similarly shown that we are capable of modulating the representational space of LLMs to boost per- formance on simple task while maintaining the model's fundamental ability generate coherent text. This finding hints that the "ability" of LLMs to perform well on reasoning tasks is encoded similarly to other model states, such as generating semantically positive negative outputs. Given prior literature, our ability to modify the outputs of an LLM via the residual stream is in and of itself not a surprising finding; the main contribution of the paper is that we can improve on reasoning tasks of varying complexity by nudging a model towards a specific state.
4.1 IMPLICATIONS
While there are many questions to be answered regarding reasoning in LLMs and many open ques- tions wrt. to our findings and experiments, they suggest that reasoning performance can be modu- lated similarly to how we can modulate the emotional valence of generated text or a model's ability to play chess. The developments over the past couple of years have been impressive, and they have

---

Published as a conference paper at ICLR 2025
affected the beliefs within the field regarding the limitations of machine learning models trained solely on text. Whether current model architectures are at all capable of anything like the System thinking done by humans seems unlikely, although work is being done to make LLMs reason "prop- erly" Whether such approaches amounts to reasoning in the traditional sense is still up for debate, especially reasoning is hard term to ground and define.
4.2 REASONING
While we acknowledge the deep philosophical questions surrounding the nature of reasoning, our work takes a deliberately narrow empirical approach. We investigate whether specific manipulations of model representations can improve performance without making strong claims about the nature of reasoning itself. We do not attempt to define reasoning. Instead, we are interested in using tasks that recognized and categorized as being related to reasoning, be it inductive, deductive, mathematical, or in theory anything else. These tasks differ in their prompt lengths and in how surface-level the correct solutions are (i.e, whether the solutions requires reasoning through intermediate steps or calculations). In the IOI task, the answer is present in the text, but needs be inductively recalled, in the bAbl tasks the answers also in the prompt, but requires computing through one or more intermediate steps to achieve the answer. Finally, in GSM8K, the answers are not present in the text anymore, and they also require parsing the results through multiple intermediate calculations. Through this exploration, we aim to identify actionable strategies for enhancing model performance across spectrum of reasoning tasks of scaling complexity, irrespective of how reasoning is formally defined.
tasks that require reasoning- like behaviors,
4.3 LIMITATIONS
While this paper showcases how representation engineering can successfully be used to induce better reasoning performance an inductive, deductive, and mathematical task, see certain limita- tions. We assess limited scope of models, models that are smaller than production-level state-of- the-art models. Working with smaller models has however provided the opportunity to assess the "emergence" of capabilities and symmetries as models are scaled. Working with models that were developed specifically for interpretability research furthermore opens the door for future research into the internal model dynamics related to L.L reasoning. And while the biggest model comprises a mere billion parameters, Mistral- B is relatively capable.
The 1OI reasoning task is furthermore relatively simple task, which has both advantages and dis- advantages. The results indicate that a task such as IOI may even be too simple to elicit a general representation. The task is mostly interesting to our research when models cannot correctly answer every question, hard balance to strike. Hence the switch to the bAbl and GSMSK datasets, and the Mistral-7B IIIItrutct model for additional experimentation. However, simple task also limits the scope of potential errors, which results in a cleaner picture of model behavior.
We furthermore suggest first steps towards creating contrastive pairs for reasoning to be used in representation engineering research, we however call for future work to focus on understanding how to optimally derive control vectors from model representations.
4.4 CONCLUSION
We successfully used the technique of representation engineering by first training control vectors on training set of reasoning related tasks and secondly intervening on the middle layer of the residual stream during inference. This allowed us to improve the performance of Pythia and Mistral models on a simple inductive task (IOI), the deductive bAbl task, and finally the GSM8K dataset. We illustrate how control vectors, trained using contrastive prompts, can be used to control and direct the output of LLMs, as well as how these modifications affect the internal representations of these models. Future research directions entail applying the intervention to much larger models, to more general tasks, to test different contrastive schemes as well as exploring ways of embedding this methodology into the models. In conclusion, our experiments indicate that reasoning can in part be understood to be encoded in the residual stream, similarly to how emotional valence or the Golden Gate feature is encoded in the models developed by Anthropic (Templeton al. 2024).
e.g. the most recent models from OpenAl that focus primarily on reasoning OpenAl 2024
10

---

pub/2021/Eramewwzz// Published as a conference paper at ICLR 2025
STATEMENT OF REPRODUCIBILITY
We are committed to aiding reproducibility of our work and encourage further research in this direc- tion. The source code for our implementation, developed as an internal framework, will be published alongside this paper facilitate this process. Our framework is built as a wrapper around PyTorch, enabling easy extraction of hidden dimension representations and application
control vectors. The models described in section 3.3 were loaded using the HuggingFace API and details model version are described there. The training process for control vectors is detailed in section [2.] The calculation of experimental results, including all metrics used, is thoroughly descrihed in section 3.2 Information about datasets used, including any preprocessing steps, can be found in section 3.1 and
Our code is publicly available through Github, along with documentation to guide users through the setup and execution of experiments.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization, July 2016. URL httg//bb//6601
attp://arxur wttp///wwww.g//////./....0 06450 arXiv:1607.0644 [es, stat].
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Lan- URL http:://rxix http:/awwwvv..a.aaaa.a//a abs/
guage Models Across Training and Scaling, May 2023.
2304.01377 arXiv:2304.01373 [cs].
Trenton Bricken, Adly Templteon, Joshua Batson, Brian Chen, and Adam Jermyn. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Antropic AI Technical Paper, 2023. URL
2uuu pul/s023/
morosseanttic features/ index..dexx.///// .htll
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020. URL 14165 arXiv:2005. 14165 [cs].
http://arxiv/ wwtp:/iwwwrrrr////. wbtpg//abb//00/
François Chollet. On the Measure of Intelligence, November 2019. URL http:://axxiv..rrt arXiv:1911.01547 [cs].
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, and Nicholas Joseph. A Mathematical Framework for Transformer Circuits, December 2021. URL html#
/ttpp:://transforme--cccc def wtt.-piiiiieeee--aaiiii
Roce Hendel, Mor Geva, and Amir Globerson. In-Context Learning Creates Task Vectors. In
Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 9318 89333, Singapore, December 2023. Association
for Computational Linguistics. doi: 00.18653/v120033 7/aclentholooyyorrq//// fww.iigs.iiiinnnniinaeme
http:////.....in/eemm/meammp//p/ findings-emmll.6224
URL https:
624
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. October 2023. URL
httpp:://ppenrviee..eee
Philip Johnson-Laird. How We Reason. Oxford University Press, October 2008. ISBN 978-0-19- 030.0/00000099513000000000000000 URL attps:/ academic.
955133-0. doi: 00.1009/apmffc0000/78/9999553333 bum.ccmm wtt..com/oooko1990

---

Published as a conference paper at ICLR 2025
Adam Karvonen.
Emergent World Models and Latent Variable Estimation in Chess- Playing Language Models, March 2024. URL PXXi:2243311599 [es].
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context Vectors: Making In Context Learn- ing More Effective and Controllable Through Latent Space Steering, 2023. URL https:
//arxiv.orq abs/ 2311/066111161 Version Number: 3.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Fac- tual Associations in GPT, January 2023a. URL /tl.:ttt////rwiw.r///aaass ar arXiv:2202.05262 [cs].
05262
Kevin Meng. Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-nditing Memory in a Transformer, August 2023b.
22222
arXiv:2210.07229 [es].
Philipp Mondorf and Barbara Plank. Comparing Inferential Strategies of Humans and Large Lan- guage Models in Deductive Reasoning, June 2024. URL httpp://axxw..org/////t abspgg//b///2/5
14856 14X25 [cs].
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent Linear Representations in World Mod- els of Self-Supervised Sequence Models. Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, Singapore, Decem- 1tcp//00002.aacoooool1.1...
ber 2023. Association for Computational Linguistics. doi: 10.
URL
OpenAl. Learning Reason with LLMs, September 2024. URL https: /ppeaai.//ppa....//// com/
index maddx//eaarnig-teee reason with-whhhhhh wit
Ellie Pavlick. Symbols and grounding in large language models. Philosophical Transactions. Series
A, Mathematical, Physical, and Engineering Sciences, 381(2251):2020041, July 2023. ISSN
1471-2962. doi: (0.10985ista.2222.0041
Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, nopperl, gakada, tttyntiann, researcher2, Julen Etxaniz, Chris, Hanwool Albert Lee, Zdeněk Kasner, Khalid, LSinev, Jeffrey Hsu, Anjor Kanekar, KonradSzafer, and AndyZwei. EleutherAViineevaaaatoo---nne pettps:/zzenor wwwpeenddoo.rgd/ooooooooooo htrg//doi/1..oo htt.:///eeoodo.....e....
v0.4.3, July 2024. URL
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, and Trenton Bricken. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, May 2024. URL
o monosemant icity/
index.html html
Eric Todd, Millicent L. Li, Arnb Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function Vectors in Large Language Models, February 2024. URL lattp://axxiy org/ abs/2310. 15213 arXiv:2310 10. 1.513 [es].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need, August 2017. URL http:
/ //arxiv. http/vorrg/bbbl abs 1aabb//11/0 1706.. 03763 arXiv:1706.03762 [cs].
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the Wild: Circuit for Indirect Object Identification in GPT-2 small, November 2022.
C0593 arXiv:2211.008: [cs].
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren. Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation Engincering: A Top-Down Ap- proach to AI Transparency, October 2023. URL httt:://txxi..tt/t//// arXiv:2310.01 [cs].
wwwpariiiii..//... org/ wwwporggggbbba.aee http:///bs/221..110 01405
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. CONTROL VECTOR NORMS
As can be gathered from equation 2 any control vector will have norm that equivalent the average of the hidden dimension activations at a given layer of the model. For the contrastive pairs (equation B it will be equivalent the norm one gets when subtracting the "opposite" state from the "desired" state. This is however not the case for the control vector based on a PCA, because standard PCA implementation scales the output vector to have an L2 norm of We cannot know exact size of |He|| as this is dependent on the learned parameters and [3 of the LayerNormfuuction Assuming y = and = O, which are the usual initial values (Ba et al. 2016). a rough estimate of the norm of the hidden dimension, is || = vd, the value however varies a lot dependent on the model and between layers. We therefore scale the PCA control vectors by ||llll of the training data to allow for a more direct comparison between the control vector methods. As indicated by figure we get identical norms for the two methods, while the actual control vectors are different.
Lene
(a) Reading Contrast CV Pythia 1.4b
(b) Reading Contrast CV Pythia 2.8b
Home 200 SyBaridt
Noverd Dotrraeee tycka
(c) PCA Contrast CV Pythia 1.4b
PCA Contrast CV -Pythia 2.8b
Figure 7: Control Vector Norms
A.2 DATA
INDDRECT-OBBICC--DEEIIICCCCC (IOI)
The IOI task examples were generated using the following schema:
[A] and [B] went to the [location]. [B] gave the [object] to [A].
the BABA (B) condition the schema was:
[B] and [A] went to the [location]. [B] gave the [object] to [A].
We provide example here, also shown in 3.1
Mary and John [3] went to the store. John gave the groceries to Mary
Examples are split into (X, pairs of the following form and fed to the model with as the input and y the correct generation:
13

---

Published as a conference paper at ICLR 2025
("Mary and John went to the store. John gave the groceries to". "Mary")
When extracting model representations (X, y) pairs are combined into a single string containing the input prompt and the correct answer. When evaluating the performance of the derived control vector the models only receives X, and we evaluate whether the token with the highest logit of the potential answers is equivalent to the correct label, y. We used an 80/20 train-test split to train and evaluate the performance on control vectors. We do not perform any additional pre-processing.
BABI DATASET
We used the bAbl dataset, focusing on task 15, which involves simple deductive reasoning. Each example in the dataset consists of: A passage containing factual statements, question based on the passage and 3) the correct answer. We provide an example here.
Passage: Mice are afraid of wolves. Gertrude is a mouse. Cats are afraid of sheep. Winona is mouse. Sheep are afraid of wolves. Wolves are afraid of cats. Emily is mouse. Jessica is a wolf.
Question: What is Gertrude afraid of?
Answer: Wolf
Examples are split into (X,y) pairs of the following form and fed to the model with as the input and y the correct generation:
("Passage: nCtts are afraid of wolves. \nGertrude is mouse. mouse.\nJessiee AJJssica is a cat. \QQustiin: What Jessica afraid of? vn" wolf""
When extracting model representations (X, y) pairs combined into single string containing the input prompt and the correct answer. When evaluating the performance of the derived control vector the models only receives X, and evaluate whether the token with the highest logit of the potential answers is equivalent to the correct label, We used an 80/20 train-test split to train and evaluate the performance on control vectors.
We do not perform any additional pre-processing and have downloaded the data directly from http:/g/....a.....oooooa/aeee/ea
1-SHOT PROMPTING
When 1-shot prompting, we simply provide the input prompt with example of how solve the task. The model is prompted with an our-of-distribttion example in order not to nudge the model towards a specific answer. This takes the following form:
Example's Saamplle.nPassageee massagee: Giraffes are afraid of lions. \nElephnnts are afraid of snakes.), nJohn is saaffe.)nQuestion: What is John afraid of? nAnswer: Lion' Lionn)n(Passagee and question)
INSTRUCTION PROMPTING
Since instruction fine tuned models able follow instructions, found that prompting the model to output one word answer made sense, and was also found to work better empirically.

) n(potential one-shot example) pleease give one word answer as in the example above. 
], and question)
14

---

Published as a conference paper at ICLR 2025
A.3 GSM8K FLIPPED RESPONSE
The following is a concrete example from the experiment with Mistral-7B-Instruct and the PCA- contrast control vector with a -266. This example showcases how the reasoning trace is still interact when the intervention is applied, and that something in the intervention leads the model to accurately end up at the correct answer of 3 as opposed to 2.5. The "reasoning" in the two samples is identical until the point where the model generates the text "which is bolt of white fiber". This additional information seemingly allows the model to generate the correct response when finally answering the question.
Response before intervention:
"A robe takes bolts of blue fiber and half that much white fiber. How many bolts in total does take?"
Question:
Correct Answer: 3.0
Predicted Answer: 2.5
solve this problem, we need to add the amount of blue and white fiber required for the robe. We know that the robe takes 2 bolts blue fiber. 2. We also know that it takes half that much white fiber. 3. So, to find the total number of bolts required, we can add the blue and white fiber together: Total bolts Blue fiber + White fiber. Total bolts = 2 (1/2) Total bolts 2.5 Therefore, the robe takes a total of 2.5 bolts."
Full Response:
Response after intervention:
Question: "A robe takes bolts of blue fiber and half that much white fiber. How many bolts in total does take?"
Correct Answer: 3.0
Predicted Answer: 3.0
Full Response:
solve this problem, we need to add the amount of blue and white fiber required for the robe. We know that the robe takes 2 bolts blue fiber. 2. We also know that takes half that much white fiber, which is I bolt of white fiber. 3. To find the total number of bolts required, we simply add the amount of blue and white fiber: Total bolts Blue fiber White fiber. Total bolts 2 + 1. Total bolts 3. Therefore, it takes 3 bolts fiber total to make the robe."
15

---

Published as a conference paper at ICLR 2025
Question
A new program had 60 downloads in the first month. The number of downloads in the second month was three times many the downloads the first month, but then reduced by 30% the third month. How many downloads did the program have total over three months?
0
3
The number of downloads in the second month was three times as many as the downloads in the first month. So, multiply 60 60 3 180 downloads. Step 2: Calculate the number of downloads the third month. The number downloads the third month reduced by 30% in the third month. So, multiply 180 0.7 (which the decimal equivalent of 30%): 0.7 126 downloads. Step 3: Calculate the number downloads over the three months. To find total number downloads, we add the number of downloads each month: 60 180 126 366
Let's salve this problem step step; 1. In the first month, the program had 60 downloads. In the second month, the number of downloads was three times many the first month, there were 60 180 downloads. An3. In the third month, the number of downloads was reduced by 50%, there
(0.3 180) 120 downloads. To find total number of downloads over three months, the number downloads each month: 180 the program total 560 downloads over the three months.
downloads. Therefore,
downloads. Therefore, the program had a total 366 downloads over the three months.
Full responses created by Mistral- 7B-nttt-B at a = 0 and ( = 3. The model seemingly changes it's reasoning trace to answer incorrectly.
Figure 8:
A.4 ADDITIONAL RESULTS
We train control vectors on the signal extracted from the condition and apply across experimental conditions. We observe slight generalization across tasks for both Pythia-1.4B and Pythia-2.8B as seen in [a] and Da Pythia-1.44 1.4B improves accuracy when the control vector is applied with positive conditions, and KI. divergence is relatively stable in the positive direction and very unsta- ble in the negative direction resulting in much higher and more varied KI. divergence (figure (b)). These findings corroborated by the entropy analysis which indicates slight decrease racy increases, but an even stronger drop in entropy as - -1, suggesting that the intervention pushing the probability mass towards specific incorrect token (figure 9c). We furthermore see how the probability mass on average accumulates around the correct answer in figure 9d although the increase with positive a is small.
16

---

Published as a conference paper at ICLR 2025
Accuracy by Alpha for pythia-1.s -bb4b
DeL by Alpha for pythia-1.4
0.5
KI. Divergence
AL
Accuracy
0.4
0.3
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.75
1.00
-1.00
-0.50
0.00
0.25
0.50
1.00
Alpha
Alpha
(a) Logit based accuracy on the 1OI task
(b) KL Divergence by a
Entropy by Alpha for pythia-1.41 1.4b
P-mass by Alpha for pythia-1.4b (Eval Task: A)
10
A
10
Correc:
Incornect
AL
Entropy
BL
Prob. Moxs
022
-1.00
-0.75
-0.50
-0.25
0.00 Alpha
0.25
0.30
0.75
1.00
0.55
(c) Entropy by a
(d) Prohahility mass by a
Figure 9: Pythia- 1.4B results. Results from PCA derived control vectors. Slight improvements in
accuracy across all conditions, and malleable model representations.
Accuracy by Alpha for Mistral-7B-Instrucce
0.4
Accuracy
0.1
Alpha
(a) Accuracy on GSM8K for Mistral-7B: Baarrll-B-Instruee
DxL by Alpha for Mistral- -nstrrctt
Entropy Alpha Mistral- Beatral-7B-Instruee
KL Divergence
--
Entropy
Alpha
Alpha
(b) KL. Divergence by a.
(c) Entropy by a.
Figure 10: Mistral-7B results. Results from PCA-based CVs on random contrastive pairs.
17

---

Published as a conference paper at ICLR 2025
A.5
CONTROL VECTOR CROSS-TASK RESULTS
Accuracy by Alpha for Mistral-7B-mmsrrutt00]
047
Acouracy
0.6
Alpha
(a) Accuracy on bABI for Mistral-7B -B-tsrrrett
DRL by Alpha for Mistral-7B Ba-all----IIntttee
Entropy by Alpha for Histral-7B-Insteee
4.50
KL Diverreace
0.08
4.00
0.06
Entropy
3.75
0.02
3.25
3.00
0.00
2
Alpha
Alpha
(b) KL Divergence by
(c) Entropy by
Figure 11: Mistral-7B results. Results from PCA-based CVs trained on the GSM8K task evaluated
on bAbl.
18


Reviews:
[{"review_id": "y1FeUideT9", "reviewer": "Reviewer_kzoH", "summary": "This paper proposes improving reasoning performance of LLMs through representation engineering. The authors introduce the \"control vector\" derived from the residual stream of the model as an inference-time intervention, which modulates the representational space to enhance reasoning capabilities without additional training. The approach is evaluated on inductive (indirect-object identification) and deductive (bAbI) reasoning tasks with models from the Pythia suite and Mistral-7B-Instruct. The authors suggest that the intervention improves task performance and reasoning may not fundamentally differ from other token generation tasks implied by the results.", "questions": "See Weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality:** The representation engineering approach utilizing the residual stream for reasoning task offers a novel view. \n\n**Quality:** The study is methodologically clear. The use of models at varying scales and the combination of several metrics (accuracy, KL divergence, entropy) offers a comprehensive evaluation.\n\n**Clarity:** The paper clearly explains the steps for deriving control vectors.\n\n**Significance:** The problem of whether reasoning requires distinct treatment than other tasks is important.", "weaknesses": "1. **Task complexity**: The reasoning tasks (IOI and bAbI) used are relatively simple, potentially limiting the conclusions drawn about \"reasoning\" in general. Results might differ on more challenging tasks. Specifically, mathematical reasoning has received great attention as a difficult type of reasoning and has been widely adopted as benchmarks for LLMs. Could you add addtional experiments on the GSM8K or MATH dataset to explore how this approach applies to more complex reasoning tasks?\n2. **Model scope**: Testing is limited to smaller models (up to 7B parameters), which could restrict applicability of the conclusions. Larger models might capture reasoning dynamics differently and yield varied results under similar interventions. Could you include experiments for Pythia-12B to investigate how the method scales with model size?\n3. **Hyperparameter choice:** As indicated by the experimental results, the choice of $\\alpha$ actually dominates the final outcome. However the paper does not provide a systematic procedure to choose $\\alpha$ beforehand. To make the framework applicable to a wide range of tasks and models, could you provide a general routine to determine the value of $\\alpha$ and its interpretation?\n4. **Contrastive pair selection and control vector derivation:** The paper uses random character strings as \"negative\" samples and PCA-based control vectors according to empirical performance. I wonder if we can gain more insights for these choices. Could this empirical advantage be due to the nature of IOI task, where answers could be semantically correct but not expected? This is also why I suggest to include math reasoning in point 1, where ambiguity of answers is minimized. Could you report the results for different types of contrastive prompts (e.g., partially correct responses) on the GSM8K / MATH dataset to investigate how they impact control vector efficacy?\n5. **Experiment settings:** The Pythia models are only reported for the IOI task, which seems insufficient to me. Could you provide results of Pythia-1.4B & Pythia-2.8B on the bAbI task also?\n6. **Result interpretation:** If I understand the figures correctly, there is an apparent decrease for B & BL condition and almost no improvement for A & AL condition with the Pythia-2.8B model in logit-based accuracy. This raises my doubts about the effectiveness of the approach and is probably my biggest concern. The performance gap between A condition and B condition may imply the statistical learning bias in disguise rather than genuine reasoning. Therefore the conclusion that \"there is no intrinsic difference between the process of reasoning and other information-processing tasks performed by LLMs\" does not seem valid to me. Could you make further explanation regarding this experiment result?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bx9NPnOOKa", "reviewer": "Reviewer_vEQT", "summary": "The paper uses control vectors with a residual stream added to transformers, trained using contrastive prompt pairs, to improve reasoning ability in a couple of tasks. Some improvement is achieved with some ranges of a controlling parameter alpha.", "questions": "- How can we find the best alpha in a new deployment situation?\n\n- How does it compare to chain-of-thought or a fine-tuning that highlights the prompts that are associated with good reasoning and bad reasoning? Can a similar information be provided contextually?\n\n- Why do the contrastive pairs have to be associated with good and bad prompts instead of good and bad reasoning instance?\n\n- What do you mean by: \"We furthermore cannot expect to be able to improve a model on a task it\ncannot correctly solve when the method of improving a model is derived directly from it’s [sic] own\nhidden states.\"? The whole purpose is to improve the model on instance it wouldn't solve without the residual stream?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This is an insightful technique that can shed light on the behaviour of LLM and reasoning.\n- Well written.", "weaknesses": "- Novelty. They apply an existing technique, adapting it to a couple of architectures and deriving prompts pairs for \n\n- The results are only slightly better for some ranges of alpha. \n\n- Very few models, very small models, and just two datasets.\n\n- The insights about reasoning are somewhat limited. They say that \"reasoning is not neccesarily different from any other form of processing done by LLMs\". This is either a tautology if we are assuming that LLMs do reasoning or needs to be compared to some other kinds of specific processing that are not reasoning, such as perhaps retrieval, etc. If the meaning is that the patterns are no different, then the paper should compare with the other patterns. In any case, for two tasks, it's limited how much we can extrapolate for the understanding of what reasoning is.\n\n- The second task (IOI) is too simple, and in some cases, as the authors say, the answer is not fully specified.\n\nMinor\n- \"illustrated in 1\" -> \"illustrated in Figure 1\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oB0NOYzBoA", "reviewer": "Reviewer_1E6m", "summary": "The paper proposes a new way to improve reasoning in LLMs via representatoin engineering. It basically reads activations from the residual streams of LLMs during reasoning, derives a control vector from this, and applies this vector during inference time to modulate the representation space. The results are evaulated on inductive and deductive reasoning tasks using Pythia and Mistral Instruct 7B. This intervention seems to improve reasoning without further training, which suggests that reasoning is not too different from other information processing done by LLMs.", "questions": "1. How does performance intervention compare to finetuning? I believe that would be a fair baseline.\n2. Are there any hypotheses as to why there is disagreement between the KL divergence and entropy results?\n3. Is performance intervention sensitive to layer choice? How so?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality - As far as I am aware, such control vector intervention is a new idea, and the proposed framework for deriving control vectors and their impact (KL div, entropy) is a useful contribution.\n\nClarity - The paper explains its methods and findings in a structured and thoughtful way. The transformer architecture for conceptually reframing this problem is helpful.\n\nSignificance - I think the result that reasoning is an emergent general information processing ability is useful and important.", "weaknesses": "1. The paper only tests the approach on two models. Larger models will need to be used to validate these findings.\n2. I think the IOI task is too simple, even though the bABI task is representative. The authors should explore a wider range of reasoning tasks to strengthen the conclusions.\n3. The contrastive approach using random strings seems a bit arbitrary. Probably a more principled method would be more fruitful.\n4. The probability mass results seem to disagree a little with the entropy KL divergence results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edXAfEI412", "reviewer": "Reviewer_imCU", "summary": "The paper proposes a method to elicit improved reasoning from LLMs at inference time. Similar to inducing writing style or emotional valence, the researchers suggest that reasoning ability is a controllable characteristic which can be suppressed or enhanced. \n\nUsing representation engineering, they read residual stream activations from a training set containing positive and negative reasoning trajectories, and these are used to derive a control vector which, colloquially speaking, points in the “better reasoning” direction. In order to improve results, the paper uses contrastive pairs, and further takes only the strongest PCA component of the pairs in order to only select for the “better reasoning dimension”. At inference time, it is applied to the residual stream to improve the model’s reasoning capabilities. \n\nThe paper carries out experiments on a bAbI simple deductive reasoning dataset task and the indirect-object-identification (IOI) task. These are applied to two Pythia models and a larger Mistral-7B-Instruct which demonstrate that applying this method yields improved results compared to the baseline (average probability of correct vs. incorrect). It also shows that the model’s entropy and KL divergence are modified due to this intervention as one would expect.", "questions": "The main questions the paper would benefit from answering (following the above discussion) to strengthen the claim:\n1. Is the “reasoning” task specific? Does it benefit other tasks? \n2. Which types of incorrect reasoning are reduced when using the method? \nEither analyzing the errors or showing improved reasoning on a held-out reasoning task (i.e. using an OOD reasoning task) would strengthen the claims appearing in the paper, which can be argued to be at least somewhat related to non-reasoining improvements. \n\nFollowing the points from above:\n- Please explain how the negative sample for the contrastive pairs were constructed or show an ablation study\n- Please explain whether the entropy behaves as expected for the Mistral result. \n\nA few more minor points that would improve readability and clarify confusions: \n1. Unless I missed something, the paper does not explain how the reading vector (or control vector) is constructed from the latent representation of the examples. Specifically, there are several tokens in each prompt, and it is not clear how these are treated. It is implicitly understood from the notation that this is not per-token, so perhaps the last one is taken. It would be useful if explained explicitly (perhaps even discuss different design choices), possibly by providing psuedocode, formulas or a diagram.\n2. The claims are that the results are improved for all tasks but in different directions. This is taken to mean “different sign of alpha” but it is not clear. There appears to be no natural direction, possibly(?) due to the PCA process. \n3. The evaluation as scoring from logits makes sense, but it is not explained how these are chosen, and so possibly may be a source of confusion. \n3. It is stated that the KL divergence is expected to rise linearly with alpha, and reported that this is indeed the case. Looking at the plots (Figures 3,4) this seems to be quadratic in nature for the larger models, so it rises linearly with the magnitude of alpha, not with alpha.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strong points:\nThe paper provides a clear overview of the motivation and methodology. It is well-structured and self contained.  \nThe paper performs a simple and novel training-free application of an existing method to improve basic capability of LLMs, where it was previously only used for eliciting style modifications. While other papers have shown that, e.g. truthfulness exists as a direction (or control vector) in representation space, it may still be counterintuitive that a capability such as reasoning is also a “direction” in representation space. \nThe paper uses auxiliary methods (Entropy & KL divergence) to analyze the results of applying this method. \nThe paper’s most validating result, as mentioned in the results section: an 8% improvement in the bAbi task over baseline.", "weaknesses": "Method:\nThe paper reports a successful application of the method to two datasets. Due to the need for contrastive pairs in order to generate the control vector, it requires positive (good reasoning) and  negative ones (bad reasoning). \nThe paper discusses 3 schemes for generating the “negative” pairs:\n- Asking the model to produce an incorrect answer. This, as the paper notes, is not really a negative example since the model uses correct reasoning when generating an incorrect answer. \n- Selecting cases in which model response is not the expected one (which, as the paper notes may still be “not actually wrong” just not the expected answer)\n- Selecting a random string of 75 letters as the model’s output reasoning trajectory, which is surely “wrong”. This is pointed out correctly as being ‘unnatural’.\n\nThere are a couple of points that are not clarified:\n- The paper mentioned testing the latter two schemes, but it is unclear from the results how these are tested. This should be explained in detail (e.g. are they equally mixed as negatives. This requires clarification (or perhaps an ablation study to understand the effects).\n  Please provide details on how the different negative example schemes were combined or compared in their experiments. \n- Furthermore, without analyzing the errors made by the model, it is not entirely obvious that either of these schemes actually push the model towards correct reasoning. For example, if most errors are “out of distribution” as in the example in the paper: ”Mary[A] and John[B] went to the store. John[B] gave the groceries to Carl[C], merely forcing the model to remain “within” distribution of tokens it’s already seen will already improve its results even if the modified reasoning paths only choose names at random from A and B. Similarly, making sure the model is pushed away from random strings of letters will do the same, and it is hard to say if reasoning has indeed been improved, as in each case (bABi and IOI) there appears to be a natural incorrect token (the “other name”) measuring its probability as an additional measure (as opposed to just the correct token) may help understand these issues better. \n\nResults and analysis:\nIt is stated that the entropy is expected to decrease as the model veers towards the correct token, concentrating its mass to it. This would mean that it would be negatively correlated with accuracy. However, this does not appear to be the case for Figure 4, the bAbi on Mistral, unless I missed something. In fact the opposite appears to be true. In this plot the entropy also appears to be positively correlated with the probability mass, which requires explanation. If this is indeed the case, please provide a more detailed analysis of the result, and possibly the discuss the correlation between these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gpDwHkIwN7", "reviewer": "Reviewer_TVGr", "summary": "This work uses the control vector method to promote task-performant representations in LLMs for IOI and certain bAbI tasks, using a new (but unspecified) software framework, with moderate success. The paper's substantive claims are:\n\n1. These tasks are reflective of LLM reasoning ability\n2. The corresponding task-representation vectors in the residual stream are appropriately representative of reasoning in LLMs\n3. The observed increase in task-performance after applying the method is evidence for an improvement in reasoning as a broad faculty (as opposed to only the specific task at hand.)\n4. There is no intrinsic difference between reasoning and other information-processing tasks as far as an LLM is concerned.", "questions": "- If you are framing this around \"Reasoning\", what kinds of philosophical commitments are you making? For example, it seems you implicitly want to frame reasoning as something that cannot be characterised purely behaviouristically, which would necessitate the kind of \"going into internal representational structure\" that you seem to want to frame the control-vector technique as; so maybe it's worth doubling down on Reasoning as something involving mental states and representations with causal force. However, since you also want to conclude that reasoning is not an innately distinguished ability in LLMs, are you implicitly discounting prompt-level interventions such as CoT and their efficacy at improving task performance? What conception of Reasoning makes it amenable to be improved by residual stream modification but apparently not by structured CoT?\n\n- Why is there nothing about the software? Evidently it was a useful suite of tools to have gotten you through all of the experiments, why say nothing about it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is very clearly presented what the paper wishes to achieve, and the methodology is clear.\nThe introduction adequately situates this work in the context of prior work in the field.\nTo the best of my knowledge, the control-vector method has not been tried before on this particular task.\nI particularly enjoyed the meticulous introduction that covered the conceptual and mathematical bases for the remainder of the work; it read well as a mini-tutorial on the techniques used here.", "weaknesses": "It feels as if this paper wanted to be about a software tool or framework, but \"reasoning\" was more attractive as a topic.\nOf the four substantive claims concerning reasoning, I don't think a single one is substantiated.\nThese 4 criticisms are the foundations of my current stance on the paper, and I would be happy to raise my score accordingly if these concerns were adequately addressed (by e.g. weakening claims or strengthening their support.)\n\n1. The tasks are reflective of LLM reasoning ability.\n\nThat is a stretch, right? IOI is out of the question. bAbI is a sort of edge-case, but consider that Weston et al frame the suite as toy tasks, and that the success of various memorisation architectures on the taskset afterwards have prompted the development of new benchmarks that aren't hackable by exploiting statistical regularities. The claim that this is about \"Reasoning\" is too heavy for the experimental foundations.\n\n2. The corresponding task-representation vectors in the residual stream are appropriately representative of reasoning in LLMs\n\nThis is essentially what the paper needs to establish the existence or nonexistence of empirically; that there exists some broad and task-general faculty in LLMs (that we would call Reasoning), and that this faculty has a manipulable representation in the residual stream. Even if I am willing to take all the literature on residual streams at face value, the onus is still on the authors to establish that the tasks they have chosen are really reflective of Reasoning --- that's claim 1.\n\n3. The observed increase in task-performance after applying the method is evidence for an improvement in reasoning as a broad faculty (as opposed to only the specific task at hand.)\n\nThis is crucial. If applying the control vector only improves an LLMs ability to do bAbI tasks, then we're not saying anything about a capacity to Reason: we would instead call it \"a capacity to do bAbI\", similarly to a \"capacity to play chess\". If the authors insist on framing the contribution around Reasoning, then at least there ought to be some evidence of generality: if it turns out that boosting performance for bAbI also increases performance on the GSM suite or ARC or anything else, then that would be very interesting to see! Due to the \"context-freeness\" of Reasoning, there is no way around this particular hurdle apart from empirical demonstration of the efficacy of the technique in general performance boosts across substantially different domains. I can't see any way to get around this purely rhetorically: whatever argument X the authors would like to make that their current experimental basis is sufficient evidence for improvement at a broad capacity for Reasoning, I can take that argument X and apply it for the cited work (line 59) that residual stream modification improves chess and othello performance, undermining the authors' claims to conceptual novelty.\n\n4. There is no intrinsic difference between reasoning and other information-processing tasks as far as an LLM is concerned.\n\nIt's unclear that the empirical observations obtained in this work are sufficient to substantiate this claim. It does not help the matter that the authors deliberately avoid taking any stance on characterising what Reasoning might be, or what properties it might or might not have (lines 38, 438), which makes it difficult to evaluate what the claim is even supposed to mean. I would like to view this refusal to take a stance on what-Reasoning-is charitably as scholarly circumspection, but when juxtaposed with unsubstantiated claims about Reasoning ability in LLMs, I'm afraid that citations like Johnson-Laird read more like a checklisting exercise than material engagement with the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)