Improving Instruction-Following in Language Models through Activation Steering
Authors: Alessandro Stolfo
Vidhisha Balachandran
Safoora Yousefi
Eric Horvitz
Besmira Nushi
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
Mechanistic Interpretability
Instruction-following
Activation Steering
LLMs
Primary area: interpretability and explainable AI
TL;DR: We improve instruction-following in language models by using activation vectors to steer models towards satisfying constraints.

Abstract: The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabilities, we derive instruction-specific vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-time control over instruction following. Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present. Additionally, we explore the compositionality of activation steering, successfully applying multiple instructions simultaneously. Finally, we demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation. Our code and data are available at https://github.com/microsoft/llm-steer-instruct.

Published as a conference paper at ICLR 2025
IMPROVING INSTRUCTION-FOLLOWING IN LANGUAGE MODELS THROUGH ACTIVATION STEERING
Alessandro Stolfo
Vidhisha Balachandran
Safoora Yousefi
Eric Horvitz?
Besmira Nushi²
ETH Zürich Microsoft Research
ABSTRACT
The ability to follow instructions is crucial for numerous real-world applications of language models. In pursuit of deeper insights and more powerful capabili- ties, we derive cnstruction-speifii vector representations from language models and use them to steer models accordingly. These vectors are computed as the difference in activations between inputs with and without instructions, enabling a modular approach to activation steering. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion, providing inference-ting control over instruction following. Our exper- iments across four models demonstrate how we use the activation vectors guide models to follow constraints even without explicit instructions and to en- hance performance when instructions are present. Additionally, explore the compositionality of activation steering, successfully applying multiple instruc- tions simultaneously. Finally, demonstrate that steering vectors computed on instruction-tuned models can transfer to improve base models. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation. Our code and data are available at https://gituu..commmmcooooofffff httprsooo//lmmmttttteinsstttttt
INTRODUCTION
Instruction Coatuucton-followingg capabilities of large language models (LLMs) have enhanced their practi- cal applications for real-world usage. These advances are powered by instruction-tuning methods (Ouyang et al. 2022 Bai et al. 2022] Wei et al. 2022] Sanh et al. 2022 Chung et al.. (0024). which align the model's 's responses with user objectives, addressing the gap between pre-training and end- user needs (Askell et al. 2021). Instruction tuning allows users to specify constraints on attributes like format, tone, or length, which direct the model's behavior and output (Zhou et al. 2023b) Zhang et al. 2023 Lou et al. 2024). Gaining deeper understanding of how LLMs internally represent and follow these instructions essential for developing more controllable and reliable models] !
In this paper, we use mechanistic method to investigate how language models internally represent various instructions and use these representations influence and control the model's behavior. Prior research has shown that vector representations can be computed for tasks learned in context (Hendel et al. 2023: Todd et al., 2024 Liu et al.) 2024) and various stylistic and semantic input
features (Zou et al. 2 Azaria & Mitchel 2023 Zheng et al. [2024] Templeton et al. | 20244] Marks
& Tegmark 2024 inter alia). These representations can be used for activation steering (Subramani ct al. 2022): directly intervening
the model's activations to guide the generation. This approach
has been successfully applied to control text attributes such as honesty (Li et al.) [023] Qiu et al,
2024 Zou et al. 2023), sentiment (Tigges al.] (024), output style and tone (Turner et al. 2023
Liu et al. 2024] Scalena et al. 2024 von Rütte et 2002), harmfulness (Arditi et al.) 2024
Wang
& Shu 2024), and sycophancy (Panickssery et al. 2024 van der Weij
2024).
However, user instructions in generative tasks can be more complex and involve multiple parameters that need to be attended and satisfied during generation (Sun et al. 2023). For example, users may
Work done while at Microsoft Research. Corresp. to @tttoooooaeehhhc and www.hha..sssssttee www.mmmcccc..oooooomom com We use instruction to refer to specific constraints that can be added during interactive sessions with I.I.Ms modular way modify extend base question or request. We further elaborate this definition in $

---

Published as a conference paper at ICLR 2025
Input
Input
Output
List some facts about
Write rubric for performance revlew of software engineer
Lionel Mosdi
"Io": pprtoroonneerreeee wttponneeeeeeeeeeeeeeeeeee rwubbbi
Format the output as JSON,
"type" "ooject"
"prspertrest:
Extract Residual Stream Activations
"cnde, quxiity"
"type "onjecr"
"propertis""
"ocumennttion" (f
Input List some facts about Lionel Messi.
"type" "string" "descripton": 'Proper
documentation
Add Steering Vector
H0C
Instruction
X x/+
"roodabiity" L
"rape" "string"
Steering
sptioni: Cadde roadaall and
Steering Vector Computation
Steering
wellstructured!
Inference Time
Figure 1: Instruction Steering Process. Steering vectors are computed the difference in residual stream activations between imputs with and without the instruction. These vectors are then applied during inference to adjust the model's activations, guiding it to follow the desired instruction.
ask for a response include bullet lists, have a certain number of sentences, or include/exclude spe- cific content. The complexity of such instructions stems from the high number of possible variations, making it impractical generate post-training data for all scenarios. In the quest for finding more efficient methods for controlling instruction following and guided by previous research in language model representations, we pose the question: Can we efficiently extract vector representations that encode and control specific cestruction-folloviiii behavior?
We investigate this using a contrastive, additive steering method TTurner et al. 2023 Panickssery et al. 2024) that computes the difference in activations between inputs with and without an instruc- tion (Figure left), and applies this difference to guide the model to better follow instructions on new imputs (Figure right). While previous work focused high-level stylistie aspects, target more diverse, verifiable instructions (Zhou et al., 2023a) that complement the hase request's seman- ties (e.g., formatting. length constraints). These instructions multiple possible instantiations (e.g.. "Do not mention the word [keyword]") It remains unclear whether models represent such instructions linearly and whether these representations can be used to reliably elicit specific behaviors, which would enable finer control of LLM outputs.
lower-level and more specific, with
We conduct experiments using the Phi-3 (Abdin al., 2024). Gemma 2 2B and 9B (Gemma Team 2024]. and Mistral 7B (Jiang et al., (023) models, focusing on three types of instructions: output format ($3), output length ($4). and the inclusion/exclusion of specific words ($5). Our results on the IFEval dataset (Zhou et al. 2023a) provide evidence that vector representations can encode wide range of instructions and enhance the model's iustuction-foolowinng performance. Notably, we demonstrate that activation steering not only helps models follow constraints when no instruction provided in the input, but it can also reinforce instruction adherence even when instructions are explicitly present, potentially countering instruction drift (Li et al. 2024p. Furthermore, we show that it is possible to simultaneously steer for multiple constraints, such as controlling both format and length ($6) Finally, present compelling evidence that cross-model steering-using vectors computed on an instruction-tuned model to steer a base model-is effective and can sometimes yield better adherence than same-model steering (. suggesting new possibilities for transferring task- specific skills across models, similar to "task arithmetic" (Ilharco et al. 2023).
Our work represents an important step toward operationalizing techniques from mechanistic inter- pretability to achieve practical improvements in scenarios with real-world utility.
2 STEERING FOR INSTRUCTION ADHERENCE
In this section, we define the types of instructions we consider 2..b. introduce the methodology used compute the vectors and the steering procedure ($2.2). and describe the experimental setup, including the data, metrics, and evaluation details 2.3
2.1 TYPES OF INSTRUCTIONS
The concept of instruction mnstruction-followinng has been used to describe the broad capability of a model to answer any zero-shot query (Wei et al. 2022 Sanh al. 2022). However, following prior work
2

---

Published as a conference paper at ICLR 2025
(Zhou et al. 2023a bj Sun et al. 2023), adopt a more specific definition of instruction that refers to a constraint applied to a particular aspect of the model's output. These constraints are self-contained, modular, and can be imposed ability of the model to follow the instruction from its factual knowledge and domain-specifie skills. In our work, we focus on three specific types of instructions:
various base queries. This dcfinition decouples the
1. Format instructions, which dictate how the output should be presented. For instance, the model may be asked produce responses in a specific format (e.g., "Provide the answer in JSON format") or highlight parts of the response in a particular way (e.g.. "Use asterisks to emphasize least two sections of the answer").
2. Length instructions, which specify the desired length of the output (e.g., "Answer using at most three sentences").
3. Word-specific instructions, which control the inclusion or exclusion of specific words or phrases in the ouiput (e.g., "Do not include the word A/ the answer").
2.2 STEERING PROCEDURE
Activation (or representation) engincering involves constructing vectors of activation values which cause desired changes to output text when added to the forward passes of a frozen LLM (Zou al.) 2023 To identify a direction in the model's residual stream that encodes information about a specific instruction,
use technique called difference-in-nee inneens (Belrose 2023). This method ef- fectively isolates key feature directions in the model's s internal activations (Marks & Tegmark] [2024] Tigges et al., 2024) and has been used to control various behaviors such as refusal and sycophancy (Arditi et al.
2024, Panickssery et al., 2024). We adapt this method to support finer-grained instruc- tions with multiple possible instantiations (e.g.. varying length, varying words to include and ex- clude). The process involves pairing two versions of the same request: one with only the base query (Litt some facts about Lionel Messi") and another that additionally includes the instruction we want to represent (e.g., "List some facts about Lionel Messi, ensuring the output is valid ISON").
Let us denote these two inputs by x (base query) and x (base query with instruction). and consider a set of N such pairs (I; xt), i E (1...., N). Let Xi,1, x E Raaddd be the values of the residual stream vector the two queries at the last token of the input at layer ( E {1, 4}. We isolate the internal representation corresponding to the instruction by computing the difference in the residual stream vectors between the paired inputs. More formally, compute vector u R representing the steering direction at layer I for a given instruction as:
N
v where
Σ (x.) -x,). N
1
Averaging different base queries allows us to capture the activation values most closely associ- ated with the instruction, independent of the base query. The computation of the steering direction is carried out using the representations at the last token of the input, which effectively encapsulate the model's behavior not only for the next-token-predictionn task but also for the entire generation that follows (Todd et al.) 2024 Scalena et al. 2024)..
After identifying the steering direction, we compute the steering vector by re-scaling the unit vector u, by a coefficient, E R. For format instructions, we use a systematic scaling approach where the value of c is selected to ensure that the residual stream activations are mapped to their mean value on inputs that contain the instruction in question. In particular, during a forward pass on a new example with residual stream values x' E X at given token position, we compute:
Ut, where
u
This dynamic adjustment allows the model effectively incorporate the constraint without over- or under-correcting its behavior. For length instructions, which have more continuous nature, we experiment and show results with different values of e, illustrating their impact on the model's
TThe model's residual stream is the per-token hidden state of dimensionality dmodel consisting of the sum of all previous component outputs (Elhage al. 202T).
3

---

Published as a conference paper at ICLR 2025
output. Finally, for word-specific constraints, we compute the weight using Eq. and additionally perform a small grid search over neighboring values on a held-out set of examples to tinetunee the steering effect. The steering vector cu, is then added to the corresponding residual stream layer and the forward pass is resumed with the updated residual stream value & X + cu,, This procedure is carried out at single layer across all token positions, motivated by previous findings that show models tend to deviate from instructions as they generate more tokens (Li et al., (0224). To find the optimal layer for steering, we perform a sweep across a subset of the model's i's layers, measuring the effect on held-out set of queries and performing a perplexity- -based quality check. Details about this procedure are provided in Appendix
2.3 EXPERIMENTAL SETUP
Data. We use augmented version of the IFEval dataset (Zhou et al. (202a),, which consists of 25 distinct instructions, each paired with multiple base queries and expressed in different phrasings, for total of 541 prompts. To evaluate format instructions, we focus on subset of 163 examples that specifically relate to the output format. This subset includes instructions such as "The entire output should be wrapped in JSON format," requests use a particular language (e.g.. "Please respond using only the Punjabi language"), formatting requirements like "Wrap your entire response with double quotation marks," and casing instructions (e.g. "Answer using lowercase letters only"). A gener-
complete list of the instructions used is provided in Appendix B For length instructions, ate prompts by concatenating base queries from IFEval to instructions derived from templates such as "Answer using at most {n} sentences." For word-specific instructions, we use a subset of IFEval containing instructions about the inclusion or exclusion of keywords. The subset has 203 examples, and each example contains prompt with single wwword-related instruction. For steering vec- for computation and layer selection, we construct a separate set of synthetically generated prompts by combining base queries from IFEval with corresponding instruction descriptions avoid test information leakage. Additional details about the data used are provided in Apps. Cland D
Metries. To quantify the models' adherence to format instructions, we compute the "loose" instruction-folloiing accuracy using the official IFEval evaluation script. For length instructions, we count the number of words or sentences in the model's output. For word-specific constraints, we verify the presence or absence of the specified keywords in the model's ' response, again using the IFEval evaluation script. We assess the statistical significance of differences in average instruction- following accuracy with and without steering using McNemar's s test (McNemar) 1947). Results where steering leads to a significant improvement (p-value < 0.01) are marked with an asterisk (*).
In addition to assessing the model's instruction-followiin capabilities, it is important to verify that the model still effectively addresses the base query, even when generating under constraints. To measure the overall quality of the response, we set up GPT-4o-based evaluation. For each base query x, GPT-4o generates a set of five yes/no questions aimed at assessing the quality of the re- sponse For instance, given the query "Write essay about the history of Martin Van Buren's 's presidency." GPT-40 generates questions such as "Does the essay provide context on the political, social, and economic climate during Van Buren's presidency?" and "Is the essay written in clear, grammatically correct English, and does it follow a logical structure?". Next, given the original query x, model's 's response v, and the generated questions we prompt GPT-4o to answer each question, provide a rationale for each yes/no response, or reply with "N/A" if the question is not ap- plicable. We compute the proportion of questions answered positively for a given query and average this score across all queries to obtain response quality score. We repeat this experiment 3 times and report the mean and standard error. Additional details are provided in Appendix
Evaluation. We conduct experiments using the instruction-tunee versions of Phi-3 Mini (Abdin jt al., 2024), Gemma 2 2B, 9B (Gemma Team (004)), and Mistral 7B vO. 1 et al., (023). In addition, we investigate the transferabilityy of steering vectors from the instruction-tuned to the base versions of Gemma 2B and 9B. All models are evaluated in a zero-shot setting, with outputs
'Although they are not strictly related to format, we include language constraints (e.g., "the answer should be in German") in this group, as, like other formatting instructions, they modify the surface-level presentation of the output without affecting the underlying content.
'Note that while the model's response may be influenced by specific instructions, these instructions are not provided to GPT-4o during the evaluation. The goal here is to focus the quality assessment solely the comprehensiveness and relevance of the response with respect the hase query.
4

---

Published as a conference paper at ICLR 2025
(a)
"Quotation" Instruction
(b) "Urdu Language" Instr.
Cosine Simlarity
Cazine Siniterity
Steering Vector Projections. The projection onto the vocabulary space of the contrastively motes tokens that are semantically related the respective instruction.
Table 1:
0.9
68
vectors pro-
10
10
20
Layer
Layer
(1) Diffenent Querry Same Initr.
Ceffernt Same Inatr.
sy
Instruction
Layer
Top Tokens
(3) Quedes
Querees
JSON Format
18
-It, -sson, -6, JSON _PRC, _T, FOR, AND
Capitalize
28
Figure 2: Residual stream similarity across layers. Phi-3's residual stream activations show higher cosine similarity between examples with the same instruction compared to those without, indicating effective cap- ture of instruction-relevan features.
Highlight Text
-t, (t, +1
Lowercase
18
_loue", _loweat, rusa .tl, %/ Inndian rai,, .India .die, .im, _aies, .gener
Builet List
*),
Quatation
_,, .7, _'</ IIalam, Fakistan, PPak
Undu Language
24
Hindi Language
German Lang.
16
decoded greedily. We assess our method's performance by steering the model under two input settings: (1) with text instructions provided in the input, and (2) without any text instructions. These settings allow us to explore two questions: Are the computed steering vectors informative enough to guide the model's behavior even without explicit instructions? And can steering further improve performance when instructions are provided in the input?
FORMAT INSTRUCTIONS
Representational Analysis. To assess whether the model effectively captures instruction-related information in its internal representations, analyze Phi-3's 's residual stream activations at the last token of the input by calculating cosine similarity across three sets of inputs: (1) pairs of inputs that share the same base query, one with and one without the instruction; (2) inputs with different base queries but the same instruction; and (3) inputs with different base queries and no instruction. High similarity in set (2) would suggest that the model captures shared feature (the instruction), while we expect set (1) to show relatively high similarity due to shared base queries, and set (3) to show lower similarity due the absence of any shared features. Figure Ssoows shows how these measures evolve across layers for two instructions. For the "quotation" instruction, where the model is asked to wrap the output in quotation marks, there a clear difference between sets (2) and (3), indicating that the instruction is partially captured (green vs. red lines in Figure (h) For the "Urdu language" instruction, set (2) shows higher similarity than set (1), suggesting strong representation of the in- struction (green vs. blue lines in Figure p). These results indicate that the model can effectively encode instruction-relevan features at the last input token, which supports steering model behavior based on specific constraints. Additional representational analyses are provided in Appendix H
Steering Vector Computation. We compute steering vectors for each of the 12 format-related instructions in the IFEval subset and for the 19 language-based instructions specified in the dataset. During the selection of the optimal steering later, we compare the validation score with and without steering to ensure that the steering intervention leads to an improvement. If no layer shows an improvement in the validation score, no steering is applied at test time. Details about the layers selected for steering are provided in Appendix E To ensure that the steering vectors effectively capture the information related to the instructions, we inspect them by projecting the vectors onto the model's vocabulary space (nostalgebraist) 2020 Geva et al. 2022). We project the vectors using the model's 's unembedding matrix and examine the vocabulary tokens with the highest logit values. Table presents the top tokens associated with several of the instructions we consider. We observe that the tokens promoted by the steering vectors are semantically related to the intended instruction, providing an initial validation that the vectors are capturing the desired features.
Steering Results We first evaluate steering on inputs without explicit text instructions (Figure 3p) In this setting, the caaruction-folloiing accuracy without steering hovers around 10% as the input has no information about the instruction. This non-zero accuracy reflects cases where the models incidentally satisfy the instruction, such as when a model rephrases a sentence without using com- mas, thus accidentally meeting the "no comma" constraint. When steering is applied, consistent increase in instruction adherence across all models, with accuracy improving to approxi- mately 30%. This shows that the steering vectors encode meaningful information of the instructions
observe
5

---

Published as a conference paper at ICLR 2025
(a)
Accuracy w/o Text Instructions
(b)
Accuracy with Text Instr.
(c) Phi-3: Accuracy per Instruction
0.4
0.9
Capital
0.3
Accuracy
Accuracy
Hightihts
0.2
0.7
Capitalze
D.2 6.4 O.G. oid conoraas
0.1
Formst
Phi-"
Germee
28 IT
76 Instr.
Mistrel
Germa 2
96 IT*
0.5
Phi-3"
Gemma 2
26
76 Instr.
Mistral
Geriiaa
96 IT
imter Sullet liets
Title
Mukipte
Stc.
Inference
Steering
Inference
Steering
Std. Inference
Steering
Figure 3: Format Instructions. (a) mostruction-following accuracy without explicit text instructions shows significant improvement with steering across all models. (b) Steering enhances accuracy even when text instructions are provided. (c) Per-instruction accuracy for Phi-3 without text instructions.
and can be effectively used to steer the models toward the intended behavior. Next, we evaluate on inputs with instructions provided in the text (Figure 3p). As expected, the instruction-following accuracy without steering is higher in this setting, ranging between 60% and 90%. Nevertheless, steering still results in a notceable performance boost for two out four models, demonstrating that steering can enhance instruction adherence even when the instructions are explicitly given.
To further analyze which instructions benefit most from steering, we break down Phi-3's perfor- mance by instruction on input without explicit text instructions (Figure 33)) Notably, we observe significant improvements for instructions such as "Lowercase," which requires the output to be en- tirely in lowercase characters, and "Highlight Text," which asks the model to emphasize parts of the response using markdown syntax. However, certain instructions display variability in steering effectiveness. For instance, the "End Checker" instruction requires the model to finish the response with a variable, input-dependent sentence (e.g.. "Hope you agree with me," or "Is there anything else I can help you with?"), Because the steering vector is computed by averaging over all examples containing this instruction, it likely fails to capture the specific sentence required in each individual case, reducing its ability to steer the model toward precisely following this type of instruction.
Response Quality Evaluation. Figure compares the aver- age differences in response quality scores for the four mod- els. The two leftmost groups of bars show the score changes due to steering, with and without input instructions (calculated only for cases where steering is applied). Satisfying specific constraints during generation may lead to less comprehensive responses and hence following instructions is expected to im- pact the response quality. To show this effect, we also present the differences in quality scores observed when explicit text instructions are added the base queries without steering. We observe small decreases in quality score due to steering in the setting including instructions (with the exception of Gemma 2B) while the decreases in the setting without instructions are slightly larger but comparable to the effect caused by simply adding the instructions text. However, we also observe in- stances where steering compromised the quality of generation- examples where the model generated nonsensical tokens or repeated itself. These failures likely result from suboptimal steering or partially insufficient objectives during the search for layers to intervene, as steering for specific properties can increase perplexity or reduce fluency (Turner al. 2023 Stickland al. 2024).. We provide some examples from the few cases we observed in Appendix
Changes in Response Quality Score
Avg. Quality Score Della
-0.1
-0.15
Steering w/o Enstr.
Steoring w/ Instr.
Stearing vs. w/o Instr.
Pri-3
Gemmaa 2n IT Garrma
Mistral
Figure 4: Quality Score. Aver- age response quality score changes for four models, comparing the ef- fects of steering with and without input instructions to the effect of adding instructions without steer- ing. Steering decreases quality similarly to adding instructions.
4 LENGTH INSTRUCTIONS
Steering Vector Computation. Length constraints be specified in various ways, such by the number of sentences, words, or lines. However, unlike format instructions, it seems impractical to compute separate steering vector for each possible length constraint. Instead, we focus captur-

---

Published as a conference paper at ICLR 2025
(a) Length Steering Weights
(b) Accuracy Per Length Constraint
(c) Length: Pre- Post-Steering
0.9
0.6
LLnnghh constraint:
Probability Density
Probabilic' Denta 
0.8
Accuracy
0.4
0.7
0.02
0.2
0.6
0
200
400
0.5
14
2
3
Length (+ words)
Weight
40
20
10
MaX. of Sentences
Length
Inference
Steoring
Inference
Steering
Figure 5: Length Instructions. (a) Modulating the steering weight e effectively adjusts response length, with larger values leading to more concise outputs. (b) Steering enhances adherence to the maximum length constraints when they are explicitly specified in the input text. (c) Shift in the response length distribution upon steering for outputs constrained to maximum of 5 sentences.
Table 2: Example of Steering for Length Control. Responses to the same prompt with increasing steering weights, resulting in progressively shorter outputs in terms of sentence and word count.
Weight
Resp. to "Write movie plot that involves dream, fist fighting, and superpower"
# of Sents. # of Words
the bustling city New Haven, young. introverted artist named Alex dreums world where his
to life. One night, Alex has vivid dream where he discovers he possesses unique superpower: the ability to bring [-.]
15
324
In world where dreams can be harnessed introverted artist named Leo discovers he has the unique ability enter and manipulate the dreams others [...]
source of power. young.
9
228
world where dreams can manifest into reality, a young woman named Elara discovers she possesses the rare ability to fight with her fists in her dreums [...]
10
10
203
world where dreams can manifest reality, young girl named Lila discovers she can unleash her latent superpower through her dreams [..]]
20
3
81
young girl dreams of superpower, fights rival in dream, and unleashes her power to save her village from destruction.
23
ing more general notions of conciseness and verbosity. To achieve this, we compute vector repre- sentations for instructions that prompt the model to be brief (c.g.. "The answer should be brief"") or to provide more detailed responses (e-g.. "Provide a long answer"). We synthetically generate these prompts by appending such instructions to base queries from IFEval. After computing these steering vectors, we evaluate the model on a separate set of IFEval base queries, this time appending length instructions specified in terms of the number of sentences (e.g.. "Answer using at most 3 sentences"),
Steering Results. Unlike format instructions, steering the model for length constraints allows for continuous modulation, enabling interpolation between varying degrees of conciseness or verbosity. To explore this, we manually adjust the steering weight e and examine how this affects the response length. On a set of 50 base prompts without any explicit length instructions, from Phi-3 using different values of c, measuring the distribution of output lengths. As shown in Figure 55,, increasing the value of effectively shortens the model's responses: larger steering weights produce increasingly concise outputs. We provide an example of different outputs generated on the same inputs with different steering weights in Table 2 Next, we assess whether steering for length is effective even when the model is provided explicit length instructions. Using a set of 200 base prompts, we introduce instructions that request the model to limit its response a maximum of n E {l.....5 5) sentences. First, we evaluate how often the model adheres to this constraint without steering (light blue bars in Figure (p).. Then, on the same inputs, we apply the steering vector for conciseness with a weight of e = 20 (dark blue bars in Figure 5p). Across all five values of n, we observe a significant and consistent improvement in how often the model's responses comply with the length constraint. Figure 5 further illustrates how steering shifts the response length distribution toward shorter outputs while still allowing for variability and avoiding overly short responses. We also conducted experiments with steering for longer outputs and for exact output length, observing positive results (Appendix D). Finally, response quality evaluation results are provided in Appendix F
generate responses
7

---

Published as a conference paper at ICLR 2025
(a) Instructions
(b) Word-Exclusion Instructions
Table 3: Example of Word Ex- clusion. The instruction alone fails to exclude "step." but steer- ing successfully removes it.
Instr.
w/ Instr.
w/o Instr.
w/ Instr.
0.5
0.8
0.5
0.5
0.0
0.5
0.4
0.4
0.4
0.2
0.2
0.2
Resp. to "Whal are the sleps be followed for the documentation of GM in SAP? Do not use the word step."
Steer
Gomma
Gemmaa
Pt-3* Gemtma 28
PH--3 Gomma
Std. Inferonce
Stoo/ing
Std. Inference
Documenting General Moster
in SAP involves (provides guide]
Figure 6: Word-specifie Instructions. (a) Steering improves keyword inclusion accuracy for Phi-3 and Gemma 2 2B IT, with and without explicit instructions. (b) Negative steering reduces the occurrence of undesired keywords, in both settings.
following these sleps, you
create compeecheniiv [..]
An Documesting a General Mustr
Yes
(GM) in SAP: jprovides guidel By following this comprehensive guide.
you create well sell-struccured
5 WORD-SPECIFIC INSTRUCTIONS
Word Inclusion. For this set of instructions, we compute word-specific steering vectors. To gen- erate these vectors, we append different phrasings of request to include specific word w (e.g., "please include the word {w} } in your response") to set of base queries. These prompts are used to compute a vector representation for the "include word {w]" instruction. While this requires a separate steering vector for each keyword at inference time, the vectors can be generated on-the-fly using arbitrary base queries unrelated to the keyword itself. Furthermore, these vectors can be com- puted with a small number of examples; in our experiments, we use only 20 examples. The models are then evaluated on the subset of IFEval that contains keyword inclusion/cxclusion constraints The results, presented in Figure (4, demonstrate the effectiveness of steering for word inclusion. By applying the word-specific steering vectors, we observe a notable increase in the frequency with which the model successfully includes the requested keywords in its responses.
Word Exclusion. For keyword exclusion, we initially used the same procedure, appending instructions like "ensure the word w does not appear in the answer" to base queries and com- puting the corresponding steering vectors. However, upon in- specting the steering vectors, we found that projecting these
(a) Multi-instr.: Format & Length
Length Accuracy
0.8
0.6
0.4
vectors onto the vocabulary space resulted in high logit val-
ues for the tokens corresponding to the words meant to be ex- cluded. In other words, adding these vectors to the model's residual stream actually increased the probability of generating the very keywords we intended to exclude. To address this is- sue, instead of computing exclusion vectors, we compute the vectors for inclusion and then subtract them from the model's 's
0.2
0.4
0.6
0.6
Fomat Accuracy
(b) Multi-instr.: Casing & Word Ex.
Casing Accuracy
residual stream. This technique effectively steers the model
away from using the words captured by the vector. Figure 6 shows the results of steering for keyword exclusion using this approach. We observe that subtracting the inclusion vectors significantly reduces the frequency of the undesired keywords in the model's 's responses, both in cases where no textual in- structions are present and where explicit exclusion instructions are provided in the input. Tabpe[]reporrts an example where the instruction to exclude a specific word ineffective on its own, but applying steering successfully enforces the constraint.
0.6
0.7
0.8
0.9
Word Exclusion Accuracy
w/o Irstr.
Steering*
w/ Instr.
Steering
Figure 7: Multi-instruction Steering. Steering for two structions improves adherence to both.
6 MULTI-INSTRUCTION STEERING
Next, present results using our steering approach handle multiple instructions simultane- ously. With the same method as in the previous experiments, we steer the Phi-3 model for two
'IFFval prompts may contain instructions for the inclusion exclusion of multiple keywords in the same example. We separate such cases into inputs with single-keyword instructions.
8

---

Published as a conference paper at ICLR 2025
(a) Cross-model Steering: w/o Instr.
Cross-model Steering: Instr.
Cross-model Steering: Length
0.6
0.2
Probability Density
0.15
0.4
0.1
0.1
0.2
0.05
0.05
Gemma 2B
Gemma 9B
Gemme
Gemma 9B
100
Length (# of words)
Std. Inference
Std. Inference
Seme-model Steering
Same-model Steering
Weight
200
150
100
Cross model Steering
Cross-mes model Steering
Figure 8: Cross-model Steering. (a.b) Steering vectors from instruction-tuned models transfer effectively to base models, with cross-model steering outperforning same-model steering in Gemma 2 2B. (c) Cross- moos--moeel steering for length instructions shortens output as steering weight increases.
instructions at once, applying the steering vectors at the layers previously identified as optimal for each individual instruction. We opt for injecting multiple steering vectors simultaneously at differ- ent locations instead of combining multiple stecring vectors into a single one, as previous work has shown the latter approach be largely unsuccessful (van der Weij et al. 2024).
Figure 7 shows results from experiments with steering two instructions the same time: format (all 13 instructions) and length (Figure 71, as well as lowercase and keyword exclusion (Figure In both cases, steering leads to improvements on both axes. These findings suggest that steering for multiple instructions
is feasible and can lead to performance gains across different constraints. How- ever, we anticipate that issues may arise in certain cases, particularly when dealing with conflicting instructions, where further refinement may be necessary balance competing constraints.
7 CROSS-MODEL STEERING
Instruction-uue tuned models show improved eestruction-oollviiii abilities, suggesting they may also form more effective instruction representations, This raises the question: can we leverage the repre- sentations computed on instruction-tuned models to steer a base model more effectively? This idea motivated by prior work showing that fine-tuning reinforces existing mechanisms language models (Prakash et al. 2024) and that instruction tuning causes minimal weight changes (Lee et al.) 2024 Jain et al. 2024). Additionally, linear representations derived through mean activation dif- ference and sparse autoencoders (Huben et al. 2024) have been shown to be transferable between base and chat models (Panickssery et al., 2024 Kissane et al., 2024). To study this our setting, conduct experiments with the base and instruction tuned versions of Gemma 2 2B and 9B. We focus on format instructions, using the same data and procedures outlined in $ The key difference that we apply steering vectors computed the instruction-tur models steer the base models, comparing this approach to same-model steering (vectors computed on the base model).
In both the base query-only setting (Figure (b) and with explicit instructions (Figure 8p), steering base models using vectors computed on instruction-tuned counterparts yields performance gains that demonstrate the transferability of instruction representations. Notably, in the case of Gemma 2 2B, cross-model steering outperforms same-model steering across both settings, highlighting its potential for more effective instruction adherence in smaller models. We also explore cross- mroel steering for length instructions. As described in $4 we compute steering vector for conciseness from the instruction-tunes version of Gemma 9B and apply to the base version. Figure R shows that increasing the steering weight shortens the outputs, demonstrating that representations learned in instruction-tume models can still meaningfully modulate the output in the base model's space.
This is the first demonstration that cross-model steering-using vectors from instruction-tuned models-cee can outperform same-model steering in base models. This finding suggests that special- ized representations from fine-tuned models can be leveraged to steer base models more effectively, opening new possibilities for composable transfer learning in instruction based tasks where task vectors may originate from different models that instruction-tunel in specialized domains.

---

Published as a conference paper at ICLR 2025
8 RELATED WORK
Instruction Following. Training models to follow instructions is crucial for improving LLM per- formance and ensuring safe deployment, with various methods developed enhance instruction adherence (Ouyang et al. 2022 (anh et al., 2022. Wei et al. 2022 Bai et al. 2022 Chung et al.,
2.24], and datasets designed to train and evaluate instruction following behavior Ye et al. 2021
Wang et al.) 2022, Gupta et al.) 2022, Finlayson et al. 2022 Mishra et al., 2022; Longpre et al. 2023 Köpf et al. 2023). Natural language instructions have demonstrated significant promise in
2023b). However, capable models still struggle with tasks that require outputs to satisfy fine aee-grandd, hard constraints (Sun et 2023)
providing fine-ge -eeeegrrnedddee control over model outputs (Zhou al.
and tend to drift from adhering vated by these challenges, our work investigates how to improve instruction-folloving behavior by directly intervening on the model's activations at inference time.
constraint as the generation lengthens (Li et al. (024). Moti-
Language Model Representations. Our approach is inspired by prior research that shows it is pos- sible obtain vector representations encoding information about tasks on which language model has been trained (Ilharco et al. 2023 Huang et al. 2024) or tasks learned in context (Hendel et al.) 2023 Todd et al. 2024). These studies are part of a broader body of work that examines the linear representation of features such as truthfulness (Li et al., 2023; Azaria & Mitchell 2023 Marks & Tegmark 2024), sentiment (Tigges et al. 2024), harmlessness (Zou et al.
2023 Zheng et al. 2024). sychophancy (Perez et al. 2023 Panickssery et al., [2024] (Sharma et al., 2024). factual knowledge
(Gurnee & Tegmark| 2024), and refusal (Arditi et al. (224)). In addition, recent works have em-
ployed sparse autoencoders to identify feature directions in unsupervised manner (Bricken al.
2023
Huben et al. 2024| Templeton et al. 2024). A shared hypothesis across these works is that
LLMs represent features concepts linear directions in activation space (Mikolov et al. 2013
Bolukbasi et al.) 2016 Elhage et al.; [021] Nanda et al.) 2023] Park et al.) [202|] 2024). While
recent studies suggest that not all features may be linearly encoded (Engels al. 2025 Csordás
ct al. 2024). the linearity assumption has been effective for both concept erasure (Ravfogel et al.
2020
Belrose al., 2023 Shao al. 2023 Guerner et al., (0224) and model steering.
Model Steering via Activation Editing. It is well-established that the generation of a language model's output can be manipulated by directly editing activation values during inference (Dathathri
et al. 2020 Subramani et al., 2022). Recent studies have shown that this approach can effectively steer models to be more honest and truthful (Li al.] 2023 Qiu et al.] (024), sycophantic (Pan-) ickssery et al. 2024| van der Weij et al., 2024). morally aligned with human values (Zou et al.)
2023 Lu & Panickssery (024), or to display different sentiments, output styles, and languages
Turrerr et al. [023] Ju et 20244 Tigges et al., (004)202 Scalena et al.] 2024 von Rütte et al. 2024).
Steering methods have also been used to control the model's uncertainty (Rahn et al. al.| 2024 adopt
different personas (Cao et al., 2024), provide alternative factual answers (Hernandez et al.
2024
and respond to harmful requests (Arditi et al.
2024
Wang & Shul 2024). Similarly to some of
these works (Burns et al. 2023 Turner et al.
2023 Panickssery et al. 2024 Arditi et al.
2024
van der Weij et al. 2024), we compute steering vectors based on input pairs that differ by a specific feature-in our case, the presence absence of instructions. However, while previous studies have focused on high-level concepts such as sentiment, style, and safety, we focus on lower-level, hard constraints defined through natural language instructions, allowing for finer-grained control of the model's 's output.
9 CONCLUSION
We demonstrated the effectiveness of activation steering for improving language models' adherence to instructions. By computing steering vectors based on activation differences between inputs with and without instructions, we guide models to follow constraints related to format, length, and word inclusion/exclusion. These vectors capture meaningful instruction representations, enabling mod- els to meet constraints even without explicit instructions, while also enhancing performance when instructions are present. Additionally, we show that models can handle multiple constraints simul- taneously, offering flexible framework for controlled language generation. Finally, we explore cross-mos mooss steering, revealing that vectors computed on instruction-tundd models can improve the behavior of base models, in some cases surpassing same-model steering.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We submitted our code and data as supplementary materials, and they will be open-sourced upon publication. $2. outlines our methodology in detail. Information about the datasets we used, as well as the procedures for obtaining and augmenting them, is provided in $2.3 along with Apps. C and D 22.3 also contains the details of the evaluation metrics we employed and the process for generating model outputs. AppendiixFpprovinn score metric. Additional implementation and experimental details are included in Appendix G
further details on the implementation of our quality
ACKNOWLEDGMENTS
We would like to express gratitude to Andy Arditi, Ahmed Awadallah, Natasha Butt, Varun Chandrasekaran, Arthur Conmy, Nico Daheim, Saibo Geng, Suriya Gunasekar, Neel Joshi, Sid- dharth Joshi, Mazda Moayeri, Neel Nanda, Harsha Nori, Kyle O O'Brien, Vibhav Vineet, and Vilém Zouhar for their valuable feedback and insightful discussions throughout the development of this project. We also extend our thanks to the Al Frontiers group at Microsoft Research, with special recognition to Ece Kamar for practical support. Alessandro acknowledges the support of armasuisse Science and Technology through a CYD Doctoral Fellowship.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jiammin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengehen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Prancetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang. Yu Wang. Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: /ttp::://aaxvvv..o www./orgg/bbb/....oo.oo 2404 11.444.1422 14219
Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal language models mediated by a single direction. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a labora- tory for alignment, 2021. URL BtLphttss:///xii// wwwp:awwwwwwww..aaaaaaaaaaaaaaaa abs//bss//// 00861
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 967-976, Singapore, dec 2023. Association for Computational Lin-
11

---

Published as a conference paper at ICLR 2025
http:////2222.iiingseeemmmma findiins--mnll.... URL https://accannthoooo.... 2023. Findings-emnll 68
guisties. doi: 10.18653//1/2023....
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfied-Doddss, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL /ttp:tt::://axxi wwtp:////rr//aao/oaooooo.oosss http://.bb/222.....o/ 05862
Nora Belrose. Diff-in-means concept editing is worst-case optimal: Explaining a re- sult by Sam Marks and Max Tegmark, 2023. URL https:://blcg.eeeuchee.
diff io--in-mee innmeanss
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL pittps://epenreeet Forum? wtwpmmmdddawIwppW/WW
Steven Bird and Edward Loper. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pp. 214-217, Barcelona, Spain, jul 2004. Associ- 3031
ation for Computational Linguistics. URL https: /alltts:/aaclnnnt http://aaaaaaacahhcceeeae
Tolga Bolukbasi, Kai- Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. Man is computer programmer woman is homemaker? debiasing word embed- dings. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http:::/proceeiing...errisssssss file/ a 0ee.4440660004422220077777746663
files//aaper20116/
pdd
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity:: Decomposing language models with dictionary learning. Transformer Circuits Thread,
wew.iiccfeeaeueeeeeeeeeeeeeee / http://mooooon---eeeeee
Indx.. html
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL https::/ http:::////eeeeeeeeeeee net wwr.moririioooooo ETKGbyyO
Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In The Thirty-eighth Annual Conference Neural Information Pro- cessing Systems, 2024. URL httphttt///ppereeeee/ /ttp:////ppereiiieeeem net htt.//frummpp (omummidiii id= kuzdYo
Hyung Won Chung, Hou, Shayne Longpre, Barret Zoph, Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttue language models. Journal of Machine Learning Research, 25(70):1-53, 2024. URL https:
//www jmlr http:/paperr/////2333002 0tmmmmmmm
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Yonatan Be- linkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 248 262, Miami, Florida, US, November 2024. Association for Computational Lin- URL https: /ttEps:/aaclanee hww.atoooonoogg.o..oooooo howgolllggyoggg/o//ooooooo
guistics. doi: 10. h/0663//w///222..baacoooxo///// 2024 bl ack.axbnpool11177
12

---

Published as a conference paper at ICLR 2025
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to con- trolled text generation. In International Conference Learning Representations. 2020. URL
net forum? 1ttp://eeweeeeee//et.i1ieee- d==l edd
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
URL https: http:////taaaaaaooooooooooo www.crrrcctt... wwtp:ppppppo/peepp 2tw00//a//eworr...... index. html
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps:: http::/:///ppeeeeeee..e..o wtwpp//wwww....ttttooraaaaaaaaaa
Matthew Finlayson, Kyle Richardson, Ashish Sabharwal, and Peter Clark. What makes instruction learning hard? an investigation and new challenge in a synthetic environment. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4 414-426, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. doi: 10.18653/v wwtp0002.emlppmmm...... URL
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL https:
2008000118
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornisaa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, dec 2022. Asso- ciation for Computational Linguistics. doi: 10. //aclanthology.oo wwwpethhooooo....22222222222 emnlp-manll....
URL https:
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric 15054
notion of causal probing, 2024. URL
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey Bigham. In- structDial: Improving zero and few-shot generalization in dialogue through instruction tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 505-525, Abu Dhabi, United Arab Emi- rates, dec 2022. Association for Computational Linguistics. doi: (0.18853/v1/20222coo 33. URL https:: pestps://aclanthholee hww.aaanthhooooy..or///22222 emn 1p- main.33
main.
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth Interna tional Conference on Learning Representations, 2024. URL
Eorum? id= 18xbmm=-88xxFF11.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with NumPy. Nature, 585(7825))357-366. sep 2020. doi: 10. 10.00888/4118860002264222 URL https: pestts://doi.. /tt://dd...o/ //tp//oooggo.//////
Kevin Sheppard, Tyler Reddy,
020 2 2649-2
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: EMNLP 2023, pp. 9318-999 99333,, Singapore, dee 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023 122.:355.....2.....n....a--a..e org/2003. i ngs emmgs-emmmll emn 624
mandiing--emnl.222. URL https: //aaaaaaaooooty......oaoa
13

---

Published as a conference paper at ICLR 2025
Evan Heraandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge rep- resentations in language models. In First Conference on Language Modeling, 2024. URL htttps://cpenreviee net/ /tt.r/oorumnii-ADSE00000
Shih- Chin-Cheng Huang. Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Ka Kai Hsiao, Richard Tsai, and Hung-yi Lee. Chat vector: A simple approach to equip LLMs with instruc- tion following and model alignment in new languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 10943-10959, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. URL https://aclantteeee
org/
acl acl-long.50 590
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. / net forum?
SG=F76bwRSLee.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:////// /ttp///ppereeeieeee/e / net/ / forum?ide 6tCKwf8- jrj
Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? mechanistic study. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL httptpss://peeeeiiiiiii
Forum?id-BS2cuUs
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. SAEs (usu- ally) transfer between base and chat models. Alignment Forum, 2024.
URL
https:/ //t::://wwww.. www.:aaliggmeentfrurmmmm wwt.posts////oooooottttoo saes usua tyanse base and-chat models
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models, 2022. URL https::/ hnttps://openrevies net/ wwt.:nettouuuaeee forum? 6p3AuaHAFiN
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte. Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov. Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. In A. Oh, T.. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 77766947681.. Curran Associates, Inc., 2023. URL ht / neur neur wttpuriisscccppaaae.ee wwtp:per.ppppppceeeeeeeeeeee..ee faapperrflles files/ (32p:///00011111333222222222eee2
Paper Datasets.
and www.BBencchha....... pdf
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada
Mihalcea. A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity. In Forty-first International Conference on Machine Learning, 2024. URL hetps //openzeview wwt./eeotemdddaeeeeeeoeeo
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from language model. In Thirty-evvntth Conference Neural Information Processing Systems, 2023. URL https: httpp:::://pppeeeieeeeee net forum?
Id=aluuuppa83y
14

---

Published as a conference paper at ICLR 2025
Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Measuring and controlling instruction (in)stability in language model di- alogs. In First Conference Language Modeling. 2024. URL https: hotp:::///peerevie.. net feet/foree forum? 1== 60a 15AtH4e)
Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learn- ing more effective and controllable through latent space steering. In Forty-first International Conference on Machine Learning, 2024. URL https: betts:://ppnrevieeet net/ forum? ?id=
@JTChKgv3a
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan (ed.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. press/ v2 http0//ooogooe22. html
PMIR, 23-29 Jul 2023. URL https: /ttp:://pprceeeeee mgs.ml.........mmmmmmmmmmmm..... mmlr.
Renze Lou, Kai Zhang, and Wenpeng Yin. Large language model instruction following: A survey of progresses and challenges. Computational Linguistics, pp. 1-43, 08 2024. ISSN 0891-2017. doi: 10.1162/coli.a 00523. URL /tLtttss:://doi org.
/ttt:pp:://dl..oo/000.11111 1et.:cl11ccooo coli 00523
Dawn Lu and Nina Panickssery. Investigating bias representations in llama 2 chat via activation steering, 2024. URL http:////aarriv..r//aaa//////2..
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024. URL httphhttpp.:/pperevvieeee net. for///orumm -ot.:/orrmmmdddaaayyyyeeee
Quinn McNemar. Note the sampling of the difference between correlated proportions percentages. Psychometrika, 12(2):153-157, 1947. URL https::/linkkkset wpriigin...pii.............. com/
article/10.1100 11t.11107///02220 1.AA:lee/0..D0077/E0299999
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed representations of words and phrases and their compositionality. Burges, L. Bottou, M. Welling, Z. Ghahramani. and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL httpss:/prooceedings.coo File/ //tpaa/2022222339331333222222222
C.J.
catpsss.ccppppppee files/paper/2013/ pdf
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3470-3487, Dublin, Ireland, may URL
2022. Association for Computational Linguistics. doi: 10. 10.18655//1/2222.allooogg
wwt./ot2..202...ooo wwt.222..cllonnn 244
Neel Nanda and Joseph Bloom. Transformerlens, 2022. URL https://githube com/
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent lincar representations in world mod- els of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, Singapore, dec 2023. Association for Computational Linguistics. doi: 10. wt..1853/w1/0223.10 btwp///0.....acooooopooo-ooo URL https::/accLattho..oo www.hhhwwwoooooooooooooeeoeooeee 20..g/2002..2 2023. http233..ccccooxlppppp 1.2
nostalgebraist.
Interpreting GPT: The logit lens.
Less Wrong 2020.
URL
https: http::://www.eswrronee htt.://wwwoooo.oooooooo////a/aaa
interpreting
15

---

Published as a conference paper at ICLR 2025
Christopher Olah.
What is a linear representation? what is multidimensiionll feature? Trans-
former Circuits Thread, 2024. URL hatt:s//ttaaaeemmmrrrreootttttpp
july-update//
OpenAI. GPT-4 technical report, 2024. UBL/httpes//aaa...rr///////33333
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https: http::///ppeeeeiiieeee net forum? ?id=
TGBKACXEON
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering Ilama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- lional Linguistics (Volume 1: Long Pupers), pp. 15504-15522, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. URL https: aeelllnnhhoogoo org/2024.
acl acl-long. 828
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geom- etry of large language models. In Salakhutdinov, Ruslan and Kolter, Zico and Heller, Kather- ine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp. Felix (ed.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 39643-394999 396666 PMLR, 21-27 Jul 2024. URL htt.rpprrreeee////aaaa22221
Emanuel Parzen. On estimation of a probability density function and mode. The Annals of Math- ematical Statistics, 336-)76 1962. ISSN 00034851. URL http::/ / /www. jstor.
Doc/stable/2237800
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Busc Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019.
neurpps........pp.ee............ cc/paper files/ Paaerr.ppPppp.......
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jack- son Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Ka- mal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tam- era Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac HatfieddDDodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernande7, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discover- ing language model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387-13434 Toronto, Canada, jul 2023. Association for Computational Linguis- tics. doi: 10. 1t.18653/v1/2023 wtt.222....nnnns---.. indings-acl.844
Findings-aci.Bal 847
Plotly Technologies Inc. Collaborative data science, 2015. URL https://ppoo....
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: case study entity tracking. In The Twelfth International
16

---

Published as a conference paper at ICLR 2025
Conference Learning Representations, 2024. Ut//atpss///pennerreeeeeee/e//ee
id-8sKcAWOE2D
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay B Cohen. Spectral editing of activations for large language model alignment. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. URL ForumPi@=pgyccEa577
net/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Lan- guage models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019. URL https:
//insightcivic: s3. us--us-eee eastseestt--- amwzzmaws wttp://s..ooomaaaaaaaaaaa mode pdf
Nate Rahn, Pierluca D'Oro, and Mare G Bellemare. Controlling large language model agents with entropic activation steering. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL
htttps://ppenreviii net wowrmmmdddeeedeeeooo
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticss, pp. 7237-7256, Online, jul 2020. Association for Computational URL
Linguistics. doi: 10. http://///22222.mmma.aaao 08200acl--o mai..-mmin.... 2020.acl-main..66 6477
coroolloyorrr...... org/
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnad Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan. Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022. URL https: // wwtp//eeeeevvveeeeeeeeeeeeeeeeee n Eorum (ew.rrummmddd-Vvbbbooo
Daniel Scalena, Gabriele Sarti, and Malvina Nissim. Multi-property steering of large language models with dynamic activation composition. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 577-603, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.28443/v1/2224 blackboxnlp-1.33 1..0. .34. URL /ttp:/:////aaathhaoa wwwphhoooyyooooooooo.oooooo 2024 www.kkoooono.... 100ipp--.... 34/
Shun Shao, Yftah Ziser, and Shay B. Cohen. Gold doesn't t always glitter: Spectral removal of linear and nonlinear guarded attribute information. In Andreas Vlachos and Isabelle Au- genstein (eds.). Proceedings of the 17th Conference of the European Chapter of the Associ- ation for Computational Linguistics, pp. 1611-1622, Dubrovnik, Croatia, may 2023, Associ- ation for Computational Linguistics. doi: 10.18665. N0.118653v11 www.222.....aaaii....aaaaaa
URL https:
//acl lanatolthhoyy...ooooooooo org/2023.3322... oww.hholloyyor//2222333333333333 eacl ma n..... 118
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dode Scott R Johnston, Shauna M Kravee, Timo- thy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024. URL. https: appeneeview.net/forrnnn....
Asa Cooper Stickland, Alexander Lyzhov, Jacob Pfau, Salsabila Mahdi, and Samuel Bowman. Steering without side effects: Improving post-deployment control of language models. In Neurips Safe Generative AI Workshop 2024, 2024. URL https:://///....... httpp::/opeereev /wwp//veeevvveeeeeeee ew net / forumrrmm?
EfXIZ8P42U
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 566-581,
17

---

Published as a conference paper at ICLR 2025
Dublin, Ireland, may 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022. andingg-acl..8.. URL httph: latpa:://aalanthooooyy. wwwp://wwwooo.ooo............ aon0222siniinnsacc.....
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. Evaluating large language models controlled generation tasks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3155-3168, Singapore, dec 2023. Association for Computational Linguistics. doi: 10. (265:///2223...mmmmmaaii
URL
ittts::/aclantbclooyy 222g/2023.rm emnlp- ma mai..mmii... 190
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter. Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL html
t.s pub/20247
Curt Tigges, Oskar John Hollinsworth, Neel Nanda, and Atticus Geiger. Language models linearly represent sentiment, 2024. URL htttps::/opperreviiee wttp:p/wwrrrr....eeoeoeoooornnna d 7Ya
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
:enttss:/oppree http:///ppeeeiieeeeee net/ wow.mmii--wwwttttraarr
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint ar arXiv:2308. 10248, 2023. httts::::
org/abs 2308 10248
Teun van der Weij. Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours, 2024. URL httpt:tt//aa.ttt. wwwpaxiwvw.ggg..e.. abs
otggabb/2/033....//////// /2403, 05767
Dimitri von Rütte. Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. A language model's guide through latent space, 2024. URL http::::awww....oo//////o/ www.//rrg/abbaaoooooo..o 240...///2222... 112..14333 14433
Haoran Wang and Kai Shu. Trojan activation attack: Red-Teamingg large language models wwwels using acti- vation steering for safety-alignment, 2024. URL
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Es- haan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob An- derson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravse- haj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. HuperrNaturalllssuuutions General- ization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zoritssa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, dec 2022. Associa- tion for Computational Linguistics. doi: 10. 11865//v122022.emmpppao main.3-. URL https:
//aclanthology.ooo emr main.340
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022. URL https::/openreviieet net/
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (cds.), Proceedings of the 9th Python in Science Conference, pp. 56 - 61, 2010. doi: 10.25080/Majora- 0atp://Maaooa9991119220000
L https: http://scppyo.oo org
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. URL www.hhttss... URL.httss://ppereee
net o
18

---

Published as a conference paper at ICLR 2025
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross- task generalization in NLP. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Nat- ural Language Processing, pp. 7163-7189, Online and Punta Cana, Dominican Republic, nov 2021. Association for Computational Linguistics. doi: 10. 10.18653/v12022 2tt.:////22..mmmmmmaa..... 572.. URL
emn.p man..ain... 572
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-hased pre-trained language models. ACM Comput. Surv., 56(3), oct 2023. ISSN 0360-0300. doi: 10.1 0.1144/3617680. URL prttps://ont org 100 1145/ 3617680
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang. Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL htttp::/
Beet/Eoum?idd-Evii
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follloiingg evaluation for large language models, 2023a. URL https:
07911
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. Controlled text generation with natural language instructions. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42602-42613, PMLR, 23-29 Jul 2023b. URL https: p/proceedings.. mlr. www.arppeesssee wwwps//ww0////o//eeeeee
html
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan. Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael Byun, Zifan Wang. Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song. Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering:: A top-down ap- proach AI transparency, 2023. URL /tt.://tppp////axxvoorr///////.. 01405
Vilém Zouhar. Ryanize bib, 2023. URL htw.:t/pp//ttthb....oo/////aaa//
19

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORK
While our work demonstrates the potential of activation steering for improving instruction adher- ence, it has multiple limitations that highlight avenues for future research. Activation steering in- volves several degrees of freedom. including the selection of the model layer to intervene at, de- termining the steering weight, and identifying where in the input output sequence to apply the intervention. While our method systematically selects these parameters based on paired examples that differ by the presence of an instruction, it does not fully explore the space of possible interven- tions, leaving room for optimization. For instance, the quality of instruction representations directly impacts the effectiveness of steering. Representations computed from inputs where the model fails to adhere to instructions may reduce the efficacy of steering. Future work could explore filtering such representations during vector computation to improve results, although this would require additional instruction-follow following accuracy checks, which our current approach avoids. Furthermore, our method applies a fixed steering weight throughout the generation process. Dynamically modu- lating the steering weight during generation (Scalena et al. 2024) could adapt the intervention to the evolving context and improve performance.
Another area for exploration is the effect of the number of examples used to compute steering vec- tors. Previous research indicates that clean concept representations can be obtained with relatively few examples (e.g., 128: Arditi al. 2024h. In our experiments, steering vectors computed from as few as 20 examples proved effective. However, we did not extensively investigate how varying the size of the example set affects the quality of these vectors. Future research could focus on optimizing this balance between computational efficiency and steering performance.
While we consider reasonably large and diverse set of instructions in our experiments, the range of possible real-world user requests is vast and cannot be exhaustively represented. Additionally, our use of perplexity (computed with GPT-2; Radford al. 2019) a proxy for output quality focuses on detecting issues in fluency and coherence but may miss subtler forms of degradation, such as factual inaccuracies or logical inconsistencies. Incorporating more nuanced quality metrics or task-specific evaluations could improve parameter selection.
Finally, an interesting direction for future work stems from our cross- cross-model steering experiments. In our work, we experiment with transferring steering vectors from instruction-tuned to base models of the same family and parameter size. An interesting avenue for future research is exploring the transferability of steering vectors across models with different architectures sizes, potentially by learning transformations that map the latent space of one model another.
B LIST OF INSTRUCTIONS
In Table we provide a list of the instructions and examples used in the subset of the IFEval dataset for our experiments about the output format ( For the "language" instructions, the dataset includes the following languages: German, Urdu, Portuguese, Korean, Marathi, Punjabi, Kannada, Farsi, Swahili, Russian, Hindi, Arabic, Nepali, Telugu, and Gujarati.
C EXTRACTING IFEVAL BASE QUERIES
To obtain base queries without instructions, we use GPT-4o (OpenAl 2024 6 to strip the instructions from the base queries, the IFEval dataset does not explicitly annotate the instructions within the prompts. We use a one-shot prompt a fixed in-context example. The prompt used is reported in Ta- 5 These base queries are then used to generate additional base query- query-instruction combinations, augmenting the original dataset. Further details on the data used for steering vector computation and evaluation provided in Appendix D
20

---

Published as a conference paper at ICLR 2025
Table 4: Format-Related Instructions and Examples. format-related instructions used in periments, along with examples illustrating the specific constraints the model is expected to follow.
Instruction
Example
Comma
You are not allowed use any commas in your response.
Lowerease
Please ensure that your response is in English, and in all lowercase letters
Language
Please respond using only the Kannada language, other language allowed.
JSON Format
Wrap the entire output in JSON format. You can use markdown ticks such as
Quotation
Wrap your entire response with double quotation marks.
Multiple Sections
Write section [base query]. Each section should be explicitly noted Section X.
Your answer must contain exactly bullet point Markdown using the following format: |n* Bullet point one \n Bullet point two. two.)n... \n Bullet point six
Number of Bullet Points
Highlighted Sections
At least 15 sections should be highlighted with markdown such as "highlighted section
Title
Your must have title contained in double angular brackets, such <tiile>>>>
Cupitalize
Make sure your entire response is English, in all capital letters.
Capital Word Frequency
Use words in all capital letters at least 3 times to highlight key points.
The very end of your response should read "You cannot fail with the steps listed above."
End Checker
No other words should follow this phrase.
Answer with exactly one the following phrases: "My answer yes.". "My answer no.". "My answer maybe."
Constrained Response
Table 5: Prompt for Instruction Removal.
Given a question that imposes set of constraints on the answez, make the question simoler by removing the constraints. You will be given the original question and constraints with the constraints removed. Nothing eise should be removed other than the listed constraiats, For example:
from it, and should cutput the simpli questior
<originallquestioon>
am planning trip to Japan, and would like thee to write isinerary for my journey in a Shakespearean style. You are not allowed to use any commas in your response. <loriginalqqustioor
kconstraints>
[punetuation:no.comm
<lponstraints>>
<output>
am planning trip to Japan, and would like thee to write an itinerary for my journey Shakespeazean style.
<loutput>
<original question>
Write a 300+ word summary of the wikipedia page https://en.we http://ia....iiiiia.a.....a wiki/Raymond IIT Count of Tripoli. Do not use any commas and highlight at least 3 sections that has titles in markdown fornat, for example shighlighted section part 1x, shighlighton sect ion part 2t, thighlighted section part 3t. <loriginal question>
Cou.
<const ra raintsa ints>>
[punctuation wtw.ctuuttoonnoooooo.... snttectabe..ormatr tnumber.higtt htghereeeitiiitttte.eeeeteiiieee length.constraints. <lconstrainte>
<output>>
21

---

Published as a conference paper at ICLR 2025
Table 6: Prompt for Instruction Addition.
Given a simple quastion, se want to make the question a bit harder by adding constraints way it can be anawered. You will given original question and a constraint to add to it, and should outsut the harder question with the constraint integrated inta it. Nothing else should be added or remaved from the question.
question. Only constreint should added For example:
<originall.questionn
The opposite of youth is not age, but ...?
</original question>
<constraintss
Clstectable.formtt htw.:::::ttbrtrhhhhhhhheeeee. biee <lconstraintss
<output>
The opposite of youth is age, but ...? Highlight least sections in your answer with markdawn, i.c. iiighlighted section..
<loutput>
Write a 300+ word summary the wikiped'a page https://en... http://nn..kkippia..rrrr viki/Raymand.II1 Sall__oonttee conntooffii Tripoi. Do not any and highlight at least 3 sectiens that haa titles in markdown format, for example +highlighted section pazt 1t, +highlighted section part 2*, thighlighted section part 3+. <lpriginal.qu -uuasttiinn
<constraint> detectable inumber highlighzed sections num highlights's 2} </conatraint>
<output>
D DATA AND VECTOR COMPUTATION DETAILS
Format Instructions. The steering vectors and layer selection for format instructions are computed using a separate set of synthetically generated prompts. generate synthetic dataset containing all base prompts with all instructions from the IFEval dataset, we start with the base prompts as described in Appendix C We then create an augmented version the IFEval datasct by combining each base prompt with every available instruction. This is done by prompting GPT-4o using in- context examples, as shown Table 6 When adding a constraint to a base prompt, we include randomly selected in-context example of the same type of constraint from the single-constraint dataset. These single-constraint examples are generated by prompting GPT-4o to remove all but one instruction from each prompt. For this, we use fixed, manually curated in-context example. This procedure carried for each base query and each format instruction. We then filter out any base query and instruction combinations that appear the evaluation used for steering vector computation and layer selection do not overlap with those in the evaluation prompts. This process yields approximately 450 examples for each instruction, which are used for both the steering vector computation and for validation. We provide examples of data generated using this procedure Table 7
ensure that the base queries
For validation, we use set of 96 examples (8 per instruction), sampled from the synthetically data described above. For format instruction evaluation, we utilize the format-related subset IFEval queries (163 in total), which includes the 13 distinct instructions detailed in Appendix B
Length Instructions. For length instructions, the steering vectors are computed using fixed set of 50 IFEval base queries, obtained as described in Appendix C To this set, append various manually-annotaeed phrasings instructions that request the model to generate concise responses, such "Be concise" or "The answer should be brief."
The evaluation data used in $4 and Appendix consists separate set of 200 base queries. Al- though IFEval contains some -ength-related instructions, it offers limited examples with significant variation in length constraints and expressed different ways (e.g., "I don't want anything longer
22

---

Published as a conference paper at ICLR 2025
Table Examples of Synthetficlll--geeeraeed Data.
Condition
Example
Write a 300+ word summary of the wikipedia page https:// en. mn.iiippedia.org/viii hetii//o/o///w///aa/a/aaa Stl.moonddII..Contt... Tripoll Do use any commas and highlight at least sections that have titles in markdown format, for example "highlighted section part 1*, "highlighted section part 2*, "highlighted section part 3
Original IFEval prompt constraints)
Synthetic sonsle-constraitt punctuation:no.com comma
Write a summary of the wikipedia page https://en. wikipedia.org Sttp:/oonddll..Coolllll _Iw.tCCuuttt... of Tripoll Do not use any commas.
/wiki/Raymond
Write a summary of the wikipedia page https://en. w wikipedia.org wiki/Ray Maii/Raymon_ill.. Ill...Counttoll Tripoll. Highlight at least 3 sections that have titles in markdown format, for example "highlighted section part 1*, "highlighted section part 2 "highlighted section part 3*.
Synthetic single-constnn constratnt detectable_ format: humberrhighhlighiddsscciii
Synthetic single-constraint congth.constrainttuubberr. words
Write a 300+ word summary of the wikipedia page https://cn.wwi wikipedia. orriiaoorr/gaooo
-tl.ndddIl,,Ciiiiii Counttof
Write a summary of wikipedia page https://en. wikipedia.org/ wiki/Raymond ColnnCo of Tripoli.
Synthetic no-constraint base query
Synthetic augmented with detectable format:
Write a 300+ word summary of the wikipedia page https:// en. en.wiiipedia.om wekii/eeii..g/wiiikaayannee wiki/Raymondd III, Count of Tripoli. Your answer should contain exactly 3 bullet points in markdown format. Use * to indicate bullets, like: xyz abe opg
number-bur bullet lists
than 30 words," or "Answer using five paragraphs"). We opt for synthetically generating data to enable consistent evaluation across set of constraints varying over a narrower range to 5 sen- tences).
Word-specific Instructions. For word word-specific instructions, we compute vector representations for inclusion constraints, where the model is instructed to include a specific keyword in the output (e.g., "The output should contain the word {w)" or "Ensure that the word {w) is included in the response"). For each keyword, we use a set of 20 base queries randomly sampled from those gen- crated as described in Appendix C ensuring none of the queries used for vector computation are reused in evaluation. The topics and content of these base queries are often semantically unrelated to the keyword being included. The same procedure is carried out for word-exclusion instructions in Appendix K
For validation, we use GPT-4o to synthctically generate a set of questions similar to the base queries in IFEval. Additionally, the prompt requests the generation of list of words likely to appear in the answer to each question. These question-word pairs (276 in total) are used for grid search validation of word inclusion and exclusion. For evaluation, we IFEval examples containing keyword inclusion and exclusion instructions. Many examples contain multiple keyword constraints in a single query, so we separate these into individual prompts, each requesting the inclusion or exclusion of single keyword. This process yields 86 evaluation prompts for keyword inclusion and 117 for keyword exclusion.
E STEERING LAYER AND WEIGHT SELECTION
When selecting the steering layer and weight, our goal to identify a combination that improves instruction-folllwinn accuracy without compromising the fluency or comprehensiveness of the out- put. While csstuuctionfoolowing accuracy can be quantified through specific checks, evaluating fluency and comprehensiveness is more challenging, as thorough evaluations, such as our GPT-4o- based procedure, are computationally expensive and impractical for validation. To address this, we perplexity computed by smaller model as a proxy (GPT-2: Radford et al. 2019). The rationale (GPT-2:|Raford)
is that very low perplexity values can indicate outputs where steering has significantly degraded quality. For each laygr- combination, compute the fraction of low-perplexity outputs (be-
23

---

Published as a conference paper at ICLR 2025
Table Steering Layers for Format Instructions. Layer indices used for steering across different models and evaluation settings (with and without explicit instructions) for each instruction in the format subset. A dash (*-") ") indicates that no steering performed.
Model Name
Phi-3
Gemma 2 2B IT
Mistral 7B Instr.
Gemma 2 9B IT w/o Instr.
Setting
w/o Instr.
w/ Instr.
w/o Instr.
w/ Instr.
w/o Instr.
w/ Instr.
w/ Instr.
Capital Word Freq.
18
20
21
23
12
24
16
Capitalize
28
22
11
11
28
30
28
Lowercase
18
30
17
Constrained Resp.
JSON Format
6
13
16
10
Multiple Sections
8
9
28
Number Bullet Lists
20
16
5
24
12
16
12
Highlighted Text
26
18
5
Title
No Comma
10
12
14
End Checker
18
Quotation
26
24
11
23
34
12
Table 9: Steering Layers for Language. Layer indices used for steering across different models and evaluation settings (with and without explicit instructions) for each language-related instruction. A dash (*-") indicates that steering is performed. In most cases with explicit text instructions, steering was unnecessary as the models typically followed the instruction and generated responses in the requested language.
Model Name Setting
Phi-3
Gemma 2 2B IT Instr.
Mistral 7B Instr.
Gemma 2 9B IT
w/o Instr.
w/ Instr.
w/o Instr.
w/o Instr.
w/ Instr.
w/o Instr.
Instr.
Language Ar
18
16
15
22
20
20
Language Bg
22
19
18
6
22
Language Bn
18
20
language De
16
16
Language Fa
Language Fi
20
20
Language Gu
14
Language Hi
22
14
22
12
Language It
16
15
18
20
Language Mr
30
14
17
15
24
Language Pa
Language Pt
16
15
16
20
Language Ru
16
15
18
Language Sw
20
24
Language Ta
Language Te
Language Th
Language Ur
28
15
28
Language Vi
20
15
12 20
low a threshold) and exclude configurations where this fraction non-zero Among the remaining configurations, we select the one with the highest validation accuracy. Additionally, we compare the validation accuracy for each layer against bascline without steering to ensure the intervention provides improvement. If no layer improves upon the bascline, steering is not applied at test time. In cases where multiple layers achieve the same highest accuracy, we select the earliest layer. The perplexity threshold provides mechanism for balancing instruction-folloiinn accuracy against the risk quality degradation. A higher threshold results a more conservative steering procedure but may exclude valid parameter combinations that could yield higher accuracy. In experiments, set the threshold to 2.5 for format instructions and 4 for word-specific instructions. This systematic approach ensures effective steering while maintaining output quality.
Layer Selection Results. In Table 8, we provide the layer indices used in our steering procedure for each instruction in the format subset (excluding language) across all models and evaluation settings (with and without explicit text instructions). We observe that for "Constrained Response" and "Ti-
In non-instructionntumn models low perplexity may occur without steering. In these cases exclude layers if the post-steering fraction exceeds the baseline value observed before steering.
24

---

Published as a conference paper at ICLR 2025
Table 10: Prompt for Quality Evaluation Question Generation.
The following is a prompt that is used to evaluate the generations from a large language model.
know how evaluate quality model answers this prompt. Can come with of less questions that can break down the quality to simpler evaluation tasks that we can then ask about the modal answer? Each question should
simple yes, anawer.
Prompt: {[ prompt without instruction }}
list all sub questions in the foll lowionngg formst:
Output:
1: Question:
<question>
2:
Question: <question>
N: Question: <quastion>
Table 11: Prompt for Quality Score Evaluation.
We need evaluate the quality of generations from large language model. You wf be given an inpus prempt, the response from language model and set of questions asseasing the quality of the reaponse. You need to zeview the response against the input promot and provide an answer each question either "Yes', "No" 'Not Applicable' if the question daes not apply to the case along with a reason for your answer.
Prompt: prompt without instruction }}
Response: [[zespense]t
Questions:
cuestiona])
list your answers in the following format:
Output:
1. Question: <question>, Reason: <reason>:
Anawer:
<answer>
2. Question:
<question>. Reason: <reason>: Answer:
<answer>
N. Question: <quastion>. Resson: <reason>:
Answer:
<answer>
tle," virtually no layer improves performance with steering. The "Constrained Response" instruction asks the model respond using only "yes," "no," or "maybe," while the "Title" instruction requires the answer to include title wrapped in angular brackets (c.g.. "<>"). In these cases, the steer- ing vectors sometimes push the model too aggressively toward outputting instruction-related tokens, resulting in poor outputs (e.g., repeating characters like "<<<<"), For language instructions, the selected layers are listed in Table g For length instructions, we intervene layer 12 Phi-3 and layer 16 in Gemma 2 9B. For word-specific instructions,
use Phi-3's layer 24 and 28, and Gemma 2 2B IT's layers 24 and 22 for for inclusion and exclusion, respectively. The validation results used for layer and weight selection selection reported in Appendix K
Candidate Weights. We base the selection of the steering weight on the computation in Eq. For format instructions, the weight from Eq. (2) is used directly (i.c., for each layer, there procedure above include a single layer- combination). For length and format instructions, the value computed in Eq. (2 is averaged across different inputs and is used as reference to determine a range of suitable steering weights. For length instructions, we experiment with multiple weight values, notice that
highlighting the continuous nature of the constraint. For word inclusion instructions, in some cases (particularly with Gemma 2B), the weight is too low to impact the model's output. This could be because the steering vector averaged across inputs regardless of whether the model satisfies the constraint. This issue tends to be more pronounced for word-specific constraints (than, e.g., for format instructions) since the strength of the instruction signal can vary greatly depending on whether the word is related to the base query (e.g., including word related to the query is easier than including an unrelated one). To address this, in the selection procedure described above, we consider small set of weights based on the value ē (i.e., the value computed as in Eq. ( averaged across different inputs). For Phi-3 ē z 52 at layer 28 and = 42 at layer 26, and we use the set {40, 60, 80, 100} as grid-search value across the layers {24, 26, 28}. For Gemma 2 2B IT, ē 55 at layer 22 and ē xs 74 at layer 24, and we use the set (60, 80, 100, 120) as grid grircs value across the layers (22, 24).
25

---

Published as a conference paper at ICLR 2025
F RESPONSE QUALITY EVALUATION
F.1 EVALUATION PROCEDURE
To evaluate the quality of the model generated responses, we first use the base queries outlined in Appendix C and prompt GPT-4o to produce 5 questions about the query to evaluate any model generated answer to the query (prompt reported in Table For each model generated response in our evaluation, we then prompt GPT-40 provide binary "yes/no" answer to each question and "Not Applicable" if the question is not applicable to the query, along with a reason (prompt in Table T.. An aggregate quality score is computed the average % of questions that is scored as "yes". We run the experiment three times for each model and report the quality score and standard error to account for any non-dcterminism from the GPT-4o based evaluator.
F.2 EXAMPLES OF QUALITY SCORE CHANGES
While our evaluation procedure assesses how well the model's answers address the base queries, the method used to select steering locations and weights is based on IFeval's cntructiio-following aceuracy metric. This metric consists of checks that verify specific aspects of the model's output related to instructions but sometimes these checks may be incomplete. For example, the default met- ric in IFEval for section-related formatting checks that the model includes sections in the generation but it does not check whether there is content in the section or the quality of the content if present. Therefore, in such cases this sometimes results in the selection of layer indices and steering weights that over-steer the model, leading to outputs that satisfy the instruction but have poor quality (e.g., repetitive token sequences). An example of this can be seen in the first row of Table These issues are more prevalent in the Phi and Mistral models, compared the Gemma models (noticeable from the smaller deltas for these models in Figure 4). This discrepancy may be due to Gemma's 's longer training and a better size/performance tradeoff, potentially making them more robust activation steering.
Another source of quality drops can be minor factual inconsistencies in the model's 's responses. Steer- ing may sometimes cause slight deviations from factual accuracy, which our quality score metric captures by incorporating GPT--4generated questions about correctness. For example, in the sec- ond row of Table [2] the model inaccurately states that Kentucky, Missouri, and Mississippi were part of the original 13 colonies. While these inconsistencies can affect quality scores, they are not widespread and represent general challenge for activation steering. Addressing factual consistency is difficult when selecting the steering location and weight, but remains an important aspect to con- sider in future work. Finally, quality score drops also result from small and arbitrary differences in the output, as shown in the third row of Table [22 in which the output looks reasonably good, but the model fails to follow the correct rhyme scheme for a limerick (AABBA), and the quality score drops by 0.6.
In conclusion, it is well-known that direct intervention compromise the quality of the output. In our case, we observe relatively few instances where this happens, but improvements could be made. Future work could mitigate these issues by conducting a more exhaustive grid search over layers and steering weights, or by better modulating the steering intensity (Scalena al. 2024 Stickland et al. 2024).
model's 's activations can sometimes
F.3 ADDITIONAL RESULTS
We apply the response quality evaluation procedure to outputs generated by steering the model for length and word-specific instructions, following the same approach used for format constraints in } As in the previous section, we analyze the outputs produced by Phi-3, reporting the score changes due to steering both with and without explicit text instructions, as well as the quality score differences when text instructions are added without steering.
Length Instructions. Figure Dh presents the results for length constraints. We analyze subset of 100 outputs from the generation process used for Figure 5p in $ (which evaluated 200 examples). The model is steered with weight of c 20 constraints requesting the model to answer using at most n sentences (n E 1,...,5). The two leftmost columns in Figure Da show the changes in quality score when steering the model on inputs
set of base prompts, applying different length
26

---

Published as a conference paper at ICLR 2025
Table 12: Examples of Changes in Quality Score. In the first case, the model steered for the "Uppercase" instruction, which causes the output to become repetitive, resulting in a significant quality score drop from 1 to 0.4. In the second case, the model inaccurately states that Kentucky, Missouri, and Mississippi were part of the 13 colonies, leading to drop in quality score from 0.8 to the third case, while the output looks fairly good, the model fails to follow the correct rhyme scheme for limerick (AABBA), and the quality score drops from 0.8 to 0.2. These examples are taken from the top 10 outputs generated by Phi-3 with the largest drop in quality score.
Query
Output w/o Steering
Output w/ Steering
The Legend of the Sword and the Fairy movie in which Wan Wan is a villain. Write a story about Wan Wan's 's character in list format.
1. Introduction to Wan Wan Wan Wan is cunning and am- bitious antagonist in the movie "The Legend of the Sword and the Fairy."( She is skilled sor- ceress with deep-seated desire for power and control. \n Early Life: - Wan Wan was born into noble family, but her parents were killed in tragic accident when she was young. \n She was raised by her grandmother, who taught her the [...]
[...] 14. WAN WAN'S RE-
DEMPTION IS MIRACLE,
AS HE AND THE SWORD
AND THE FAIRY'S GIRL FIND
HARMONY AND LOVE, 15, WAN WAN'S REDEMP- TION IS A TEACHING, AS HE LEARNS THE VALUE OF LOVE. HONOR, AND SELFLESSNESS. 16. WAN WAN'S REDEMPTION IS MIRACLE, AS HE AND THE SWORD AND THE FAIRY'S GIRL FIND HARMONY AND
LOVEA 17. WAN WAN'S REDEMPTION IS TEACH- ING. AS [...]
What is the difference hetween the 13 colonies and the other British colonies in North Amer- ica?
The 13 colonies, also known as the Thirteen Colonies, were a group of British colonies on the Atlantic coast of North America founded the 17th and 18th cen- turies that declared independence in 1776 and formed the United States. The other British colonies in North America, often referred to as the Middle and Southern colonies, were established later and had different characteristics. An 1. Location: The 13 colonies were located along the Atlantic [...] \n 2. Economy: The 13 colonies [...] \n 3. Settlement
The 13 colonies, also known as the Thirteen Colonies, were a group [..J] These colonies were: L. Virginia \2. Massachusetts 33. New Hampshire \n 4. Rhode Island {5. Connecticut (n 6. New
York), in 7. New Jersey',n 8. Penn- sylvania'n 9. Maryland 10. Delaware n 11. \n *Kentucky (not originally a British colony, but later admitted as a (t 12. "Missouri* (not originally a British colony. but later admitted as a state) in 13. *Mississippi* (not originally a British colony, but later admitted as state) [...]
Patterns: [...]
Write a limerick about writing limerick.
Whose verses were both sharp and clean.\p \n He penned limer- ick, (n With wit so quick, \n And the rhythm was quite the ma- chine.
There once was a poct named Kent, in Whose limericks were quite the event. \n He wrote one with glee,'n see, (n That was about writing a limerick, test.
limerick. you
with and without length instructions. These values are averaged over the constraints, as the steering procedure applied is the same way for all 5 constraints (adding the conciseness vector with weight = 20). A negative delta is expected, as steering the model to produce shorter responses generally reduces the comprehensiveness of the answer. This pattern is confirmed by the rightmost columns in Figure Gh, which represent the difference in quality score when length instructions are explicitly provided in the input text. The different colors correspond to the number of sentences (n), and as expected, shorter length constraints lead to larger drops in quality score. Although the effect of steering is not directly comparable to any specific length constraint (since steering applies an additional reduction in length without a one-to-one mapping between steering weight and sentence count), we observe that the changes in quality scores due to steering remain within a reasonable range when compared to those induced by text instructions.
27

---

Published as a conference paper at ICLR 2025
Changes in Response Quality: Length
(b) Word Inclusion
(c) Word Exclusion
Quel. Score Delta
Avg. Qual. Score Delta
Length
0.05
0.05
-0.05
Constraint
Qual, Score
Avg.
-0.05
-0.1
-0.1
-0.15
-0.15
-0.15
Steering
Stoering
No Steening
Staering
Strertig
No Staering
w/o frestr
w/ Insor.
wro Inder.
w// Initr.
w/ wio lnstr.
Stoor w/o Ins. Ins.
Steering vs, w/o Instr:
sale tnstr.
Figure 9: Changes in the Response Quality Scores. Quality score deltas for (a) length instructions, (b) word inclusion, and (c) word exclusion, across three conditions: steering without instructions, steering with instructions, and no steering with vs. without instructions.
(a) Steering w/o Instr.
(b) Steering w/ Instr.
(c) No Steering w/ vs. w/o Instr.
Mean: 0.03
60
Mean: -0.00
Moan: 00.00
Frequency (9%)
50
Frequency
40
40
Frequency
40
30
20
20
20
10
10
Quality Score Delta
Quality Score Delta
Quality Score Delta
Figure 10: Distribution of Quality Score Changes for Word Inclusion. Histograms of qual- ity score deltas under three conditions: (a) steering without explicit instructions, (b) steering with explicit instructions, and (c) no steering, comparing outputs with and without instructions. The dis- tributions show minimal impact quality scores across all settings, with mean values close to zero, with most shifts falling within the [-0.25, 0.25) range.
Word-specific Instructions. In Figures 96 and 99. report the changes in quality scores for instructions that request the inclusion or exclusion of a specific word in the answer.
The decrease in quality is minimal across all scenarios, the largest drop observed when steering for word inclusion without explicit instructions (around -0.03). We further investigate this by examin- ing the empirical distributions of the score deltas for word inclusion instructions (Figure I0P. The distributions appear centered around 0, with a few outliers showing deviations larger than 0.4. We perform paired wide t-test assess whether the means of these three distributions are signif- icantly different. The p-values for the three tests (one for each pair) are all greater than 0.05 (0.25, 0.98, and 0.26). While the lack significance does not prove that the distributions are identical, we interpret this as evidence of the minimal impact our steering procedure has on the model's 's ability to address the base queries.
Breakdown by Instruction. Figure 11 presents the quality score deltas for format instruc- tions, broken down by individual instructions across all four models (Phi-3, Gemma 2 2B IT, Mistral 7B IT, and Gemma 2 9B IT). The three settings compared are steering without explicit instructions, steering with explicit instructions, and no steering but comparing outputs with and without explicit instructions.
Format:
F.4 QUALITY SCORE/ACCURACE TRADE-OFF
We investigate how the steering weight mediates the trade-off between instruction-following ac- curacy and output quality degradation for word exclusion constraints. Figure [2] 12sshows the quality score and accuracy obtained by steering Phi-3 with varying negative values of (as we subtract the word-inelusion vector, as explained in $5 and Appendix]] D). In both settings (with and without explicit text instructions), larger absolute steering weights lead to higher cestuuctionfollloiinn curacy. This improvement comes at the cost of gradual decrease in quality score, indicating a of between strict adherence constraints and maintaining output comprehensivennss. The
28

---

Published as a conference paper at ICLR 2025
Score Delta
(a)
Qual. Scores of Phi-3 Format Instr.
Avg. Qual. Score Delte
(b) Qual. Scores of Gemma 2 28 IT Format Instr.
0.2
-0.6
Copital Word Freq.
Constr ainad Resp.
End Checker
English Capital
Lowercase
ISON Format
Multple Secttors
No Comma
Number Bullst Lats
Highlight Taxt
Quotation
Tible
Capital Word
Corotralnde alroad Rosp
End Checker
English Capital
Lowerrcaee
JISM Format
Multiple Sections
No Comma
Number Bullet Lads
Highlight Toxt
Quotation
Title
Setting
Steering w/o
Setting
Steering Instr.
Steering w/ Instr.
Steering w/ Instr.
Steering w/ vs.
Steering w/o
(c) Qual. Scores of Mistral 7B IT on Format Instr.
Avg Qual Score Delta
(d) Qual. Scores of Gemma 2 9B IT on Format Instr.
-0.4
-0.4
Avg.
Capital Word Frea.
Constrained Relp.
Endd Checker
Englitsh Capital
Lowercase
SSDN rormat
Hultiple Sections
No Comma
Number Gutlet Lats
Highlight Taat
Quolatton
Title
Capiral Word freq.
End Checker
English Cantal
Loweraate
JSDN Format
Multrppl Sections
Hightight Text Number Bullet Litre
Quatation
Tide
mnstrainee Resp.
Comma
Setting
Stooring w/o Inatr.
Setting
Stearing w/o Instr.
Steering
Steering w/ Instr.
No Steering w/ w/o
No Steering w/ w/o Instr.
Figure 11: Quality score deltas for format instructions across all models and settings. (a) Phi-3, (b) Gemma 2B (c) Mistral 7B IT. (d) Gemma 9B IT. Bars represent the average quality score delta under three conditions: steering without explicit instructions (purple), steering with explicit instructions (orange), and no steering but comparing outputs with and without explicit instructions (light blue).
Quality Score Accuracy: w/o Instr.
(b) Quality Score Accuracy: w/ Instr.
0.9
0.9
Value
0.8
Value
0.8
-150
-80
Steering Weight
Steering Weight
Quality Accuracy
Quality Accuracy
Figure 12: Trade-off Between Quality Score and Accuracy Word Exclusion. (a) Results for steering without explicit text instructions, showing increasing accuracy at the cost of a gradual decrease quality. (b) Results for steering with explicit instructions, demonstrating similar trend with higher baseline accuracy.
smooth transitions in both metrics suggest that tuning offers ime-ggained control over this balance, highlighting the flexibility of activation steering.
G IMPLEMENTATION AND EXPERIMENTAL DETAILS
Tokenization and Decoding. Steering vectors are computed at the last token of the input. For instruction-tuned models, this typically corresponds to the < | ilassistante stant |> token, which marks the transition from user input to the start of the model's generation For non-instruction-tun models, follow previous work (Kojima al. 2022 Yang et al. 2024 and structure the prompt as "Q: (problem] InA." InA::" Model outputs are decoded greedily, with maximum generation length
29

---

Published as a conference paper at ICLR 2025
(a) "Lowercase" Instruction
(b) "JSON" Instruction
(c)
"German Language" Instr.
Similariyy
Cosine Similerity
Cosine Similarity
0.9
0.8
0.8
0.8
0.7
0.7
0.7
10
20
30
10
20
30
10
20
20
Layer
Layer
Layer
(1) Different Query Same Instr.
Different Query Same
(1) Differant Query Instr.
(2) Same Query w/ and w/o Instr.
(2) Same Query w/ and w/o Instr.
(2) Same Query w/ and w/o Instr.
(3) Different Quertes
(3) Different Queries
(3) Different quertes
Figure 13: Residual stream similarity across layers. Cosine similarity of residual stream activa-
tions across layers for three instructions: (a) "Lowercase," (b) "JSON," and (c) "German Language." Comparisons are made between: (1) different queries with the same instruction (green). (2) same query with and without the instruction (blue), and (3) different queries without instruction (red). Results are shown for Phi-3 across all layers.
of 2048 tokens for format and length experiments, and 1024 tokens for keyword experiments. For efficiency, validation runs use a reduced maximum length of 384 tokens.
Tools and Libraries. For length-related experiments, word counts are measured as the number of space-separated sequences of characters and sentence counts are determined using NLTK (Bird) & Loper, 2004). To assess whether the score improvement from steering is significantly differ- ent from standard inference, we carry out McNemar's test (McNemar) 1947). In particular, we use the exact version of the test, which uses the binomial distribution and is more conservative. The error bars reported Figures 4 and 9 represent the standard error of the quality score com- puted over three different runs of GPT-4o. The smoothed empirical distributions shown in Figures Figure 5b. Figure 5p. Figure Sp. Figure [7], and Figure [7p are obtained using kernel density es- timation (Parzen| 1962). Our experiments were carried out using PyTorch (Paszke et al., 2019) and the Trans oormersssee
library (Nanda & Bloom, 2022). We performed our data anal- ysis using NumPy (Harris et al. 2020) and Pandas (Wes McKinney 2010). Our figures were made using P Plotly (Plotly Technologies Inc. 2015). The paper's bibliography was curated using Ryani ze ze-bib (Zouhar 2023).
H ADDITIONAL RESULTS: REPRESENTATIONS
Similarity of Activations. Figure 13 shows the cosine similarity of residual stream activations in Phi-3 across different input sets. As described in $3 we compare the similarity of representations between: (1) pairs of inputs sharing the same base query, with and one without the instruction; (2) inputs with different base queries but the same instruction; and (3) inputs with different base queries and instruction. The similarity scores reported across Phi-3's layers for three instruc- tions: "lowercase" (no uppercase characters, Figure 134). "JSON" (response formatted as JSON, Figure 13p), and "German Language" (Figure I3d).
Geometry of Steering Vectors. We perform a t-SNE dimensionality reduction on per-example in- struction vectors, calculated as the difference in activations between inputs with and without explicit instructions. at layer 20 of Phi-3. The visualization for format instructions shows varying levels of cluster separation (Figure 144). Specifically, instructions like "No Comma." "Lowercase," "JSON Format," "Title," and "Quotation" form distinct and well-separated clusters, indicating the model's ability to encode these constraints clearly. Instructions such as "Capitalize" and "Capital Word Frequency" cluster closely together, reflecting their similar semantic nature. In contrast, vectors for instructions like "End Checker" and "Constrained Response" are more dispersed, with no clear clus- tering, suggesting these instructions are less distinctly represented in the model's activation space.
Figure 14p presents a t-SNE dimensionality reduction of vectors computed for length instructions at layer 12 of Phi-3. Specifically, we include vectors derived from sentence-specifie instructions (e.g., "answer using n sentences," n E 1,....5) 5)) as well as more general "conciseness" and "verbosity" instructions, described in $4 clear linear trend emerges, with vectors for longer output in-
30

---

Published as a conference paper at ICLR 2025
(a) t-SNE of Format Instruction Vectors
(b) t-SNE of Length Instruction Vectors
Instruction
1.5
Instruction
Comma
Highight Text
t-SME companent
Captalin
SME cam compan
0.5
End Chocker
no4
Capial Word Free.
Constreined
Lowsrtaaee
Number Bullet Lists
Dulputs
nm1
JSON Format
Multtple Sections
1-SNE component
1-5NE
Figure 14: t-SNE visualization of per-example instruction vectors. (a) Format instruction vectors at layer 20 of Phi-3 for format instructions. Distinct clusters emerge for certain instructions (c.g., "No Comma," "Lowercase,"" "JSON Format"), while others (e.g., "End Checker," "Constrained Re- "JSOON
sponse") show less clear separation. (b) Vectors derived from length instructions at layer 12 of Phi-3. Vectors corresponding concise outputs cluster on one end, while those for larger sentence counts (n > 2) cluster on the opposite end.
(a) Accuracy of Phi-3 on Format Instr.
(b) Accuracy Gemma 2 2B on Format Instr.
Accuracy
Accuracy
Quntation
End Checkes
Language
Tible
Highlight Test
Humber Bultat Llats
Multinl Sectiona
JSON Format
Constenined Resporree
Lowercane
English Captal
Capital Word Freq.
Quatation
End Checker
No Comma
Langunge
Title
Highight Text
Number Buller Uste
Hultiple Sections
ISDN format
Constraimed Reggonee
Lowarcase
English Capttll
Capital Word Freq.
Comma
Setting
Instr.
w/o Instr. Steering
Setting
w/o Irstr.
w/o Steering
w/ Instr.
w/ Instr. Steering
w/ Instr. Steering
(c) Accuracy of Mistral IT Format Instr.
(d) Accuracy Gemma 2 9B IT Format Instr.
Accuracy
Accuracy
Quatation
End Checeer
No comma
Language
Title
Highight Test
Numban Bullet Litts
Momple Sections
YSOW Format
Loweraaee Comatrainve ad Resporoe
Engiinh Capital
Capital Word Freq.
Quatatton
End Checker
No Comma
Langungo
Hugniges Teet
Number Bullet Uists
Muttppie Sections
ISDN Format
Consrainne anndtaaded Responua
Lowercase
Engiish Capital
Capibal Word Freq.
Setting
w/o Instr.
w/o Enstr. Steering
Setting
w/o Instr.
w/o Instr. + Steering
w/ Instr.
Instrr Steering
w/ Enstr. Stoering
Figure 15: Accuracy Breakdown by Format Instruction Across Models and Settings. (a) Phi-3, (b) Gemma 2 2B IT, (c) Mistral 7B IT, and (d) Gemma 2 9B IT. Bars represent accuracy for format instructions under four settings: explicit instruction (blue), steering without explicit instruction (red), explicit instruction without steering (green), and explicit instruction with steering (purple).
structions clustering at end of the plot and those for concise outputs clustering the opposite end.
I ADDITIONAL RESULTS: FORMAT INSTRUCTIONS
Validation Results. Figure 16 provides examples of the results from the layer selection procedure. As detailed in Appendix E evaluate instruction-following accuracy and the fraction of low- perplexity outputs after steering at different layers. Panels (a) and (b) show examples obtained with Phi-3 in the no-instruction setting. In the first case ("Multiple Sections" instruction), the perplexity
31

---

Published as a conference paper at ICLR 2025
(a) Accuracy vs. Law Perpl.: Multiple Sect.
(b) Accuracy vs. Low Perpl.; Lowercase
(c) Accuracy vs. Low Perpl.: ISON Format
Value
Value
Value
Layer
Layer
Layer
Accuracy
Accuracy
ACDurSCY
Tractin of Law parslesity Cusputs
Praction of Low perp perpeeiity Outputs
Fraction Low parplesity Curputs
Figure 16: Results from the layer selection procedure. b) Examples from Phi-3 in the no- instruction setting: (a) for "Multiple Sections," the perplexity check prevents selecting a layer with degraded output quality; (b) for "Lowercase," accuracy improves without quality loss. (c) For Gemma 2 2B IT with explicit "JSON Format" instructions, later layers lead to over-steering, with a degradation in both accuracy and quality.
(a) Length vs. Steering Weights
(b) Acouracy
Length Constraint
(c) Length: Pre- vs. Post-Steering
Probability Density
Probabiitty Density
0.8
Length constraint:
Accuracy
0.8
0.6
0.005
200
100
600
0.5
Length (# of words)
1
2
10
Weight
40
8 of Sentences
Length
Std. Inference
Steering
Std. Inference
Stoaring
Figure 17: Additional Results for Length Instructions. (a) The distribution of output lengths when steering Phi-3 using vector for verbosity, with varying steering weights. (b) Steering significantly improves accuracy for sentence-specific length constraints (n E 1,...,5) in 4 of 5 cases. (c) For 4, steering shifts the output length distribution closer the target.
check prevents the selection of a layer that would otherwise lead to a decrease in output quality, as indicated by a sharp rise in the fraction of low-perplexity outputs. Panel (c), which corresponds to the "JSON Format" instruction applied to Gemma 2 2B IT with explicit input instructions, illustrates scenario where accuracy and output quality are correlated: steering at later layers degrades both quality and accuracy.
Accuracy Breakdown by Instruction. Figure 15 provides breakdown of instruction-followingg accuracy across specific format instructions for all models and evaluation settings: explicit in- struction, steering without explicit instruction, explicit instruction without steering, and explicit instruction with steering. Across all models, steering improves accuracy for many instructions in absence of explicit text instructions (blue vs. red bars). Notably, casing-related instructions ("Lowercase" and "English Capital") benefit strongly from steering, with Phi-3 and Gemma 2 2B IT achieving higher accuracy with steering alone than with explicit instructions (red vs. green bars). Certain instructions like "Title" and "Constrained Response" show near-perfect adherence with plicit instructions alone, leaving little room for improvement through steering.
J ADDITIONAL RESULTS: LENGTH INSTRUCTIONS
Steering for Longer Outputs. In Figure 17a, we show the distribution of output lengths when the model steered using a verbosity vector. This vector computed from inputs containing instruc- tions to generate verbose responses (e.g.. "Provide long answer"). Similar to procedure for the conciseness vector in $4 we apply different steering weights to modulate verbosity. Using the same 50 base prompts in Figure the results show that higher steering weights result in longer outputs, demonstrating that activation steering can effectively control output length by adjusting a single scalar value.
32

---

Published as conference paper at ICLR 2025
Fract. Low-perpl. Outputs
Validation Acc. Low Perplexity: Phi-3 Inclusion
Fract. of Low-perpl Outputs
Validation Acc. vs Low Perplexity: Phi-3 on Exclusion
0.06
0.04
0.02
L25,
C4-00
L28, ca-100
0.8
0.85
0.9
0.95
0.3
0.4
0.5
0.6
0.7
Accuracy
Accuracy
Run with Max Accuracy with Low-Perplexiy Outouts
Run with Max Accuracy with Low-Perplexty Outputs
Outputs
Validation Acc.
Perpl.: on Inclusion
Frad: of Low-perpl. Outputs
Validation
Low Perpl.: Gemma Exclusion
0.3
L24,
0.1
L22, co-120
Accuracy
Accuracy
Volidation Runs
Valication Runs
Perploxity
Parplexity
Figure 18: Steering Layer and Weight Selection for Word-specific Instructions. Each panel shows instruction-followii accuracy (x-axis) vs. fraction of low-perplexity outputs (y-axis). Blue dots represent validation runs; red markers indicate the selected configuration with maximum accu- racy and zero low-perplexity outputs. Results are shown for Phi-3 and Gemma 2 2B IT.
Table 13: Examples of Tokens Promoted
Froct, of Low parpl. Outputs
Validation
Perplexity: Phi-3 Exclusion
by Word-exclusion Vectors. Projecting word- exclusion vectors onto the vocabulary results in cases where the excluded word is promoted. This suggests that word-exclusion vectors may not effec- tively steer the model toward avoiding these words. The vectors computed at layer 30 of Phi-3.
0.15
0.05
0.3
0.4
0.5
0.6
0.7
Accuracy
Word
Tokens
Subtraction Inclusion Vedtor
books
bocks, Books, books, book, engl Congress, parlement, _eegis European, europé, Europe, Europe urban, urban, .Urbaa, _urb, .ursl chrom, chrom, engl, genon, 2ob water, Nater, water, _aqua _ishh Fish, fish, _aqu, marine yooth,, Youth, ,hhidd _Children _instead, engl, _invece, _Instead
vongress
Europe
Figure 19: Validation Accuracy vs. Per- plexity Trade-off for Word Exclusion. Subtracting word-inclusion vectors (green) outperforms adding word-exclusion vectors (red), showing the limitations of the latter.
urbunization
chromosomes
water vapor
fish
adult
hypothesis
Steering for Exact Length. We experiment with sentence-specific length instructions to evaluate the effectiveness of steering improving adherence explicit length constraints. Specifically, we compute steering vectors for instructions such as "Answer using sentences" (n E 1,..., 5) and these experiments,
apply them to steer the model toward satisfying the corresponding constraints. we use Phi-3 and the same combination of steering layer and weight as in $4 (layer 12, 20). The results in Figure I7p show that steering significantly improves the model's adherence to the specified lengths when explicit instructions are present in the input. Notably, steering produces statistically significant improvements steering the distribution of output lengths for specific constraint (n 4 sentences). Steering shifts the distribution closer to the target length, with a sharper peak at the desired value, while still maintaining some natural variability.
4 out of the cases. Figure 17c illustrates the effect of
33

---

Published as a conference paper at ICLR 2025
K ADDITIONAL RESULTS: WORD-SPECIFIC INSTRUCTIONS
Validation Results. As described in Appendix E. to determine the optimal steering layer and weight parameter for each model-setting combination, we validate our approach by jointly considering instruction-follooing accuracy and the fraction of low-perplexity outputs. The x-axis in each panel of Figure I8 represents the instruction-folloinng aceuracy, while the y-axis shows the fraction of low-perplexity outputs, which we aim to minimize. For each model and setting, we select the layer and weight combination that achieves the highest accuracy, provided the fraction of low-perplexity outputs is zero. Red markers indicate the chosen configurations.
Steering using Word-exclusion Vectors. Following our approach for word inclusion, we compute instruction vectors for word exclusion. However, projecting these exclusion vectors onto the model's vocabulary via the unembedding matrix often results in high logit values for tokens corresponding to the word that should be excluded. Examples of this phenomenon this effect does not occur in all cases (e.g., for words like "adult" and "hypothesis") ") the presence of positive signal for the excluded word in the logit space suggests that these vectors may coun- teract the intended steering effect. To confirm this, we compare the performance of steering using word-exclusion vectors versus subtracting word-inclusion vectors for the in Figure 19| subtracting word-inclusion vectors consistently yields higher accuracy and achieves better soseccy-o-oo--ppplleee trade-off than adding word-exclusion vectors. These results indi- cate that subtracting inclusion vectors is more effective and reliable method for steering the model avoid specific words.
shown in Table 13 While
words. As shown
34


Reviews:
[{"review_id": "hWYg6ZYXIj", "reviewer": "Reviewer_NNrP", "summary": "This paper proposes a novel method to improve the instruction-following capabilities of language models using activation steering. Experiments on four different language models demonstrated the effectiveness of activation steering for three different tasks.", "questions": "1. line 159, \"...perform a small grid search over neighboring values on a held-out set of examples to fine-tune the steering effect\". How the examples are chosen? How many examples and what's the granularity of the grid? Does it need to be done for every layer?\n2. Figure 4 is a bit confusing. Maybe it's better to mention how the delta is computed (e.g. after steering - original, w/ inst - w/o inst), instead of using \"w/ vs. w/o\", or simply \"steering w/\". Is this quality score a good metric? It's almost opposite with the results of accuracy.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear presentation and writing.\n2. The proposed method is simple and effective.\n3. The experiments and analysis are comprehensive and solid.", "weaknesses": "1. The three instruction-following tasks are related to format, length, and word inclusion/exclusion, which are not broad enough for general instruction following cases.\n2. No limitation or discussion section in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZCbVwpjDp", "reviewer": "Reviewer_R96q", "summary": "The paper \"Improving Instruction-Following in Language Models Through Activation Steering\" proposes using activation steering to enhance the instruction-following abilities of language models. By deriving instruction-specific vectors based on activation differences between inputs with and without instructions, the method adjusts model behavior during inference to meet specific constraints such as output format, length, and keyword control.", "questions": "The need for unique vectors per instruction could impact scalability, particularly for highly variable, user-specific instructions. Clarifying how the approach handles diverse variable instructions would add practical insights.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel approach to activation steering through instruction-specific vector representations, allowing dynamic control over model behavior without retraining. \n\nThe paper covers multiple constraint types (format, length, word-specific) across several models, highlighting the robustness and adaptability of the approach. The cross-model transferability experiments are valuable, showing that steering vectors from instruction-tuned models improve base models. \n\nThe paper is well-structured and clearly explains each step.\n\nSignificance: activation steering offers a scalable solution for fine-grained control over LLM outputs. The demonstrated cross-model transferability suggests a cost-effective way to get instruction-following improvements.\n\nIn sum, this paper presents an original, well-executed, and practical contribution, for various real-world applications.", "weaknesses": "Cross-model transferability is a promising aspect of this work, yet the analysis here could benefit from more quantitative depth. Specifically, it would be useful to test how much performance degrades when steering vectors from instruction-tuned models are applied to base models of different architectures or parameter sizes. \n\nThe paper notes minor drops in response quality when adhering to certain constraints, particularly in the length and word-specific steering tasks. While the authors discuss this, it would be beneficial to implement strategies to mitigate these effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVaBgHOGrW", "reviewer": "Reviewer_B97K", "summary": "This paper applies an activation steering method to improve the constraint following of LLMs such as response length and format. It also shows that multiple steering can be applied simultaneously and some steering generalizes from the instruct model to its base version.", "questions": "- Maintaining quality is important for steering methods, but only one of the experiments shows the quality evaluations. I saw some results in the appendix, but I think it needs to be discussed in the main text. \n- In the length experiment, c=20 steering seems to help with different length constraints. I want to know if this steering makes the response generally shorter, which then happens to increase accuracy, or it makes it adhere to the specific length instruction more strongly? Because of the fixed c=20 value, I feel like it is likely the former.\n- The plots are too small and sometimes use similar colors (dark blue vs light blue), which makes them hard to see. I think the writing can be made more concise to make more room for the plots.\n- How are the base models following instruction? Is few-shot prompting used? \n- Seems to be missing this related work that also does “capitilzation” https://arxiv.org/pdf/2311.06668", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The steerability of LLMs, which this is tackling, is an important research area that has real-world implications.\n- There are interesting findings in the paper, such as combining steering vectors and generalization to the base model.\n- The paper is well written and easy to understand. The experimental setups look solid and diverse.", "weaknesses": "- Novelty: the method of steering by activation itself is not novel and has been used in other papers. The authors claim the application is different, but previous works on writing style and topic changing are related to response format and mentioning a word. Some aspects of the method (combining steering vectors) seems novel, but maybe the author can clarify exactly which parts are novel.\n- Motivation: From reading the paper, the motivation was not that very clear. Why is this method necessary? Explicit instruction in words seems to work better and simpler, while the proposed method can lead to nonsensical responses. What is the main motivation of using this method instead? \n- The proposed method doesn’t seem very general and may require adjusting for each instruction. For example, the method led to an opposite effect in the word exclusion task. In the length task, it is not clear how the exact length is translated into the vector scaling parameter.\n- The dev set is using the same base queries as the evaluation? This makes it hard to judge how the method will generalize to unseen queries.\n- It seems some task have extremely low number of samples. For example, there are only 12 sample for the format task, is that correct? \n- There are some typos around L175 where commas are placed incorrectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OEuOp6Xgdx", "reviewer": "Reviewer_XmmQ", "summary": "This paper develops a way to obtain steering vectors from  instruction prompts. Focusing on output modifiers such as “answer in uppercase,” they develop a way to obtain steering vectors by contrasting instructions with and without the modifier and averaging representation vectors from many paired samples, then they propose a steering approach that introduces scaling to the mean target magnitude.  They gather 12 format modifiers and 19 language modifiers and test their method on a data set of instructions on four language models ranging from about 3b to 9b parameters, sweeping over layers.  They test the ability to use the steering alone to add modifiers, and to use it to strengthen a modifier in a prompt. They also test vector negation by reversing word-inclusion modifiers to become word-exclusion, and they test composition of modifier vectors.  They also find that vectors from instruction-tuned models work very well on original models in some cases.", "questions": "* Can the the full grid of performance measurements of every modifier task (including both format and language) in all four cases (bare prompt, prompt+modifier, prompt+steer, prompt+modifier+steer) across models be included, perhaps in appendix?\n* Similarly, quality degradation is broken out for some settings but not others. Can quality degradation measurements be shared in the same way over all modifiers?\n* What are the measured results of word exclusion prompt steering vectors and specific numeric length modifiers? If they are failure cases, it will be helpful to see the extent to which they fail, and to share some of the typical behavior.\n* Do the populations of sampled vector differences separate cleanly for different tasks, or is there some overlap?  It will be informative to plot these.\n* Does the scaling factor induce a smooth tradeoff between output quality and accuracy in following the instruction?  It would be helpful o measure this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The derivation of steering vectors from instructions is an interesting research target and a good extension beyond similar work deriving such vectors from binary states or ICL prompts. The paper’s choice of deriving steering vectors from instruction modifiers is clever and novel, which allows the authors to create diverse training sets and also easily quantify the accuracy of results. The paper investigates nearly all the natural applications with useful measurements, including measurements of quality degradation in the presence of steering.  Positive results on negation and composition are interesting to see, and it is particularly interesting to see that vectors derived from instruction-tuned models work better than vectors from pretrained models when applied to pretrained models.", "weaknesses": "The presentation was sometimes uneven, giving the impression that the results might be curated to avoid showing results that did not work very well. Such cherry-picking should be resisted. For example: Figure 3 slices efficacy data by model and by instruction in different ways but does not provide breakouts by task for all four cases (bare prompt, prompt+modifier, prompt+steer, prompt+modifier+steer), which would help the reader build intuition about the failure cases.\n\nSimilarly: quality degradation was measured differently for different tasks, making it hard to compare. The paper would be improved if the appendix plotted or had a uniform table of quality degradation, computed the same way for every comparable task.\n\nOther failure cases are mentioned but not measured: word exclusion is described as unpromising due to the presence of an embedding signal, but measurements of its failure are not shown. Full results should be shown.\n\nIt is stated that “it seems impractical to compute a separate steering vector for each possible length,” however that doesn’t seem impractical at all. Such specific-length steering vectors should be computed and compared to one another, and also to the conciseness concept described in the paper. If they do not work, that should be quantified and shown.\n\nIn appendix tables 8 and 9, many configurations are omitted with the explanation that \"steering was unnecessary as the models typically followed the instruction.\"  Again, negative or \"unnecessary\" results should not be omitted. A key goal should be to explore and explain the limits of the observed effects. Failure or unhelpful cases are an important subject of experiments.\n\nSome natural questions are unanswered, for example, whether the clusters of steering vectors over which means are taken are cleanly separated from each other (i.e., before taking means) or not.  It would be informative to plot a projection of the raw steering vectors for several of the tasks, for example, in a scatterplot as done in Hendel 2023.  In particular it would be interesting to see how closely-related vectors such as “answer of length n” for various n are arranged with respect to each other in representation space.\n\nThe choice of “c” is not fully justified, and it seems that the scaling factor “c” might be arbitrary. For example, c might mediate a tradeoff between efficacy and quality degradation.  It would be informative to plot tradeoffs over a sweep of c, if that is the case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)