Do LLMs ``know'' internally when they follow instructions?
Authors: Juyeon Heo
Christina Heinze-Deml
Oussama Elachqar
Kwan Ho Ryan Chan
Shirley You Ren
Andrew Miller
Udhyakumar Nallasamy
Jaya Narain
Venue: ICLR 2025 Poster (2025)
Keywords: Instruction following
Large language models
Linear probing
Representation engineering
Interpretation
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify a dimension in LLMs' input representations tied to instruction-following success, showing that modifying it enhances adherence without compromising quality, offering insights into LLM behavior and advancing reliable AI agents.

Abstract: Instruction-following is crucial for building AI agents with large language models (LLMs), as these models must adhere strictly to user-provided constraints and guidelines. 
However, LLMs often fail to follow even simple and clear instructions.
To improve instruction-following behavior and prevent undesirable outputs, a deeper understanding of how LLMs' internal states relate to these outcomes is required.
In this work, we investigate whether LLMs encode information in their representations that correlates with instruction-following success—a property we term ``knowing internally''.
Our analysis identifies a direction in the input embedding space, termed the instruction-following dimension, that predicts whether a response will comply with a given instruction.
We find that this dimension generalizes well across unseen tasks but not across unseen instruction types.
We demonstrate that modifying representations along this dimension improves instruction-following success rates compared to random changes, without compromising response quality.
Further investigation reveals that this dimension is more closely related to the phrasing of prompts rather than the inherent difficulty of the task or instructions. 
This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged. 
This work provides insight into the internal workings of LLMs' instruction-following, paving the way for reliable LLM agents.

Published as a conference paper at ICLR 2025
Do LLMs "KNOW" INTERNALLY WHEN THEY FOLLOW INSTRUCTIONS?
Juyeon Heo" Christina Heinze-Demll Oussama Elachqar* Kwan Ho Ryan Chan Shirley Ren' Udhay Nallasamy² Andy Miller Jaya Narain²
'University of Cambridge Apple Unniersitty of Pennsylvania
0123340ccamaccuu
jnarain@apple.com
ABSTRACT
catruction-following crucial for building AI agents with large language mod- els (LLMs), these models must adhere strictly to user-provided constraints and guidelines. However, LLMs often fail to follow even simple and clear instruc- tions. To improve instruction-followin behavior and prevent undesirable outputs, a deeper understanding of how LLMs' internal states relate to these outcomes is required. In this work, we investigate whether L.MMs encode information in their representations that correlates with Aarructioonfollowingg success-a prop- erty we term "knowing internally". Our analysis identifies direction in the in- put embedding space, termed the instruction-followe whether a response will comply with a given instruction. We find that this dimen- sion generalizes well across unseen tasks but not across unseen instruction types. We demonstrate that modifying representations along this dimension improves instruction-folloinng success rates compared to random changes, without com- promising response quality. Further investigation reveals that this dimension more closely related to the phrasing of prompts rather than the inherent difficulty the task or instructions. This work provides insight into the internal workings of LLMs' instructioon-ollloiing paving the way for reliable LLM agents
dimension, that predicts
INTRODUCTION
Given the potential of large language models (LLMs), there has been significant interest utilizing these models to build personal AI agents. For instance, one could imagine deploying an LLM as a personal healthcare assistant, such as a fitness or nutrition planner, or for psychological counsel- ing (Li et al. 2024b; Wang et al. 2023 Tu al. 2024). Compared to traditional machine learning- based Al agents, LLMs offer the advantage of being easily adaptable through prompting, allowing users to provide guidelines and personal information without the need retrain model weights.
Instruction-Fol followiin critical in the development of personal AI agents with LLMs through prompts because these models must adhere the constraints and guidelines ensure safe and trustworthy interactions. For example, suppose an LLM is building a personal fitness plan for user with knee problems. To avoid knee problems for the user, the LLM must follow the instruction of not recommending knee-intensive movements or any exercises that could lead to potential injury. Similarly, in a nutrition planner, the LLM should avoid generating harmful recommendations, such as suggesting inappropriate food for pregnant women or children with diabetes.
However, LLMs often fail follow even unambiguous and simple instructions (Zhou et al. 2023) Qin et al. 2024 Xia et al.] 20244) Kim et al.] 20244 Yan et al.) 2024) like including keywords or following formatting guidelines. GPT-4 achieves around an 80% success rate on IFEval (Zhou et al. 2023), an instruction-followii
benchmark dataset, while smaller models have success rates around 30% to 40%. This raises the question: why do LLMs fail to follow instructions, even when those instructions are clear and familiar?
To gain better understanding of instruction- -fllowiinnn outcomes, we analyze the internal state of LLMs, focusing the differences in representations between success and failure cases of
Work done while at Apple. Code and data available http://siicccoooppppomocmmmmmmmm

---

Published as a conference paper at ICLR 2025
Example of agent
probes: Instructien-following
Representation Engineering
Task Please generans o wort-up plan for 50 year old
femele with knne proslem
intractions: Do include knee-recuiree posiiions
WAPM-LPP
Train-Test data splt
Rapder = Rongisal +
Hed dep
5 tyttretitzeeese
Shoulee
R: represenation
D: Instruction tolloiing direction
a: scaling tactor
Success case
Failure case
Figure I: Overview our paper. Left: Success and failure cases a personalized Al fitness planner. The task is to generate a warm-up plan while avoiding knee-required positions. The success case follows the instruction, while the failure case violates it. Middle: Linear probing is applied to an- alyze internal representations from success and failure cases, identifying the crrrructoonfollowiinn dimension. The probe tested on unseen tasks (e.g., writing a CV) and instruction types (e.g., include/exclude keywords). Right: Representation engineering is used to shift failure cases into success by adjusting the representations along the cnltruction-oolowinng dimension, improving ad- herence without compromising task quality.
instruction-followee following across different tokens and layers. Our approach involves disentangling the effects of tasks and instructions in input prompts, where the instruction specifies the action (c.g., 'please do not use keywords') ) and the task provides the context for executing the instruction (c.g.. 'please write resume'). ). By applying linear probing- widely used method for interpreting model
representations (Alain & Bengio 2016; Belinkov 2022 Elazar et al. 2021 2 -w identify a specific dimension within the input embedding space that is strongly associated with instruction-followee While previous work has primarily used linear probing to explore representations related to truthful-
ness and reducing hallucinations (Azaria & Mitchell] 2023; Marks & Tegmark| (2033] MacDiarmid
et al. 2024), our study extends this method to investigate instruction-followinn We demonstrate
that this dimension generalizes unseen tasks, however not to unseen instruction types.
To validate the significance of the instruction-followii dimension, we applied representation engi- neering techniques to enforce cetrructionfolloviiii based on insights from our linear probes. Our experiments show that adjustments along this specific dimension are more effective in enhancing success rates than random modifications, while maintaining the overall qual- ity of the generated responses. These results indicate that the instraction-followinn dimension plays a crucial role in shaping the model's behavior, toward better adherence instructions.
iestruction-follwwie
To further interpret the meaning of this dimension, we conduct a sensitivity analysis based on three key perturbations to the input prompt: task familiarity, instruction difficulty, and phrasing. Our findings reveal that this dimension is more related the rephrasing of prompts rather than the inherent difficulty of the task or instructions. This suggest that the way a prompt is encoded within the model's input representation space plays significant role in whether the instruction is followed correctly. This observation not only provides a deeper understanding of why LLMs sometimes fail to adhere to straightforward instructions but also offers explanation for the effectiveness of prompt engineering, even when the content of the prompt remains largely unchanged.
Overall, this work sheds light on the underlying mechanisms of instruction-following in LLMs by uncovering a critical dimension in the model's representation space. These insights enhance our understanding of LLM behavior and offer practical approaches improving instruction adherence, bringing us closer to developing more reliable and trustworthy AI agents.
1.1 CONTRIBUTIONS
We identify specific dimension within the input embeddings space of LLMs that is closely linked to foollwiinnfollooiiggg using linear probes, by carefully designing our setting to disentangle the effects of tasks and instructions in input prompts.
2

---

Published as a conference paper at ICLR 2025
We demonstrate that this dimension generalizes to unseen tasks and that modifying representa- tions along this dimension effectively converts mnstuuciionfolloviin failures into successes with- out compromising response quality.
Through a sensitivity analysis, our findings reveal that this dimension is linked to how prompts are rephrased, underscoring that darruction-following LLMs is influenced by how prompts are encoded within the model's s imput embeddings. This explains why LLMs sometimes fail to follow clear, simple instructions and why prompt engineering can enhance instruction adherence, even when the content remains largely unchanged.
2 Do LLMS KNOW WHEN THEY SUCCEED OR FAIL TO FOLLOW INSTRUCTIONS?
In this section, aim identify the dimension within the models' representation space that is closely associated with instruction-follovii We use linear probes to determine the internal signals that separate successful mnslluction-follooingg from failures and examine whether this dimension generalizes to different tasks and instruction types. By exploring different tokens and layers within the models, we seek understand how and when Istruction-following information is encoded.
2.1 SIMPLEE
To objectively evaluate LLMs with simple and verifiable instructions, we select IFEval (Zhou et al, 2023] as our base dataset. The motivation is that, while complex and multi-purpose instruction prompts are more realistic, they require using LLLM-based evaluators that may induce further errors and biases in assessing success or failure. To avoid this potential issue, we focus on simple, single- purpose and verifiable instructions from IFEval, such as "Please do not include keywords: "answer in lower-case only", that can be automatically validated with deterministic programs like string-matching, thereby minimizing uncertainties from ambiguous evaluation criteria. We provide a more detailed justification in Appendix A.6
or
The IFEval dataset comprises 25 instruction types under 9 categories, with each instruction type paired with a distinct set of tasks - approximately 20 tasks per instruction type. Furthermore, due to the relatively small number of tasks per instruction type, internal model states resulting from these prompts contain a mix of both dimension related specifically to cnstruction-folloinnn val data, called IFEval-simple? First, we selected 5 instruction types that are likely to be used in real-world applications for Al agents. For example, ensuring the inclusion (keywords:xxistence) or exclusion kkcywords.orbbddee) of specific keywords, specifying the frequency of certain keywords skeywords:frequency generating responses with placeholders (detectable content:place holders), and requiring responses to end with predefined sentences (startend:end checker). We excluded more complex or impractical instructions, such as those requiring omission of punctuation, as they are less relevant for practical use cases.
and task-specific details. To isolate the we generated modified version of the IFE-
Second, we generated 100 tasks using GPT-4, similar to the original tasks IFEval, where each instruction type is paired with the same set of 100 tasks. By pairing each instruction type with the same set of 100 tasks, we ensure that linear probes trained on the model's 's representations are more likely to capture information solely related to instruction-folloinng ence of varying tasks. The instructions assigned to each task vary in detail based on the context. For example, for an instruction type focused on keyword inclusion or exclusion, a resume- werttegrringgg task might require keywords like 'skills' and 'career', while a joke about a programmer might involve terms like 'syntax" or 'code'. These variations introduce diverse challenges, testing the model's adaptability in following instructions. Example tasks are provided in Appendix Table and Table The instruction toaruciion-oolooiingg accuracy for -iEypeesimple datasets is presented in Appendix Table
without the confounding influ-
2.2 METHOOSS
Representations We analyzed four language models: LLaMA-2-7B-chan Touvron et al. 2023),
LLaMA-2-13B-cha (Touvron al.] 2023). 10. (Tiang et al.) (023). and Phi-
"The IFEval-simple data is available https//gtttubbcoomaalleemmmmennn -tt.://emmiiiiiiiiiiiinttooooooo fnstractioon-ollloinngg.
3

---

Published as a conference paper at ICLR 2025
Task generalization
Instruction-type generalization
Model
First token
Middle token
Last token
First token
Middle token
Last token
LLaMA-2-chat-2 (14 lyr)
0.77 0.04
0.55 0.07
0.73 1 0.04
0.52 1 0.03
0.50 1 0.07
0.52 1 0.05
LLaMM--2chat-113 (16 lyr)
0.83 + 0.03
0.58 + 0.06
0.82 + 0.03
0.56 + 0.06
0.58 + 0.06
0.53 + 0.03
Distral-7B-inst--ee (14 lyr)
0.74 + 0.02
0.54 + 0.05
0.72 + 0.04
0.50 + 0.05
0,51 + 0.05
0.51 + 0.05
Phi-3-mini--2288 (14 lyr)
0.88 1 0.03
0,56 1 0.04
0.86 1 0.03
0.55 1 0.04
0.48 1 0.03
0.50 T 0.03
Table 1: Task and instruction-type generalization AUROC scores for task and instruction-type generalization using a 70-30 train-test split for task generalization on unseen tasks, and leave-one- out cross-validation for instruction-tye deviation is calculated from five runs with different random seeds for task generalization and across instruction types for instruction-type generalization.
generalization across different instruction types. Standard
Early layors
Middle Bayers
Last layers
Model
First token
Middle token
Last tokeen
First token
Middle token
Last token
First token
Middle token
Last token
clut
0.75
0.04
0.75 4: 0.05
0.04
0.03
0.02
LLaMA shat-13
0.52 0.00
0.02
0.04
0.79
Mistra-70 7B -0.3
0,74 0.02
0.51 0.05
0.72
0.01
0.71
0.05
0.51 0.03
0.67 0.04
0.71
0.03
0.49
0.04
0,70 0.03
Phi-3 1ai33-mini-225
0.88 0.03
0.56 0.04
0.86 = 0.03
0.85
+ 0.03
0.56 #: 0.03
0.83 + 0.02
0.68 + 0.05
0.53 0.03
0.63 0.04
Table 2: Task generalization (detailed across layers) AUROC scores for the first, middle, and last tokens across early, middle, and last layers of various models. The layers selected for L-aMA-2- 13B-chat are 16, 32, and 40, while for the other three models, the layers used are 14, 26, and 32.
2024). For each model, we looked at the representations on T2, Fn), where x; are the tokens in the input prompt; (2) middle token, LLM 222 22..... Tn, 1/, v....., -Um/2), where yj are the first m/2 tokens of the response; and (3) last token, (Z1, J2, In. y1, U...... .ym), representing the full input and response. We also examined three layers (early, middle, last) to identify where information is encoded within the models' internal state. Specifically, we used layers 16, 32, and 40 and for 1-MMA-2-13B-hha and 14, 26, and 32 for other three models. To avoid randomness in decoding, we employed greedy decoding without sampling.
Siini-22kkinnse (Abdin et al.
three tokens: (1) first token, LLM
cnerructionfollowiiii
Linear Probes We trained linear probes the representations identify the instruction-following dimension. A simple linear model was trained on cerrructio-followiiii success outcome, optimized for 1000 epochs with AdamW, 0.001 learning rate, and 0.1 weight decay.
Train-test split and metric We assessed task generalization and instruction-type generalization by splitting the data into training and testing sets, as shown in Figure [ IFEval-simple has 5 instruc- tion types, each paired with the same set of 100 tasks. To evaluate task generalization, we split the data by the task dimension, using a 70-30 train-test split across the 100 tasks. To evaluate instruction-type generalization, we applied a leave-one-out approach, over the instruction-type di- mension. To evaluate performance, we use the Area Under the Receiver Operating Characteristic Curve (AUC) et al. 2011), assessing the accuracy of binary predictions for each model on unseen tasks and instruction types.
2.3 RESULTS
Linear probes generalize across unseen tasks The task generalization results in Table show that linear probes performed well across different tasks when the instruction type remains consis- tent. The AUROC scores, which range from 0.7 0.8 using the first token, suggest that the input embeddings of these models possess shared geometry related to alizes well across varied tasks. This is particularly beneficial in the context of buliding Al agents, where a pre-defined consistent set of instructions needs to be followed across different tasks. For example, if a probe is trained on examples of an instruction type like "Please do not include these keywords" using examples from resume writing and nutrition coaching, the linear probe can predict the model follows the same instructions type even unseen tasks, such as creating a warm-up plan without knee-intensive exercises. Additionally, using representations from the first token and early layers, fitting the PCA on the training split and visualizing the results on the test split (unseen tasks) in Figure 2 They show clear separability, sup-
that gener-
plot the principal components analysis (PCA)
4

---

Published as a conference paper at ICLR 2025
Instructions
LLaMA-2<huu-78
Mistrul 10-
Phi-3-mini-125k
Early byr
Middlle
Last
Early
Middle byr
Last
Early
Middle byr
Last by
Early
Middle
Last byr
key ley:feeiddenn
0.52
0.51
0.56
0.45
0.45
0.44
0.44
0.41
0.46
0.52
0.54
0.53
segresist
0.50
0.50
0.51
0.67
0.68
0.66
0.55
0.50
0.50
0.63
0.67
0.65
ley:dree
0.57
0.59
0.59
0.57
0.57
0.57
0.56
0.56
0.55
number placeholders
0.56
0.54
0.52
0.58
0.58
0.54
0.50
0.49
0.50
0.50
0.53
046
end checker
0.48
0.46
0.47
0.35
0.57
0.56
0.41
0.42
0.45
0.55
0.59
0.7
AVERAGR
0.53
0.52
0.53
0.56
0.57
0.55
0.50
0.45
0.49
0.55
0.58
0.56
Table 3: Instruction-type generalization (detailed) AUROC across different models and selected layers on first token representations. A lcave-one-out approach was employed. and the standard deviation from training linear probe is small enough be omitted from the table. The '-" mark in keywords:frequency instruction type is due to an insufficient number of data points caused by 100% success rate, making it impossible to compute reliable AUC scores.
(a) Llama-2- 1-13b-chathth
(b) Llama-2- 2-7b-chat-hf
(c) Mistral-7B ---st---
(d) 1-ik---i8t
Figure 2: PCA plot of first token representations from early layers across four L.MMs. PCA is fitted on the training split and visualized on the test split (unseen tasks). The PCA shows separability, suggesting the consistent capture of the cnttruttionfollowwwit includes three instruction types from the keyword category in IFEval-simple. Additional PCA results for all five instruction types across different categories are provided in Appendix Figure 6
dimension across tasks. The analysis
porting the idea that the instruction-folloving dimension is consistently represented across different tasks. Further PCA analysis is provided in Figure the Appendix.
Linear probes do not generalize across unseen instruction types In contrast to task generaliza- tion, the models exhibit no clear generalization when tested across unseen instruction types. The AUROC scores for instruction-type generalization are notably lower, ranging from 0.50 to 0.55, close to chance (Table Dp. A potential explanation for this poor generalization could be the limited number of instruction types used during training. where the linear probe was trained on just 4 in- struction types. To investigate, we expanded the dataset to include 25 instruction types, each paired with 20 tasks. However, shown in Appendix in Table & this expanded experiment yielded similar results, with models still failing to generalize well across unseen instruction types. This indicates that models struggle to generalize instruction-folle following across different instruction types, implying the absence of a 'global' instruction-follovii dimension that can be leveraged regardless of the instruction type, which may be due to varying representation geometries.
First token is as informative as last token Interestingly, the first and last tokens-reprreenting the model's state before and after response generation-Shon high AUROC scores, implying that LLMs may already "know" whether they will follow instructions even before they start generating their responses. This early indication of instruction following is valuable, since early intervention or correction could be applied. In contrast, the middle tokens showed lower AUROC scores, likely because the representation contains information about next token generation more than information about contrucction-oolloviii
Layer-wise performance is similar, with early layers slightly better for task generalization The performance across different layers shows only slight variations, with early layers marginally out- performing middle and last layers, as detailed Table For example, in the 13B model, the early layers achieve an AUROC of 0.83 for the early token, which is slightly better than the performance of middle and last layers. This suggests that the intructtionfollowinn dimension may be more prominently represented in the earlier stages of the model's processing. However, for instruction- type generalization, there is no clear pattern across layers (Table B3), indicating that the challenges associated with generalizing across different instruction types pervasive throughout layers.
5

---

Published as a conference paper at ICLR 2025
Model
Original SR
Random SR
Inst-follow SR
Original QR
Random QR
Inst-follow QR
ITaMA-2--cht--7
0.57 + 0.00
0.55 0.00
0.59 + 0.00
0.87 + 0.09
0.85 + +.10
0.87 0.08
LLaMA-2-chat 13B
0.61 1 0.00
0.54 1 0.12
0.65 1 0.02
0.92 1 0.00
0.91 1: 0.02
0.94 0.00
Mistral-7B-inst-chl
0.58 = 0.00
0.56 = 0.02
0.64 = 0.02
0.95 + 0.02
0.86 + 0.02
0.98 0.06
Phi-3-mini-1233
0.71 0.00
0.63 0.04
0.74 0.01
0.76 0.01
0.76 0.01
0.78 0.00
Table 4: Representation Engineering results on the last layer across four models. Success rate (SR) for instruction-folloiing and quality ratio (QR) for task quality are compared across the orig- inal outputs, outputs using the instruction-followe tions. RE along the instruction-follee following dimension improves SR while maintaining or enhancing QR, unlike random adjustments which often reduce both SR and QR. Standard deviations are across three runs with different random seeds.
dimension, and outputs using a random direc-
Success Rates 1SR)
Success Conversion Rabio (SCR)
Success Presenvation Rabio (SPR)
0:50
5.9
A.2
0.55
0.19
0.4
0.50
Model
Model
Model
Original Random
Inst-flluw
Figure 3: Transition metric for Representation Engineering on the last layer of four models Success rate (SR) only on high quality responses task execution (scoring above by GPT-4, scale from 0 to 9). The Success conversion ratio (SCR) indicates the proportion of originally failed responses that became successful after modification, while Success preservation ratio (SPR) reflects the proportion of originally successful responses that remained successful.
REPRESENTATION ENGINEERING
We identified dimension within the input embedding space associated with cetruuction-folloinng To evaluate whether this dimension significantly impacts the models' behavior, we manipulated the representations along this direction using representation engineering (Marks & Tegmark 2023 Zou et aL. 2023). An increase in the models' instruction-followim success rate tied to manipulations along the identified direction validates the role of the dimension in shaping the models' generation outcomes toward instruction adherence.
3.1 SETTINGS
Method For each input representation Reriginats we applied transformation in the identified di- rection D using the formula Rapdated Roriginal D, where is scaling byper-parameted We applied this transformation to all input representations, including both success and failure cases, evaluate whether RE could improve instruction following universally, without disrupting cases where the model was already successful. This adjustment was applied to the representations in the last layer of the model, as it was more robust variations a. We focused on the representation of the first token, which corresponds to the input embedding before any response generation, since the goal of representation engineering (RF) adjust internal representations before the response is generated to improve the model's 's instruction adherence. The direction D) is the weight of linear probes trained on all IFEval-simple dataset. [
Metric We evaluated the success rate (SR) of instruction-follooiing using predefined evaluation
functions from the IFEval (Zhou et al. 2023). Additionally, we assessed the quality of the responses
We also experimented with training the linear probe on 70% of the IFEval-simple dataset and applying RE to the remaining 30% test set. The results were similar but slightly worse than when the linear probe was trained and RE was applied the entire dataset. Since our primary focus is analyzing the variance caused by RE itself, rather than variance from train-test splits, we present the results using the full dataset here.

---

Published as a conference paper at ICLR 2025
using GPT-4, scoring each response scale from 0 to based on its relevance to the given task. We defined quality ratio (QR) as the number of responses scoring above divided by the total number of responses that successfully follow instructions (this cutoff was defined based on the distribution of quality scores). F2T (False to True) and T2T (True to True) show how many failed responses became successful and how many successful ones remained after modification. The F2T Success conversion ratio (SCR)= = FFFT+F2F) indicates the proportion of originally failed responses
that became successful after modification, while Success preservation ratio (SPR) = (T2T4T2F) reflects the proportion of originally successful responses that remained successful.
T2T
Baseline and hyperparameter selection To demonstrate the effectiveness of the identified mnstruction-followiin dimension, we compared it against random directions. Each model and in- struction type required different a value based on their specific geometry. If a is too large, it can degrade the quality of responses; if too small, it may not effectively improve instruction-fllloing We selected a for each model and instruction type using a validation set comprising 10% of the instruction data. The selected values were: 0.3 for Llama-2-chat 13b and Llama-2-chat 0.1 for Phi-3, and 0.15 for Mistral-7B.
Prompt for scoring task quality
You are helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to score given output for the given instruction. You should give an overall score integer) on scale of 9, where higher score indicates better overall performance. Do NOT provide any explanation for your evaluation.
# Instruction: (Task-only-input)
# Output:[Responss Response)
# Score of the Output (Your response should be ONLY the score, an integer between 0-9);
3.2 RESULTS
RE on ceeruction-followiin direction improves success rate while maintaining quality Our ex- periments demonstrate that applying the RE direction generally improves the instruction followiigg success rate (SR) across most models and instruction types. As shown in Table 4 the SR with the instruction-following direction usually outperforms the original success rate and is lower bounded by the the original SR that is, the instruction-followe followwigg dimension does not lead to worse than original SRs. Additionally, the QR remains equal to or higher than the original, indicating that RE can be applied with minimal risk of reducing response quality. Figure an illustrative example of modified responses. In this case, the task was to write a resume with the instruction include three specific keywords. The original response only included one keyword, whereas the modified response, guided by the instruction- -oolloiion-fllooiing direction, successfully incorpo- rated all three keywords, demonstrating the effectiveness of RE in enhancing instruction adherence.
the Appendix provides
Instraction-fflloving direction is better than random directions When comparing RE direction random directions, RE consistently outperforms random directions in increasing the success rate across all instruction types and models, as illustrated in Table 4 and Figure 3 The ratios of True- to-True (T2T) and False-to- True (F2T) transitions are typically larger for the instruction-flllowigg direction than for random directions, indicating a more reliable improvement in success rates.
4 INTERPRETING THE INSTRUCTION DIMENSION
While manipulating representations along the instruction- -ostuucioo-follloiing dimension reveals that it influ- ences a model's behavior, the meaning behind this manipulation remains unclear. To interpret the meaning of the instruction-following dimension, relative of perturbations on the internal state of LLMs, compared to our identified direction. We consider three perturbation types: task familiarity, instruction difficulty, and phrasing. We (1) sys- tematically alter the original input prompts in IFEval -imppeeimppl dataset for each perturbation, (2) com- pute the resulting difference in internal state representation space before and after the perturbation,
conduct a sensitivity analysis to investigate the

---

Published as a conference paper at ICLR 2025
and (3) compute the cosine similarity between the perturbation induced difference vector and the instruction instructionfoolooiing dimension we identified. We designed prompt changes for each perturbation:
(1) Task Familiarity: We investigated whether the instruction-followin dimension might be lated to how familiar the model is with given task. For example, the task "Write a resume for soft- ware engineer" might be more familiar to the model than "Write summary about current events", it was more common in the data used to train the LLMs. If a task is more familiar to a model, may be easier for the model follow instructions regarding that task. To perturb the model on task familiarity, we kept the instruction constant while changing the task to one with lower perplexity JJelnnkk et al. [977). Perplexity measures the probability of tokens in generation, reflecting task familiarity (Gonen et al.) (022). where high perplexity indicates a familiar task and vice versa.
(2) Instruction Difficulty: We investigated the relationship of the cestruction-folloviin dimension with the complexity of the instructions. We perturbed the instruction difficulty by simplifying in- structions by relaxing instruction-relatee constraints. For example, in the original instruction "Please include keywords: coding, Python, computer, experience", we reduced the complexity by reducing the number of keywords required in the instruction to "Please include the keywords: coding".
(3) Phrasing Modification: Finally, we examined whether the instraction-following dimension was correlated how the prompt is phrased. We rephrased the prompts while keeping the meaning of the task and the instruction unchanged. For example, we modified "Write resumee for software engineer. Please include keywords such as coding, Python, computer, experience" to "I want you to write about software engineer resume including four words coding, Python, computer, or expe- rience". We used GPT-4 to rephrase both the task and instruction in the input prompt, and applied GPT-4 again to validate that the meaning of the contents remained the same after rephrasing.
We selected 20 prompts, each containing a task and an instruction from the 'forbidden keyword' instruction type in IFEval-simple dataset. For each perturbation type, created five modified versions of each prompt. We then averaged the representations of these modified prompts and cal- culated the difference between this averaged representation and the representation of the original prompt. Finally, we assessed how well this difference vector aligned with the instruction-follovin dimension by computing the cosine similarity.
Our findings, illustrated in Figure 4 show the sensitivy analysis results for two models: L-ama-2- 13b-chat and Llama-2-7b-char In both models, the results indicated that phrasing modifications have a stronger correlation with the instruction-folloiin dimension than task familiarity instruc- tion difficulty. These results support the hypothesis that the instruction- -orructtoo-ooloiinng dimension is more closely tied to how prompts are phrased rather than the inherent difficulty of the task or the complex- ity of the instruction. This suggests that how prompts are phrased plays a critical role in determining whether LLMs will successfully follow the instructions, aligned to observations Lu al. (023) Selar et al. (203)) showing LLMs are sensitive to prompt formatting.
5 RELATED WORK
Instructioon-follvii in LLMs Recent research has introduced various benchmark datasets to eval- uate the instruction-followiii capabilities of LLMS across different contexts(Zhou et al.) 2023 Qin al. 20044] Yan et al. 2024 Xia al. 2024). Beyond evaluation, several approaches have been proposed to improve mnsructiionffolloiii performance, such as modifying attention mechanisms (hhng et al., 2023) and applying fine-tuning strategies (He et al. 2024 Sun al. 2024). In con-
ciorrcction-ollowinn our study aims understand why LLMs sometimes fail to follow instructions by analyzing internal represen-
trast to prior work that primarily focuses on evaluating or enhancing
tations.
Linear Probing and Representation engineering on LLMs Linear probes have been widely used for interpreting and analyzing the representations of neural networks (Alain & Bengio, 2016) and
language models [Belinkov] 2022 Elazar et al. (02T). Specifically, probing for the trustworthiness
of LLMs has been an active area of research (Azaria & Mitchell 2023 Marks & Tegmark 2023
MacDiarmid et al., 2024 Li et al. 2024a Burns et al.) [022] Zou et al. 2023] Rimsky et al.
2023
Li et al. 2022 Nanda et al. 2023 Subramani et al. [022] Tigges et al. 2023 Todd et al.
2023
Farquhar et al. 2024 Ahdritz et al. [2024] Duan et al. 2024). These probing methods are closely
related to representation engineering and editing techniques aimed at modifying model knowledge
8

---

Published as a conference paper at ICLR 2025
Sensitivity anelysis of three perturbation innstructionfolllweee dimension
Modifranion
Modificttion
Figure 4: Cosine similarity alignment for modified data in the 'forbidden keyword' instruction type across two models (Llama-2-7b-che) (Left) and Llama-2-13b-chat (Right)). The figure shows the cosine similarity between the cneructtio-followiini dimension and the difference vector (com- puted as the difference between the original prompt's representation and the average representation of five modified prompts) across 20 sampled prompts. Modifications include changes in task fa- miliarity, instruction difficulty, and phrasing. The results indicate that phrasing modifications align more closely with the instruction-follovinn dimension, suggesting that how prompts are phrased plays a erucial role in determining instruction adherence.
and behavior (Zou et al. 2023 Rimsky al. 2023: Li et al. 2024a Park al., 2023 Chen & Yang 2023 Luo et al. 2024 Turner al. (2023). Our work is distinct from these previous efforts, which primarily focus on representations related to truthfulness and reducing hallucinations. In contrast, study centers representations related to mnstruction-oolloiing highlighting the importance of understanding how models internally handle instructions.
DISCUSSION AND CONCLUSION
6.1 LLMS INTERNALLY RECOGNIZE WHETHER THEY WILL FOLLOW INSTRUCTIONS
Our findings suggest that LLMs may possess inherent ability to predict whether they will success- fully follow instructions, even before the generation process begins. This capability is supported by several key observations:
LLMS generalize well across tasks but struggle with different instruction types We find that while LLMs generalize across different tasks, they struggle with generalization across different instruction types. This suggests that distinct instruction categories may have unique geometries within the models' internal representation space, making it more challenging generalize across them.
LLMs predict instruction success from the first token We observe that the model's inter- nal representations are separable from the very first token, which corresponds to the embedding of the input prompt. This indicates that the likelihood of instruction-follovin success can be deter- mined early in the process, before the model generates any responses. This highlights the critical role of how the input prompt encoded and the importance of input representations in predicting instruction-followinn outcomes.
Representation engineering increases instruction-follovine success We further validate the sig- nificance of the identified instruction-followwe dimension by adjusting the model's representations. By moving failure cases into the success class along this dimension and comparing the results to ran- dom adjustments, observe significant increase in the success rate while keeping the task quality. This demonstrates that the identified dimension is both meaningful and can used practically.
The instruction-follot followwng dimension is closely tied to prompt phrasing Our findings, in Figure reveal that the -oolloiioon-olooiinng dimension is most closely associated with the phrasing of prompts, rather than the inherent difficulty of the task the specific details of the instructions. This suggests that how instructions are phrased plays a crucial role in whether LLMs will follow them and is consistent with finding the separability of representations from the early token.

---

Published as a conference paper at ICLR 2025
6.2 THE ROLE OF INPUT PROMPT REPRESENTATION IN INSTRUCTION-FOLLLON FAILURES
Our findings highlight the role of representation of the input prompt in determining instruction- following success LLMs. We discover that the instruction-folloving dimension identified in our analysis is sensitive to changes in how the input prompt is phrased. This sensitivity explains several behaviors of LLMs:
Why LLMs fail in following instructions LLMs may fail to follow even simple, clear instructions because the encoding of the input prompt within the models' internal representation space can be easily disrupted. Our findings suggest that small variations in how a prompt is phrased can result in significant differences in how the model processes the instruction, leading failures in adherence. This issue arises not from ambiguity in the instruction itself, but from the LLM's sensitivity to the exact structure and phrasing of the input, which influences how the instruction is embedded and processed internally. As a result, the model might not consistently follow instructions, even when they are clear and familiar.
Why Prompt Engineering (PE) works PE operates by slightly altering the phrasing of prompt, which in turn changes how the input is encoded within the model. This subtle shift in encoding can move a representation from failure class to success class in terms of instruction followwng within the input embedding space. Our work with representation engineering achieves a similar outcome, but instead of modifying the input text, make adjustments directly in the representation space. Both approaches influence the model's internal states, highlighting the importance of the input encoding process. Our observations align with prior research showing LLM sensitivity to prompt formatting (Lu et al. 2023 Sclar et al. 2023 Gonen et al. 2022).
success or failure can be altered by slight prompt rephrasing shows that the LLM's input embedding space is semantically sensitive. This sensitivity suggests that the model's internal representation of prompts is brittle, making LLMs vulnerable to small changes in how an input is framed or phrased. This fragility, likely driven by the model's large size and the complexity of its training dynamies, creates challenges in ensuring robust instruction adherence. Given this sensitivity, future efforts should focus on making LLMs' input embedding space more robust and reliable. One potential approach is to fine-tune models with an explicit focus stabilizing instruction-following by utilizing the identified instruction-follooinn dimension.
Semantic sensitivity of LLM input embedding space The fact that instruction-
Our findings highlight the crucial role of prompt encoding in cnstructionfoollwiing success for LLMs. The sensitivity of the input embedding space to slight changes in phrasing explains why LLMS may fail follow even clear instructions and why prompt engineering is effective. By ad- justing the representations directly, as we did with representation engineering, we show that it is possible to significantly improve instruction adherence. Going forward, improving the robustness of LLMs' input embeddings through training can make models more reliable and consistent in fol- lowing instructions across a variety of tasks. This crucial for building trustworthy Al systems, especially in real-world applications where accuracy and reliability are essential.
6.3 LIMITATIONS AND FUTURE WORK
Our analysis was primarily focused on a specific set of tasks and models. Although our current results are consistent across the models we studied, future work could extend these findings by evaluating additional models to determine whether the identified instraction-followin dimension generalizes across different LLM architectures. Additionally, expanding the dataset to include cases could enrich the analysis and improve the generaliz-
wider variety of instruction-followii ability of our findings. We focused our investigation on simple modeling approaches to identify an mnstruction-followii clude additional methods train linear probes, particularly in handling domain shifts. Similarly, better approaches to representation engineering (Zou et al. (203)) could further improve the suc- cess rate of instruction-following modifications. Finally, unanbiguously interpreting the meaning of the cestruction-folloiin dimension remains an open question. We considered three hypothe- ses and found that phrasing modification was most closely related to the dimension associated with cioructtio-followwiii using a perturbation based approach. Additional investigations develop sys- tematic approaches to interpret the dimension could add to a deeper understanding of its meaning and implications.
dimension and evaluate its practical significance. Future work could in-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was conducted during an internship at Apple AIML. We sincerely thank Fahad Kamran and Feng Zhu for their valuable feedback and insightful suggestions on this work. We are also grateful Guillermo Sapiro for his unwavering support and guidance throughout the research.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 2024.
Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L Edelman. Distinguishing the knowable from the unknowable with language models. arXiv preprint Skki::2402.035633 2024.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. arXiv preprint 13734, 2023.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXivv22222.03827,22 2022.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXiv:2310. 20150, 2023.
Hanyu Duan, Yi Yang, and Kar Yan Tam. Do Ilms know about hallucination? an empirical investi- gation of Ilm's 's hidden states. arXiv preprint arXhv:2402.09733 2024.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021.
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(801 630, 2024.
Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettleoyer. Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:22/2.04037, 2022.
Qianyu He, Jie Zeng. Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint (tl::2004//844 2024.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): $63-$63, 1977.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 1000022310.06825, 2023.
Jihoo Kim, Wonho Song. Dahyun Kim, Yunsu Kim, Yungi Kim, and Chanjun Park. Evalverse: Uni- fied and accessible library for large language model evaluation. arXiv preprint arXiv:2404.009.0 009444 2024.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210. 10. 2022.
11

---

Published as a conference paper at ICLR 2025
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tine intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024a.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal Ilm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.0542 240.0204..5499 2024b.
Sheng Lu, Hendrik Schuff, and Iryna Gurevych. How are prompts different in terms of sensitivity? arXiv preprint arXiv:2311.07255 2023.
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison-Burch, and René Vidal. Pace: Parsimonious concept engineering for large language models. arXiv preprint 1433:3240060433,, 2024.
Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duve- naud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes can catch sleeper agents, 2024. URL hnttpss:/www.antrrror com hoom/news/probese cttp://ppppoeeeeccc-ec-cccc-c-ee weeerr-ageeee
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint (tfi::2311.036555 2023.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401 03601, 2024.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features prompt design or: How learned to start worrying about prompt 2023.
formatting. arXiv preprint
Nishant Subramani, Nivedita Suresh, and Matthew E Peters. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv:2205.05124. 2022.
Haoran Sun, Lixin Liu, Junjie Li. Fengyu Wang. Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained inslraction-folloiing ability of large language models. arXiv preprint arXiv:2404. 020823, 2024.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154, 2023.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXi:23377009288 2023.
Tao Tu, Anil Palepu, Mike Schackermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. Towards conversational diagnostic ai. arXiv preprint ar 400i 4,055444 2024.
12

---

Published as a conference paper at ICLR 2025
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint 2XX1:23088.0245 2023.
Hongru Wang, Rui Wang, Fei Mi, Yang Deng. Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. Cue-cot: Chain-of-thought prompting for responding to in-depth dialogue questions with Ilms. arXiv preprint arXiv:2305.1199. 2023.
Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang. Yihao Feng. Ran Xu, Wenpeng Yin, and Caiming Xiong. Fofo: A benchmark evaluate Ilms' format-following capability, arXiv preprint vrXiv:2402.18663 2024.
Jianhao Yan, Yun Luo, and Yue Zhang. Refutebench: Evaluating refuting instruction-followinng for large language models. arXiv preprint arXiv:2402.1346. 2024.
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for 1lms. arXiv preprint arXiv:2311.02222, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followig evaluation for large language models, arXiv preprint arXiv:2311.09911, 2023.
Andy Zou, Long Phan, Sarah Chen, James Camphell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310001005, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXAMPLES OF LIEVAL-SIMPLE DATASET
The IFEval-simple dataset is created to focus specifically instruction-followingg removing the confounding influence of varying tasks present in the IFEval dataset (Zhou et al. (023). In this modified version, we select instruction types commonly used in real-world Al applications, such as including excluding keywords, generating responses with placeholders, and ensuring specific phrases are present in the generated text. These instructions are paired with the same set of 100 tasks to help isolate the entruction-followiii dimension. By using the same set of tasks across all instruction types, we ensure that any differences in model behavior are attributed to instruction- following rather than task-specific features. This allows to more effectively probe the model's internal representations and evaluate how well it can follow instructions across various scenarios.
Table 5 presents examples from the IFEval-simple dataset, such as tasks like writing resume or creating a joke about programmers. The instructions assigned each task vary, requiring the model to follow specific guidelines such as including excluding certain keywords, ensuring word us- age meets a specific frequency, and adhering to formatting rules. The keywords that must be in- cluded excluded differ based the task. For instance, in the resume task, keywords might include "resume", "software", or "engineer", whereas in the joke task, the focus may shift to terms like "syntax" or "code". These varied instructions introduce diverse challenges for the model in instruction-folloingg
Type
Example
Task
Write resume for software engineer with 5+ years of experience in the Bay Area,
keywordstexistence
Make sure to include the keywords: "skills", "technology", "career".
keywords:forbidden
Do include the following keywords: resume, software, engineer, experience.
Instruction
Make to the word "qualifications" least 2 times.
startend:end checker
Your resume must end with the exact phrase "Looking forward to contributing to innovative projects."
detectable content:number placeholders
Make sure to include at least 5 placcholders represented by square brackets, such as [name].
Task
Write joke about programmers.
keywordstexistence
Make to include the keywords: "humor", "code", "life".
keywords.orrbideen
Do not include the following keywords: joke, programmers.
keywords:frequency
Instruction
Make sure to use the word "syntux" at least times.
startend:end checker
Your programmer joke must end with exact phrase "And the real bug the of life."
detectable content:number placeholders
Make sure to include least placcholders represented by square brackets, such [name].
Table Examples from the IFEval-simple dataset. This table shows two tasks: writing resume and crafting joke about programmers. Each task is paired with multiple instruction types, such as including/excluding keywords, ensuring word frequency, and adhering to specific content format- ting rules. The uniform set of tasks across different instruction types helps isolate the instruction- following dimension by removing task-specific variations.
14

---

Published as a conference paper at ICLR 2025
Index
Task
Write a story about the importance of understanding the truths that are not obvious.
2
Write serious riddle about trips and stitches in poem style.
Wrile a rubric for leenagers on how to review a book.
4
Write a persuasive email lo a teenager who lives in Aberdeen, Scotland.
5
Write resumee for software engineer with 3+ years of experience in the Bay Area, CA.
6
Wrile song about regrets in the style of Taylor Swilt.
Wrile an essay about Alvin and the Chipmunks.
8
The Legend of the Sword and the Fairr is a movie in which Wan Wan is a villain. Write a story about Wan Wan's ' character.
9
Write story about a family that goes camping in the woods.
10
Write an obviously fake news article saying that aliens have invaded earth. Make it funny.
11
Write a song about the benefits of eating your vegetables.
12
Write startup pitch for "Ward and Guerre".
13
Is Scoul good place to live?
14
Write a letter to a friend asking them to go and vote.
15
Write a resume for a fresh high school graduate who is seeking their first
16
Is praying for someone's 's health a good idea?
17
What's the difference between a 2-stroke and a 4-stroke motor?
18
Explain a group of elementary school students why we have seasons.
19
Can you re-create a story from a lictional newspaper with the tille: "A man mysteriously died in his house, and police are investigating""
20
Come up with proposal for new research project on how to improve the quality of life for people with disabilities.
21
Write a blog post about the benelits of meditation for busy protessionals.
22
Create a recipe for vegan gluten-Iree chocolate cake.
23
Draft a comprehensive guide on how to start podcast.
24
Develop character sketch for villain in fantasy novel.
25
Compose a haiku about a sunset over the ocean.
26
Summarize the plot of the film "Inception".
27
Explain the theory of relativity in simple terms.
28
Write review of the latest iPhone model.
29
Descrihe the lifecycle a butterfly.
30
Propose a business plan for a sustainable fashion brand.
31
Outline the steps for training a puppy.
32
Discuss the impact of social media tcenage mental health.
33
Draft a speech for a climate change conference.
34
Write a joke about programmers.
35
Explain how to change
tire.
36
Develop fitness routine for beginners.
37
Compose a sonnet about the city of Venice.
38
Write a user manual for a smartwatch.
39
Describe typical day in ancient Rome.
40
Provide advice on how to improve public speaking skills.
41
Discuss the effects of global warming on polar bears.
42
Draft a letter of recommendation for a student.
43
Summarize the story of "The Great Gatsby".
44
Explain the process of photosynthesis.
45
Write a critique of a famous painting.
46
Develop a marketing strategy for new video game.
47
Compose a limerick about mischievous cat.
48
Describe the benefits of yoga for athletes.
49
Write instructions for assembling a desk.
50
Discuss the history of the internet.
Table 6: Sample of 50 tasks from the IFEval-simple dataset. This table provides subset of 50 tasks from the IFEval-simple dataset, which includes a total of 100 tasks designed to evaluate mnstruction-following performance.
15

---

Published as a conference paper at ICLR 2025
A.2 EXAMPLE OF REPRESENTATION ENGINEERING
Instruction
Write resure for software engineer with years of experience in Bay Area, CA.
Make sure to include the keywards: "skills", "technology", "career"
Original response
Modified response
ber's snple rearr sufbere angiresr with part rxparienss the day Area,
wen't uabls resuse for sefbars arginaer with Se warr expertance in Day Arrt,
Setrrn Zal
Aetron) Srai. vial
deltonury countiay wiltere orgireria pasllae that viltiss arpenicatio.
skills
technology carser
the capaery
kree, CL Prolicient in vide raree of bed bolleo and proguentry iapago, with strmng
trick "tard
krey, Posticiinn a viee near of pregeemnn laggupts, onc schere ten isstes, and
Freme track worrd decivering high-quliity softuare prodects tim and Ealget.
Tacinica) Skills:
Jwateerigt
patsaitas rosk, Pacig
1a02n, Ejarpo loact
Frenere:: grtre S1889, kanct
tervior centrol systess Git, 50
torsicr centrol nstens Gt. 9A
Career lightggns::
Deve uppc mircsise nu 11ple was paplication using Jea and Spring franwary,
resutting
Fuing bart Hosbit reseting If 39h trcreas in cpp dwntlaa ard In ieroas ner
ieerease in selm wibeii serts
Lete cats aralytics platfore,
precessteg the
with cresi-factiee sare devlly maiis upliicstinn resiting
systee uptins.
cactug isclalitt,, realiting
radectior reppass tase and de iscreas 11
in Jecreaee 1n
polmesnatiin
torked clestly with de repieers peeatity and restive de oetects, reactttey in 39
cefect
Fatticleoo ad: rosive areviko todmaces palaenc to peir orginars, resuttug
feam prodettadry
Eacartio::
Caarter:
Pukeler
jiiversity Nan), (rraaatien batrl
tertstoatte::
Lertifiad Daa awoloper, tricle (211))
tartstine Jes8 doolicer, tricle cerseratior (artiiicatiin decal
upm repest.
repest.
Figure 5: RE example An illustrative example of modified responses. In this case, the task was to write a resume with the instruction to include three specific keywords. The original response only included one keyword, whereas the modified response, guided by the instruction-following direction, successfully incorporated all three keywords, demonstrating the effectiveness of RE in enhancing instruction adherence.
A.3 INSTRUCTION GENERALIZATION ON EXPANDED EXPERIMENT
In the main paper, we observed that models struggle to generalize across unseen instruction types, with AUC scores ranging from 0.50 to 0.55, which is close to random chance, as shown in Table [] and Table 3 of the main paper. One hypothesis for this poor generalization is the limited number of instruction types used in the initial experiments, where the linear probe was trained on just 4 instruction types. To further investigate this, we expanded the dataset to include 23 instruction types across 8 categories, each paired with 20 tasks.
Unlike the IFEval dataset, which contains 25 instruction types across categories, we omitted the 'combination' category, which includes the 'combination: Repeat Prompt' and 'combination: Two Responses' instruction types. This is because combined instructions can lead to conflicting signals in our analysis, where success in one instruction type but failure in another may produce mixed rep- resentations. By focusing on single instruction types, we aim to more clearly capture the represen- tations associated with catructtoon-oolowiin success and failure. In comparison to IFEval- simple, which features 5 instruction types across categories, this expanded dataset includes 23 instruction types across 8 categories, helping to prevent overfitting to small number of instructions.
The results from this expanded experiment, shown in Table 7 for different layers and Table g for different tokens, reveal that despite increasing the number of instruction types, the models still demonstrate limited generalization across unseen instruction types. The AUC scores remain close to chance levels, similar to the initial experiments. As shown in Table 7 and g the results indicate
16

---

Published as a conference paper at ICLR 2025
that adding more instruction types does not significantly improve instruction generalization. These findings reinforce the conclusion that models struggle to generalize istruction-followiin across different instruction types. This suggests that "global" instruction followiinn dimension, applicable across diverse instruction types, may not exist.
Models
2
Mistral-7-iinst-0...
Phi---mink- 128k
Instractions
Midddl
Last
Karly
Middle
Middle
Middle
0.70
0.61
0.57
0.67
0.54
0.52
0.55
0.62
0.59
0.60
0.46
0.48
keywards
01.48
detectable format
0.52
change_cane
denetahle.comien
language
AVERAGE
Table 7: Instruction-type generalization on FEval-siiple---ppanee across layers AUC scores across different models and instruction types from "'Eval-simpleexxpee The 'punctuation' in- struction type is marked with due to an insufficient number of data points caused by a low success rate, making impossible to compute reliable AUC scores.
LLaMel-daa--7
LLaMaa-chatt133
Pli-3-misi-1256
Rarly bikes
Middl taken
East lkken
Early baken
Middle token
East bakes
Early taken
Midile takes
Last Diken
Early tokon
Midile takes
Last taken
conest
0.30
0.51
0.56
0.90
0.30
krasweds
1::5
0.69
0.56
13.5%
0.32
0.10
0.42
0.90
0.9)
0.99
0:31
D:17
ddietable
penciratio
11:17
0.17
1109
0.9
BA1
0.11
0.1
044
0.36
0.12
0.50
0.1
0.99
0.75
0.76
040
046
Table Instruction-type generalization on IFEeal-simple--xppade across tokens AUC scores
across early, middle, and late token representations, showing instruction-type generalization per- formance on IFFyal-simple-expandee The results indicate that despite expanding the number of instruction types, models continue to struggle with unseen instruction types, with scores close to chance levels across different token positions. The 'punctuation' instruction type is marked with due an insufficient number of data points caused by a low success rate, making it impossible to compute reliable AUC scores.
A.4 SUCCESS RATE
This section presents the success rate for Instruction-followiin which measures the accuracy of responses adhering instructions. The success rates for the IFEval dataset(Zhou et al.) 2023 shown in Table for our IFFivalssipple dataset Table and for IFEEal--implle-exeneee in Table
II which is used in Section of the Appendix. The IFEval dataset consists of 25 instruction types categorized under 9 broader categories, with approximately 20 tasks per instruction type. For details on IFEval and IFEval-simple, please refer Section 2.1 of the main paper. We use the success rate (loose) metric from Zhou et al. (2023). To ensure consistent results without randomness in decoding, we used greedy decoding without sampling when calculating the success rate.
IFEval inst
LLaMa2-chat-77
LLaMa2-chat-155
Mistral- -isttal-B-innttvO.3 Phi-3-mini-
change case
0.48
0.52
0.62
0.29
detectable_content
0.85
0.89
0.79
0.89
detectable_forg format
0.66
0.68
0.78
0.67
keywords
0.68
0.71
0.73
0.75
language
0.68
0.58
0.87
0.97
length constraints
0.46
0.48
0.55
0.41
punctuation
0.24
0.14
0.17
0.11
startend
combination
0.24
0.22
0.17
0.22
Table 9: Success rate the IFEva Zhou al.
2023
across 9 categories of instruction types
17

---

Published as a conference paper at ICLR 2025
IvEll inst
LLaMa2-chat-7b
LLaMa2-chut-33
Mistral-7B-ins-ee
Phi-3-minn--122
key deywords:exiseence
0.79412
0,87255
0.86275
0,94118
keywordsforbidddc..oo words
0.18627
0.28431
0,36275
0,32353
keywords.frequency
0.86275
0,92157
0.91176
1.0000
startendeend_cheee _chckke
01235229
0.16667
0.27451
0.13725
delectable delectable_commenntin haaleeettmummbbe.pppetthheer
0.76471
0.80392
0.5098
0.87255
Table 10: Success rate [FEval-simple across instruction types under 3 categories
IFEval inst
I.LaMa2-chat-7
Mistral-7B-inst-v0..3
Phi-3-mini-1
change.com case
0.53
0.70
0.46
0.31
detectable content
0.65
0.90
0.75
0.94
Setectuble_formal
0.67
0.72
0.72
0.64
keywords
0.91
language
0.40
0.10
0.94
0.83
length_ constraints
punctuation 0.53
0.56
0.69
0.40
startend
0.98
0.93
0.69
0.28
Table 11: Success rate IFFrxal-iimlle-exteneed across categories of instruction types (exclud-
ing the 'combination" category)
A.5 PCA ACROSS ALL FIVE INSTRUCTION TYPES
In this section, we extend the PCA analysis to include all five instruction types used in our experi- ments. This analysis contrasts with the PCA plot in Figure the main paper, where we focus on three instruction types within the keyword category. In the main paper, the PCA plot show clear tendency towards separability of the instruction-follwwee data points were not perfectly linearly separable. However, in this extended analysis with all five instruction types in Figure 6 the representations are less linearly separable the 2-dimensional PCA plot. This highlights that different instruction types (or categories) may exhibit distinct ge- ometries in the representation space. The lack of clear separability further supports our findings in the main paper that linear probes trained unseen instruction types in Se.tion[..3] This suggests that there no "global" instruction following dimension that can be applied across different types of instructions, likely due the varying internal geometries of these categories.
dimension across tasks, even though the
set of instruction types struggle to generalize to
(a) Llama-2-13b-chat-a 1-bbhchhtthhf
(b)
(c) Mistral-7B IB-Inst-00.3
(d) Phi-3-128k-inst
Figure 6: PCA plot of representations from four LLMs across all five instruction types. This PCA plot of first-token representations from early layers shows that the inclusion of all five instruc- tion types results in less separability compared to the three instruction types in the main paper in Figure 2 This indicates that different instruction types possess distinct geometries, supporting the conclusion that linear probes do not generalize well to unseen instruction types.
A.6 WIIY DO WE CHOOSE IFEVAL DATASET?
Here, we would like to emphasize why we choose IFEval as our primary dataset instead of using real-world dataset with different contexts and domains.
First, we select IFEval to focus on our scope which is 'single, simple, and non-ambiguous instruc- tions'. Real-world datasets often involve complex, ambiguous, multi-instruction prompts, which
18

---

Published as a conference paper at ICLR 2025
conflate multiple factors affecting cnttructtio-following As initial exploration of the ge- ometry of LLM representations in instruction cetrructio-foolooingg we chose to focus on single, simple, and verifiable instructions ensure clarity and disentangle multiple factors. The IFEval dataset is well- suited for this purpose, as it provides 25 distinct types of simple and clear instructions that align with our goal of establishing robust baseline.
Second, we want to avoid evaluator-induced uncertainties. Most real-world tasks and benchmark datasets rely on LLM-based evaluators to determine whether a response follows an instruction. However, LLM-based evaluators may introduce their own uncertainties or make errors in assess- ing success or failure, which could obscure our analysis on representations of the tested models. The IFEval dataset avoids this issue by including instructions with deterministic evaluation pro- grams that objectively verify compliance. For instance, an instruction like "please do not include keywords: ..." can be automatically validated using a simple program to check for the presence of those keywords. This feature eliminates ambiguity in evaluation and allows us to isolate the directions related specifically to instruction-following
One of our main contribution is the careful design of data settings specifically tailored to analyze internal states of LLMs in instruction-followe point for this research, we hope our work inspires future efforts to tackle analysis of LLMs in more complex, real- world instruction-followe tasks.
contexts. While IFEval serves as an ideal starting
A.7 REVERSE REPRESENTATION ENGINEERING
We conducted initial experiments on reverse representation engineering with two models: Phi-3- mini- 128k and Mistral-7B-inst--bl In these tests, we try move representations towards the failure class by flipping the adjustment vector -a D)
Model Original SR
Random SR Reverse Inst-follow SR
Mistral
0.58 = 0.00
0.56 # 0.02
0,54 + 0.01
Phi 0.71 0.00
0.63 0.04
0.60 0.02
Table 12: Success rates for various models under different settings.
Notably, we set the values conservatively to keep the quality ratio (QR) of reverse RE remains similar that random directions (0.86 for Mistral and 0.77 for Phi). The results indicate that the success rate (SR) for reverse RE is worse than random directions, expected, but the difference is not significant. We anticipate that finding on a validation set will amplify the difference between reverse and random directions. We plan to conduct additional experiments to refine a and better evaluate the effectiveness of reverse RE disrupting instruction adherence.
19


Reviews:
[{"review_id": "UW8hslAs4x", "reviewer": "Reviewer_MsZm", "summary": "This paper investigates the internal representation of LLMs and identifies a \"dimension\" that corresponds to their success in instrution-following. They show that manipulating this dimension could improve success in instruction-following without degrading the quality of the outputs. They also measure correlation of the dimension with different perturbations and show that it is more sensitive to the phrasing of the prompt than the task difficiulty or faimiliarity. \n\n'", "questions": "- See weaknesses\n- What is the motivation behind the update in line 303-304? Could you provide more intuition on what is being done to the input representation there?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Paper offers very interesting and operationalizable insights about the internal representations of LLMs.\n* Paper tells a very good, satisfying story: from identifying the dimension, to manipulating and interpreting it.", "weaknesses": "- The methodology description lacks clarity due to insufficient mathematical notation. It is especially difficult to follow the procedures outlined in Sections 3 and 4. The authors should clearly define mathematical objects, such as \"input representation,\" and provide precise equations for the computations (e.g., lines 370-374). Specifically, given a prompt \\( x_1, x_2, ..., x_n \\) of n words and an output \\( y_1, y_2, ..., y_m \\) of  m words, what is the \"input representation\" as a function of these tokens? It appears to be computed solely from the output words.\n\n* The terminology is confusing. The term \"instruction\" would be more accurately described as \"constraints.\" It’s challenging to distinguish \"instruction\" from \"task,\" as \"instruction\" could imply the entire command given to the model. Additionally, the meaning of \"dimension\" is unclear. Does it refer to a hypothetical concept rather than an actual dimension of a hidden vector computed by the model?\n\n* The experiments were conducted on a single, relatively small dataset, raising questions about whether the findings will generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJ3SuMf5cP", "reviewer": "Reviewer_mAEy", "summary": "This work analyzes and compares LLM internal representations when it can successfully follow instructions, vs. when it fails at following instructions. By training linear probes on representations, authors find a latent direction that predicts instruction following success, and shows it generalizes to unseen tasks and instructions. In addition, this work uses representation engineering to make adjustments along the instruction following dimension they found, and finds these adjustments can improve instruction following performance. Finally, this work also finds that the instruction following dimension is more correlated with how the prompt is phrased, rather than the difficulty of the instruction or the task.", "questions": "- At the start of section 2, it would be helpful to elaborate what the IFEval dataset is used for \n- Make text in Figure 1 bigger", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is well-structured and the findings are very interesting - while using linear probes to find a latent dimension for a specific phenomenon is not that interesting or original, the follow-up analyses demonstrating that instruction following can be made more accurate with representation engineering, and comparing the correlation of this dimension to different properties of the prompts, are both interesting. In addition, most claims made in the paper are sufficiently backed up with experiments and evidence.", "weaknesses": "In section 3, I'm confused what the direction D taken is for different instructions, since the previous section states that linear probes do not generalize well across instructions. Does this mean representation engineering promotes different directions for different instructions, or is there a universal direction that enhances all instructions generally? It would be helpful to clarify this in the paper.\nFor section 4 as well, one instruction type is chosen to perform the analysis on correlations. Could you elaborate on how much the findings in this section generalizes to other instruction types? Or could there be instruction types whose representation is more correlated with task familiarity rather than phrasing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Pbk0rl4is", "reviewer": "Reviewer_wmTS", "summary": "This work finds that the representation of token is related to the success rate of instruction following. The authors apply linear probing to identify a specific dimension and use representation engineering (RE) to manipulate the representations. Experimental results show that RE  improves success rate while maintaining quality. Further analysis reveals that phrasing modifications plays a critical role rather than task familiarity or instruction difficulty. This paper provides a deeper understanding to this field.", "questions": "1. Have you tried reverse representation engineering? Will it be worse than random?\n2. Why did you focus on the representation of the first token in the last layer? Any insight?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides a new understanding of instruction following, using token representation to interprete LLMs themselves.\n2. The experiments are well designed and the writing is easy to follow.", "weaknesses": "1. Some of the conclusions in this paper are as expected, e.g. results from Table 1 and Table 2 are related to \"Lost in the Middle\" phenomenon. [1]\n2. The performance improvement brought by this method is limited as shown in Table 4.\n3. The IFEval dataset is small and such problem may be addressed by extending instruction following data.\n\n[1] Lost in the Middle: How Language Models Use Long Contexts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTkMieDbq0", "reviewer": "Reviewer_UuRY", "summary": "The authors expand prior work on finding meaningful directions in the latent space of transformer language models to find a direction associated with whether models follow a given instruction or not. \n- For the analysis, they adapt and expand an existing dataset and study four open-source models (7b to 13b parameters). \n- Using linear probes, the authors find directions associated with instruction following across the studied tasks, but not generalizing to hold-out instruction types. \n- They show that manipulating the latent representation along the found direction can improve instruction following success and maintain instruction tuning quality for already successful cases.\n- The authors find that the found latent space direction is affected by prompt sensitivity/phrasing of the instructions (less so by task complexity and perplexity)", "questions": "Generally, please just respond to my weaknesses above. I'm willing to increase the scores (total, contribution, and soundness), especially if you address the crucial ones related the generalizability and usability with new experiments or strong arguments. \n\nQ1: Can you explain where the \"middle\" token was for the different prompts? Was it the middle token of each sequence and thus changing the position for each prompt?\n\n### Comments (Not considered for the review)\n\n- For Figure 2, I feel LDA (fitted on training data) might be a better fit than PCA to show separability, given that LDA actively tries to separate the known classes.\n- In lines 213-214, you state that the instruction type generalization values are close to chance. Given the stated 1-sigma uncertainties in Table 1, the presented values are not statistically significantly different (depending on the definition, 2-5 sigma) from random chance.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The goal of finding a way to manipulate latent representations to improve the instruction following of language models (or detect whether instructions are not being followed) has the potential for significant usability and safety impact while being novel. \n- The paper is very well written and the presentation is generally clear.\n- The presented methodology is well structured and each section is clearly motivated by the previous results.\n- The authors test four open-source models across many different tasks, demonstrating a certain amount of robustness.\n- The authors generally do not overclaim their results and are very transparent.", "weaknesses": "The authors do not claim that they find a universal direction in these models that fully represents the idea of instruction following. However, to make a significant contribution via usability and understanding of latent representations, this work would benefit from addressing a few weaknesses to enable more robust future work and enhance the usability of the results.\n \n### Weakness 1\n\nThe studied tasks are somewhat general but the instructions are fairly simple and not necessarily representative of general language model uses. I wonder how well the results of this work might generalize: It is also unclear how well instructions are being followed in a general use case. To use the example of the paper: You might ask a health bot to be aware of your left knee injury when coming up with training plans, but does adding the found latent space direction have side effects like avoiding leg training altogether or creating asymmetric training plans leading to other issues?  The results in this paper could be significantly strengthened by finding applications that allow for a more nuanced analysis of how instructions are being followed after representation engineering to demonstrate the usefulness of their technique. \n\n### Weakness 2\n\nIn Sec. 3, I think an opportunity was missed to strengthen the analysis by demonstrating that the found direction also works in a counterfactual way: Can you use the fitted alpha values and directions for the F2T and T2T tests and only flip the sign of the respective alpha value to check if it also works on (intentional) T2F and F2F? I think this is necessary to demonstrate a minimum of reliable usability and support the claim of safety impact (to potentially get a model to not comply).\n\n### Weakness 3\n\nIt is unclear (to me) if using only one direction is sufficient for modeling whether an instruction is being followed or not and even if, whether the presented analysis finds that direction. For example, the prompt sensitivity studies in Sec. 4 affecting the performance could be caused by other meaningful \"directions\" being entangled in the one you care about. To be more precise: If changing \"write a resume for [a] software engineer [...]\" to \"I want you to write about [a] software engineer resume [...]\" makes a difference, how do we know this is not caused by, e.g., confounding the learned directions with the instruction data set with a \"dialog in first-person\" direction (not unlikely,  given the colorful interpretations modern SAE approaches find).\n\nTo link it to another, more concrete use case: Is it possible to circumvent safety training of a language model using the proposed methodology? Although also only necessary and not sufficient, such a more complex example would demonstrate that using only one direction is powerful enough to be useful. \n\nIf one direction is not sufficient, it might be worth testing if a higher-order approach (2D, 3D, ...) might be more robust to find that direction. There are plenty and different works on this, but for example, [1] uses PCA on latent representations to find the one relevant direction to model how small backdoored language models process trigger inputs to switch to toxic text.\n\n[1] Lamparth, M. and Reuel, A. Analyzing and editing inner mechanisms of backdoored language models. arxiv:2302.12461, 2023.\n\n### Weakness 4\n\nIn Sec. 3, fitting an alpha for each model and instruction type greatly limits usability, especially given that you would need a cleanly separate dataset for all relevant tasks. This is also more difficult for more realistic tasks mentioned in Weakness 1. \n\n### Weakness 5\n\nThis is a minor weakness that did not impact the final review score, but I wanted to explain why I rewarded a 3 instead of the maximum score of 4 for the presentation. I would have rewarded a 4 if figures 2 and 4 had used a larger font size for the axis labels, plot legend, and larger marker sizes. Please make these plots more readable for future versions of this work. \n\n### Weakness 6\n\nIn Sec. 3, it is not clear whether and how the authors calibrated the GPT-4-based scoring from 0 to 9 to construct the quality ratio. The results in Sec. 3 could therefore be strengthened by stating how the GPT-4-based analysis is being conducted (how exactly is a quality rating of 4 different than a quality rating of 5, in-context examples or not, ...) and how accurate and reliable it is (True positive rate, false positive rate, ...). \n \n### Weakness 7 \n\nThe stated uncertainties seem very low, especially in Tables 3 and 4 (negligible/0 uncertainties). I think the claims in this work could be strengthened if other methods, e.g., bootstrap resampling, are being used to determine a base uncertainty beyond uncertainties purely determined from the linear probes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJFcxbCh4C", "reviewer": "Reviewer_jGSV", "summary": "The authors explore whether LLMs have an internal representation related to successful instruction following. They train a linear probe on the models' internal representations to predict success/failure cases. This probe works well when tested on held-out tasks. However, when tested on held-out instructions, the probe fails, suggesting that the representation found is not a general \"instruction-following\" representation.\nPotential practical use: Adding this identified direction to a model’s internal representation improves instruction following -- although it is unclear how far this generalizes.", "questions": "Define “know internally”. The title and various parts of the paper reference this. After reading, I understand it as “encoded instructions in a way correlated to correct responses”. But perhaps the authors disagree with this definition. Defining it specifically would clarify this.\n\nFrom above: The authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types. So the internal representation seems an encoding of the specific instruction type, rather than a general knowledge of instruction following. Clarifying this earlier would be useful. Readers skimming the paper may have a wrong impression.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper demonstrates a practical use case of their findings. The instruction-following direction improves model instruction-following success. This can be a useful case way to steer models to follow instructions better.\n\nAlso, the method could help us detect when a LLM will not follow a particular instruction. We can detect this even before the LLMs generate a response — because the method works on the first response token.\n\nThis is step towards scientifically understand key processes in how LLMs complete practical tasks\n\nThe paper reproduces results across different models. The paper conducts experiments across multiple models (Llama, Mistral, and Phi-3) and shows strong results on all 3 models.", "weaknesses": "To strengthen the claim of usefulness, compare your method to other interventions e.g. few-shot prompting. Do we observe a similar increase in success rate by just few-shot prompting? In general, the claim of practical usefulness could be investigated in much more detail (with more baselines and tests of generalization to other datasets). \n\n You write: “This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged.” You then point to properties of the instruction-following dimension (sensitivity to phrasing modification). I don’t see how this helps *explain* why models sometimes fail to follow clear instructions or why prompt engineering helps.\n\nThe authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types.\n\nMy impression is that more recent LLMs (e.g. the latest from OpenAI, Anthropic, Meta) are much more reliable at instruction-following and less sensitive to prompts than the models tested here. This is presumably not due to exploiting the instruction-following direction. More generally, it is unclear whether the insights gained from these weaker models would carry over to more recent models. (I’m not expecting the authors to conduct white-box experiments with much larger open models such as Llama-3-405b but it might be possible to gain evidence on this question in other ways). \n\nSmaller points:\n1. “we demonstrate that this dimension generalizes to unseen tasks, indicating that it captures a fundamental aspect of instruction adherence in LLMs.” This claim is ambiguous. What level of generalization performance justified something being a “fundamental aspect”? It would be helpful to make more concrete and falsifiable claims. \n\n2. I find the claim related to Fig 4 questionable. The authors claim that phrasing plays a critical role in accuracy because the phrasing modification direction aligns more with the instruction-following direction. However, they don’t empirically demonstrate that the phrasing changes affect success rates more than the other factors. To make the argument stronger, consider measuring the impact of each factor. How much does the success rate change when modifying phrasing vs difficulty vs task type?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)