Do LLMs ``know'' internally when they follow instructions?

## Abstract

Abstract Instruction-following is crucial for building AI agents with large language models (LLMs), as these models must adhere strictly to user-provided constraints and guidelines.
However, LLMs often fail to follow even simple and clear instructions.
To improve instruction-following behavior and prevent undesirable outputs, a deeper understanding of how LLMs’ internal states relate to these outcomes is required.
Our analysis of LLM internal states reveal a dimension in the input embedding space linked to successful instruction-following.
We demonstrate that modifying representations along this dimension improves instruction-following success rates compared to random changes, without compromising response quality.
Further investigation reveals that this dimension is more closely related to the phrasing of prompts rather than the inherent difficulty of the task or instructions.
This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged.
This work provides insight into the internal workings of LLMs’ instruction-following, paving the way for reliable LLM agents. 1 1 1 We will release the data and code on GitHub upon publication.

## 1 Introduction

Given the potential of large language models (LLMs), there has been significant interest in utilizing these models to build personal AI agents. For instance, one could imagine deploying an LLM as a personal healthcare assistant, such as a fitness or nutrition planner, or for psychological counseling (Li et al., ; Wang et al., ; Tu et al., ). Compared to traditional machine learning-based AI agents, LLMs offer the advantage of being easily adaptable through prompting, allowing users to provide guidelines and personal information without the need to retrain model weights.

Instruction-following is critical in the development of personal AI agents with LLMs through prompts because these models must adhere to the constraints and guidelines to ensure safe and trustworthy interactions. For example, suppose an LLM is building a personal fitness plan for a user with knee problems. To avoid knee problems for the user, the LLM must follow the instruction of not recommending knee-intensive movements or any exercises that could lead to potential injury.
Similarly, in a nutrition planner, the LLM should avoid generating harmful recommendations, such as suggesting inappropriate food for pregnant women or children with diabetes.

Figure: Figure 1: Overview of our paper. Left: Success and failure cases in a personalized AI fitness planner. The task is to generate a warm-up plan while avoiding knee-required positions. The success case follows the instruction, while the failure case violates it. Middle: Linear probing is applied to analyze internal representations from success and failure cases, identifying the instruction-following dimension. The probe is tested on unseen tasks (e.g., writing a CV) and instruction types (e.g., include/exclude keywords). Right: Representation engineering is used to shift failure cases into success by adjusting the representations along the instruction-following dimension, improving adherence without compromising task quality.
Refer to caption: x1.png

However, LLMs often fail to follow even non-ambiguous and simple instructions (Zhou et al., ; Qin et al., ; Xia et al., ; Kim et al., ; Yan et al., ) like including keywords or following formatting guidelines. GPT-4 achieves around an 80% success rate on IFEval (Zhou et al., ), an instruction-following benchmark dataset, while smaller models have success rates around 30% to 40%.
This raises the question: why do LLMs fail to follow instructions, even when those instructions are clear and familiar?

To gain a better understanding of instruction-following outcomes, we analyze the internal state of LLMs, focusing on the differences in representations between success and failure cases of instruction-following across different tokens and layers.
Our approach involves disentangling the effects of tasks and instructions in input prompts, where the instruction specifies the action (e.g., ‘please do not use keywords’) and the task provides the context for executing the instruction (e.g., ‘please write a resume’).
By applying linear probing—a widely used method for interpreting model representations (Alain & Bengio, ; Belinkov, ; Elazar et al., )—we identify a specific dimension within the input embedding space that is strongly associated with instruction-following. While previous work has primarily used linear probing to explore representations related to truthfulness and reducing hallucinations (Azaria & Mitchell, ; Marks & Tegmark, ; MacDiarmid et al., ), our study extends this method to investigate instruction-following. We demonstrate that this dimension generalizes to unseen tasks, indicating that it captures a fundamental aspect of instruction adherence in LLMs.

To validate the significance of the instruction-following dimension, we applied representation engineering techniques to enforce instruction-following based on insights from our linear probes.
Our experiments show that adjustments along this specific dimension are more effective in enhancing instruction-following success rates than random modifications, while maintaining the overall quality of the generated responses. These results indicate that the instruction-following dimension plays a crucial role in shaping the model’s behavior, toward better adherence to instructions.

To further interpret the meaning of this dimension, we conduct a sensitivity analysis based on three key perturbations to the input prompt: task familiarity, instruction difficulty, and phrasing.
Our findings reveal that this dimension is more related to the rephrasing of prompts rather than the inherent difficulty of the task or instructions.
This suggest that the way a prompt is encoded within the model’s input representation space plays a significant role in whether the instruction is followed correctly. This observation not only provides a deeper understanding of why LLMs sometimes fail to adhere to straightforward instructions but also offers an explanation for the effectiveness of prompt engineering, even when the content of the prompt remains largely unchanged.

Overall, this work sheds light on the underlying mechanisms of instruction-following in LLMs by uncovering a critical dimension in the model’s representation space. These insights enhance our understanding of LLM behavior and offer practical approaches to improving instruction adherence, bringing us closer to developing more reliable and trustworthy AI agents.

### 1.1 Contributions

- •
We identify a specific dimension within the input embeddings space of LLMs that is closely linked to instruction-following, using linear probes, by carefully designing our setting to disentangle the effects of tasks and instructions in input prompts.
- •
We demonstrate that this dimension generalizes to unseen tasks and that modifying representations along this dimension effectively converts instruction-following failures into successes without compromising response quality.
- •
Through a sensitivity analysis, our findings reveal that this dimension is linked to how prompts are rephrased, underscoring that instruction-following in LLMs is influenced by how prompts are encoded within the model’s input embeddings. This explains why LLMs sometimes fail to follow clear, simple instructions and why prompt engineering can enhance instruction adherence, even when the content remains largely unchanged.

## 2 Do LLMs know when they succeed or fail to follow instructions?

In this section, we aim to identify the dimension within the models’ representation space that is closely associated with instruction-following. We use linear probes to determine the internal signals that separate successful instruction-following from failures and examine whether this dimension generalizes to different tasks and instruction types. By exploring different tokens and layers within the models, we seek to understand how and when instruction-following information is encoded.

### 2.1 IFEval-simple

First, we selected the IFEval dataset (Zhou et al., ) as a base, due to its objective evaluations with verifiable instructions, thereby minimizing uncertainties from ambiguous evaluation criteria.
The IFEval dataset comprises 25 instruction types under 9 categories, with each instruction type paired with a distinct set of tasks — approximately 20 tasks per instruction type.
Because of the relatively small number of tasks per instruction type, internal model states resulting from these prompts contain a mix of both instruction-following and task-specific details.

To isolate the dimension related specifically to instruction-following, we generated a modified version of the IFEval data, called IFEval-simple.(^2^22We will release the IFEval-simple data along with code on GitHub upon publication.)
First, we selected 5 instruction types that are likely to be used in real-world applications for AI agents. For example, ensuring the inclusion (keywords:existence) or exclusion (keywords:forbidden) of specific keywords, specifying the frequency of certain keywords (keywords:frequency), generating responses with placeholders (detectable_content:place_holders), and requiring responses to end with predefined sentences (startend:end checker).
We excluded more complex or impractical instructions, such as those requiring omission of punctuation, as they are less relevant for practical use cases.

Second, we generated 100 tasks using GPT-4, similar to the original tasks in IFEval, where each instruction type is paired with the same set of 100 tasks.
By pairing each instruction type with the same set of 100 tasks, we ensure that linear probes trained on the model’s representations are more likely to capture information solely related to instruction-following, without the confounding influence of varying tasks.
The instructions assigned to each task vary in detail based on the context. For example, for an instruction type focused on keyword inclusion or exclusion, a resume-writing task might require keywords like ‘skills’ and ‘career’, while a joke about a programmer might involve terms like ‘syntax’ or ‘code’. These variations introduce diverse challenges, testing the model’s adaptability in following instructions. Example tasks are provided in Appendix Table 5 and Table 6. The instruction-following accuracy for IFEval-simple datasets is presented in Appendix Table 11.

### 2.2 Methods

Representations
We analyzed four language models: LLaMA-2-7B-chat (Touvron et al., ), LLaMA-2-13B-chat (Touvron et al., ), Mistral-7B-Instruct-v0.3 (Jiang et al., ), and Phi-3-mini-128k-instruct (Abdin et al., ). For each model, we looked at the representations between tokens – the first, middle, and last tokens, representing the LLMs before, during, and after they generate responses. We also examined three layers (early, middle, last) to identify where instruction-following information is encoded within the models’ internal state. Specifically, we used layers 16, 32, and 40 and for LLaMA-2-13B-chat and 14, 26, and 32 for other three models. To avoid randomness in decoding, we employed greedy decoding without sampling.

Linear Probes
We trained linear probes on the representations to identify the instruction-following dimension. A simple linear model was trained on instruction-following success outcome, optimized for 1000 epochs with AdamW, a 0.001 learning rate, and 0.1 weight decay.

Train-test split and metric
We assessed task generalization and instruction-type generalization by splitting the data into training and testing sets, as shown in Figure 1.
IFEval-simple has 5 instruction types, each paired with the same set of 100 tasks.
To evaluate task generalization, we split the data by the task dimension, using a 70-30 train-test split across the 100 tasks. To evaluate instruction-type generalization, we applied a leave-one-out approach, over the instruction-type dimension.
To evaluate performance, we use the Area Under the Receiver Operating Characteristic Curve (AUC)(Pedregosa et al., ), assessing the accuracy of binary predictions for each model on unseen tasks and instruction types.

### 2.3 Results

**Table 1: Task and instruction-type generalization AUROC scores for task and instruction-type generalization using a 70-30 train-test split for task generalization on unseen tasks, and leave-one-out cross-validation for instruction-type generalization across different instruction types. Standard deviation is calculated from five runs with different random seeds for task generalization and across instruction types for instruction-type generalization.**
|  | Task generalization | Instruction-type generalization |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | First token | Middle token | Last token | First token | Middle token | Last token |
| LLaMA-2-chat-7B (14 lyr) | 0.77 $\pm$ 0.04 | 0.55 $\pm$ 0.07 | 0.73 $\pm$ 0.04 | 0.52 $\pm$ 0.03 | 0.50 $\pm$ 0.07 | 0.52 $\pm$ 0.05 |
| LLaMA-2-chat-13B (16 lyr) | 0.83 $\pm$ 0.03 | 0.58 $\pm$ 0.06 | 0.82 $\pm$ 0.03 | 0.56 $\pm$ 0.06 | 0.58 $\pm$ 0.06 | 0.53 $\pm$ 0.03 |
| Mistral-7B-inst-v0.3 (14 lyr) | 0.74 $\pm$ 0.02 | 0.54 $\pm$ 0.05 | 0.72 $\pm$ 0.04 | 0.50 $\pm$ 0.05 | 0.51 $\pm$ 0.05 | 0.51 $\pm$ 0.05 |
| Phi-3-mini-128k (14 lyr) | 0.88 $\pm$ 0.03 | 0.56 $\pm$ 0.04 | 0.86 $\pm$ 0.03 | 0.55 $\pm$ 0.04 | 0.48 $\pm$ 0.03 | 0.50 $\pm$ 0.03 |

**Table 2: Task generalization (detailed across layers) AUROC scores for the first, middle, and last tokens across early, middle, and last layers of various models. The layers selected for LLaMA-2-13B-chat are 16, 32, and 40, while for the other three models, the layers used are 14, 26, and 32.**
|  | Early layers | Middle layers | Last layers |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | First token | Middle token | Last token | First token | Middle token | Last token | First token | Middle token | Last token |
| LLaMA-2-chat-7B | 0.77 $\pm$ 0.04 | 0.55 $\pm$ 0.07 | 0.73 $\pm$ 0.04 | 0.75 $\pm$ 0.05 | 0.51 $\pm$ 0.04 | 0.76 $\pm$ 0.04 | 0.73 $\pm$ 0.03 | 0.54 $\pm$ 0.02 | 0.70 $\pm$ 0.02 |
| LLaMA-2-chat-13B | 0.83 $\pm$ 0.03 | 0.58 $\pm$ 0.06 | 0.82 $\pm$ 0.03 | 0.81 $\pm$ 0.02 | 0.56 $\pm$ 0.05 | 0.80 $\pm$ 0.04 | 0.78 $\pm$ 0.04 | 0.49 $\pm$ 0.03 | 0.79 $\pm$ 0.05 |
| Mistral-7B-inst-v0.3 | 0.74 $\pm$ 0.02 | 0.54 $\pm$ 0.05 | 0.72 $\pm$ 0.04 | 0.71 $\pm$ 0.05 | 0.51 $\pm$ 0.03 | 0.67 $\pm$ 0.04 | 0.71 $\pm$ 0.03 | 0.49 $\pm$ 0.04 | 0.70 $\pm$ 0.03 |
| Phi-3-mini-128k | 0.88 $\pm$ 0.03 | 0.56 $\pm$ 0.04 | 0.86 $\pm$ 0.03 | 0.85 $\pm$ 0.03 | 0.56 $\pm$ 0.03 | 0.83 $\pm$ 0.02 | 0.65 $\pm$ 0.05 | 0.53 $\pm$ 0.03 | 0.63 $\pm$ 0.04 |

**Table 3: Instruction-type generalization (detailed) AUROC across different models and selected layers on first token representations. A leave-one-out approach was employed, and the standard deviation from training a linear probe is small enough to be omitted from the table. The ‘-’ mark in ‘keywords:frequency’ instruction type is due to an insufficient number of data points caused by a 100% success rate, making it impossible to compute reliable AUC scores.**
| Instructions | LLaMA-2-chat-7B | LLaMA-2-chat-13B | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr |
| key:forbidden | 0.52 | 0.51 | 0.56 | 0.45 | 0.45 | 0.44 | 0.44 | 0.41 | 0.46 | 0.52 | 0.54 | 0.53 |
| key:exist | 0.50 | 0.50 | 0.51 | 0.67 | 0.68 | 0.66 | 0.55 | 0.50 | 0.50 | 0.63 | 0.67 | 0.68 |
| key:freq | 0.57 | 0.59 | 0.59 | 0.57 | 0.57 | 0.57 | 0.56 | 0.56 | 0.56 | - | - | - |
| number_placeholders | 0.56 | 0.54 | 0.52 | 0.58 | 0.58 | 0.54 | 0.50 | 0.49 | 0.50 | 0.50 | 0.53 | 0.46 |
| end_checker | 0.48 | 0.46 | 0.47 | 0.55 | 0.57 | 0.56 | 0.44 | 0.42 | 0.45 | 0.55 | 0.59 | 0.57 |
| AVERAGE | 0.52 | 0.52 | 0.53 | 0.56 | 0.57 | 0.55 | 0.50 | 0.48 | 0.49 | 0.55 | 0.58 | 0.56 |

Linear probes generalize across unseen tasks
The task generalization results in Table 1 show that linear probes performed well across different tasks when the instruction type remains consistent. The AUROC scores, which range from 0.7 to 0.8 using the first token, suggest that the input embeddings of these models possess a shared geometry related to instruction-following that generalizes well across varied tasks. This is particularly beneficial in the context of buliding AI agents, where a pre-defined consistent set of instructions needs to be followed across different tasks. For example, if a probe is trained on examples of an instruction type like “Please do not include these keywords” using examples from resume writing and nutrition coaching, the linear probe can predict if the model follows the same instructions type even unseen tasks, such as creating a warm-up plan without knee-intensive exercises.
Additionally, we plot the principal components analysis (PCA) using representations from the first token and early layers, fitting the PCA on the training split and visualizing the results on the test split (unseen tasks) in Figure 2. They show clear separability, supporting the idea that the instruction-following dimension is consistently represented across different tasks. Further PCA analysis is provided in Figure 7 in the Appendix.

Linear probes do not generalize across unseen instruction types
In contrast to task generalization, the models exhibit no clear generalization when tested across unseen instruction types. The AUROC scores for instruction-type generalization are notably lower, ranging from 0.50 to 0.55, close to chance (Table 1).
A potential explanation for this poor generalization could be the limited number of instruction types used during training, where the linear probe was trained on just 4 instruction types. To investigate, we expanded the dataset to include 25 instruction types, each paired with 20 tasks. However, as shown in Appendix in Table 8, this expanded experiment yielded similar results, with models still failing to generalize well across unseen instruction types.
This indicates that models struggle to generalize instruction-following across different instruction types, implying the absence of a ‘global’ instruction-following dimension that can be leveraged regardless of the instruction type, which may be due to varying representation geometries.

First token is as informative as last token
Interestingly, the first and last tokens—representing the model’s state before and after response generation—show high AUROC scores, implying that LLMs may already “know” whether they will follow instructions even before they start generating their responses. This early indication of instruction following is valuable, since early intervention or correction could be applied. In contrast, the middle tokens showed lower AUROC scores, likely because the representation contains information about next token generation more than information about instruction-following.

Layer-wise performance is similar, with early layers slightly better for task generalization
The performance across different layers shows only slight variations, with early layers marginally outperforming middle and last layers, as detailed in Table 2.
For example, in the 13B model, the early layers achieve an AUROC of 0.83 for the early token, which is slightly better than the performance of middle and last layers.
This suggests that the instruction-following dimension may be more prominently represented in the earlier stages of the model’s processing. However, for instruction-type generalization, there is no clear pattern across layers (Table 3), indicating that the challenges associated with generalizing across different instruction types are pervasive throughout layers.

Figure: (a) Llama-2-13b-chat-hf
Refer to caption: x2.png

## 3 Representation Engineering

**Table 4: Representation Engineering results on the last layer across four models. Success rate (SR) for instruction-following and quality ratio (QR) for task quality are compared across the original outputs, outputs using the instruction-following dimension, and outputs using a random directions. RE along the instruction-following dimension improves SR while maintaining or enhancing QR, unlike random adjustments which often reduce both SR and QR. Standard deviations are across three runs with different random seeds.**
| Model | Original SR | Random SR | Inst-follow SR | Original QR | Random QR | Inst-follow QR |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMA-2-chat-7B | 0.57 $\pm$ 0.00 | 0.55 $\pm$ 0.00 | 0.59 $\pm$ 0.00 | 0.87 $\pm$ 0.09 | 0.85 $\pm$ 0.10 | 0.87 $\pm$ 0.08 |
| LLaMA-2-chat-13B | 0.61 $\pm$ 0.00 | 0.54 $\pm$ 0.12 | 0.65 $\pm$ 0.02 | 0.92 $\pm$ 0.00 | 0.91 $\pm$ 0.02 | 0.94 $\pm$ 0.00 |
| Mistral-7B-inst-v0.3 | 0.58 $\pm$ 0.00 | 0.56 $\pm$ 0.02 | 0.64 $\pm$ 0.02 | 0.95 $\pm$ 0.02 | 0.86 $\pm$ 0.02 | 0.98 $\pm$ 0.06 |
| Phi-3-mini-128k | 0.71 $\pm$ 0.00 | 0.63 $\pm$ 0.04 | 0.74 $\pm$ 0.01 | 0.76 $\pm$ 0.01 | 0.76 $\pm$ 0.01 | 0.78 $\pm$ 0.00 |

Figure: Figure 3: Transition metric for Representation Engineering on the last layer of four models Success rate (SR) only on high quality responses in task execution (scoring above 7 by GPT-4, scale from 0 to 9). The Success conversion ratio (SCR) indicates the proportion of originally failed responses that became successful after modification, while Success preservation ratio (SPR) reflects the proportion of originally successful responses that remained successful.
Refer to caption: x6.png

We identified a dimension within the input embedding space associated with instruction-following.
To evaluate whether this dimension significantly impacts the models’ behavior, we manipulated the representations along this direction using representation engineering (Marks & Tegmark, ; Zou et al., ). An increase in the models’ instruction-following success rate tied to manipulations along the identified direction validates the role of the dimension in shaping the models’ generation outcomes toward instruction adherence.

### 3.1 Settings

Method
For each input representation $R_{original}$, we applied a transformation in the identified direction $D$ using the formula $R_{updated}=R_{original}+\alpha\times D$, where $\alpha$ is a scaling hyper-parameter.
We applied this transformation to all input representations, including both success and failure cases, to evaluate whether RE could improve instruction following universally, without disrupting cases where the model was already successful.
This adjustment was applied to the representations in the last layer of the model, as it was more robust to variations in $\alpha$.
We focused on the representation of the first token, which corresponds to the input embedding before any response generation, since the goal of representation engineering (RE) is to adjust internal representations before the response is generated to improve the model’s instruction adherence.
The direction $D$ is the weight of a linear probes trained on all IFEval-simple dataset.
(^3^33We also experimented with training the linear probe on 70% of the IFEval-simple dataset and applying RE to the remaining 30% test set. The results were similar but slightly worse than when the linear probe was trained and RE was applied to the entire dataset. Since our primary focus is on analyzing the variance caused by RE itself, rather than variance from train-test splits, we present the results using the full dataset here.)

Metric
We evaluated the success rate (SR) of instruction-following using predefined evaluation functions from the IFEval (Zhou et al., ). Additionally, we assessed the quality of the responses using GPT-4, scoring each response on a scale from 0 to 9 based on its relevance to the given task.
We defined quality ratio (QR) as the number of responses scoring above 7 divided by the total number of responses that successfully follow instructions (this cutoff was defined based on the distribution of quality scores). F2T (False to True) and T2T (True to True) show how many failed responses became successful and how many successful ones remained so after modification. The Success conversion ratio (SCR) $\coloneq$ $\frac{F2T}{(F2T+F2F)}$ indicates the proportion of originally failed responses that became successful after modification, while Success preservation ratio (SPR) $\coloneq$ $\frac{T2T}{(T2T+T2F)}$ reflects the proportion of originally successful responses that remained successful.

Baseline and hyperparameter selection To demonstrate the effectiveness of the identified instruction-following dimension, we compared it against random directions.
Each model and instruction type required a different $\alpha$ value based on their specific geometry. If $\alpha$ is too large, it can degrade the quality of responses; if too small, it may not effectively improve instruction-following. We selected $\alpha$ for each model and instruction type using a validation set comprising 10% of the instruction data. The selected $\alpha$ values were: 0.3 for Llama-2-chat-13b and Llama-2-chat-7b, 0.1 for Phi-3, and 0.15 for Mistral-7B.

### 3.2 Results

RE on instruction-following direction improves success rate while maintaining quality
Our experiments demonstrate that applying the RE direction generally improves the instruction-following success rate (SR) across most models and instruction types. As shown in Table 4, the SR with the instruction-following direction usually outperforms the original success rate and is lower bounded by the the original SR – that is, the instruction-following dimension does not lead to worse than original SRs. Additionally, the QR remains equal to or higher than the original, indicating that RE can be applied with minimal risk of reducing response quality.
Figure 5 in the Appendix provides an illustrative example of modified responses. In this case, the task was to write a resume with the instruction to include three specific keywords. The original response only included one keyword, whereas the modified response, guided by the instruction-following direction, successfully incorporated all three keywords, demonstrating the effectiveness of RE in enhancing instruction adherence.

Instruction-following direction is better than random directions
When comparing RE direction to random directions, RE consistently outperforms random directions in increasing the success rate across all instruction types and models, as illustrated in Table 4 and Figure 3.
The ratios of True-to-True (T2T) and False-to-True (F2T) transitions are typically larger for the instruction-following direction than for random directions, indicating a more reliable improvement in success rates.

## 4 Interpreting the instruction-following dimension

While manipulating representations along the instruction-following dimension reveals that it influences a model’s behavior, the meaning behind this manipulation remains unclear.
To interpret the meaning of the instruction-following dimension, we conduct a sensitivity analysis to investigate the relative of perturbations on the internal state of LLMs, compared to our identified direction. We consider three perturbation types: task familiarity, instruction difficulty, and phrasing. We (1) systematically alter the original input prompts in IFEval-simple dataset for each perturbation, (2) compute the resulting difference in internal state representation space before and after the perturbation, and (3) compute the cosine similarity between the perturbation-induced difference vector and the instruction-following dimension we identified. We designed prompt changes for each perturbation:

(1) Task Familiarity: We investigated whether the instruction-following dimension might be related to how familiar the model is with a given task. For example, the task “write a resume for software engineer” might be more familiar to the model than “write a summary about current events”, if it was more common in the data used to train the LLMs. If a task is more familiar to a model, it may be easier for the model to follow instructions regarding that task.
To perturb the model on task familiarity, we kept the instruction constant while changing the task to one with lower perplexity (Jelinek et al., ).
Perplexity measures the probability of tokens in generation, reflecting task familiarity (Gonen et al., ), where high perplexity indicates a familiar task and vice versa.

(2) Instruction Difficulty:
We investigated the relationship of the instruction-following dimension with the complexity of the instructions. We perturbed the instruction difficulty by simplifying instructions by relaxing instruction-related constraints. For example, in the original instruction “please include keywords: coding, Python, computer, experience”, we reduced the complexity by reducing the number of keywords required in the instruction to “please include the keywords: coding, Python”.

(3) Phrasing Modification:
Finally, we examined whether the instruction-following dimension was correlated to how the prompt is phrased. We rephrased the prompts while keeping the meaning of the task and the instruction unchanged. For example, we modified “Write a resume for software engineer. Please include keywords such as coding, Python, computer, experience” to “I want you to write about software engineer resume including four words coding, Python, computer, or experience”. We used GPT-4 to rephrase both the task and instruction in the input prompt, and applied GPT-4 again to validate that the meaning of the contents remained the same after rephrasing.

We selected 20 prompts, each containing a task and an instruction from the ‘forbidden keyword’ instruction type in IFEval-simple dataset. For each perturbation type, we created five modified versions of each prompt. We then averaged the representations of these modified prompts and calculated the difference between this averaged representation and the representation of the original prompt. Finally, we assessed how well this difference vector aligned with the instruction-following dimension by computing the cosine similarity.

Our findings, illustrated in Figure 4, show the sensitivy analysis results for two models: Llama-2-13b-chat and Llama-2-7b-chat. In both models, the results indicated that phrasing modifications have a stronger correlation with the instruction-following dimension than task familiarity or instruction difficulty.
These results support the hypothesis that the instruction-following dimension is more closely tied to how prompts are phrased rather than the inherent difficulty of the task or the complexity of the instruction. This suggests that how prompts are phrased plays a critical role in determining whether LLMs will successfully follow the instructions, aligned to observations Lu et al. (); Sclar et al. () showing LLMs are sensitive to prompt formatting.

Figure: Figure 4: Cosine similarity alignment for modified data in the ‘forbidden keyword’ instruction type across two models (Llama-2-7b-chat (Left) and Llama-2-13b-chat (Right)). The figure shows the cosine similarity between the instruction-following dimension and the difference vector (computed as the difference between the original prompt’s representation and the average representation of five modified prompts) across 20 sampled prompts. Modifications include changes in task familiarity, instruction difficulty, and phrasing. The results indicate that phrasing modifications align more closely with the instruction-following dimension, suggesting that how prompts are phrased plays a crucial role in determining instruction adherence.
Refer to caption: x7.png

## 5 Related work

Instruction-following in LLMs
Recent research has introduced various benchmark datasets to evaluate the instruction-following capabilities of LLMs across different contexts(Zhou et al., ; Qin et al., ; Yan et al., ; Xia et al., ). Among these, we selected the IFEval dataset (Zhou et al., ) as the foundation for our study, due to its structured and objective evaluation framework that minimizes ambiguity by using verifiable instructions. Beyond evaluation, several approaches have been proposed to improve instruction-following performance, such as modifying attention mechanisms (Zhang et al., ) and applying fine-tuning strategies (He et al., ; Sun et al., ).
In contrast to prior work that primarily focuses on evaluating or enhancing instruction-following, our study aims to understand why LLMs sometimes fail to follow instructions by analyzing their internal representations.

Linear Probing and Representation engineering on LLMs
Linear probes have been widely used for interpreting and analyzing the representations of neural networks (Alain & Bengio, ) and language models (Belinkov, ; Elazar et al., ). Specifically, probing for the trustworthiness of LLMs has been an active area of research (Azaria & Mitchell, ; Marks & Tegmark, ; MacDiarmid et al., ; Li et al., ; Burns et al., ; Zou et al., ; Rimsky et al., ; Li et al., ; Nanda et al., ; Subramani et al., ; Tigges et al., ; Todd et al., ; Farquhar et al., ; Ahdritz et al., ; Duan et al., ). These probing methods are closely related to representation engineering and editing techniques aimed at modifying model knowledge and behavior (Zou et al., ; Rimsky et al., ; Li et al., ; Park et al., ; Chen & Yang, ; Luo et al., ; Turner et al., ).
Our work is distinct from these previous efforts, which primarily focus on representations related to truthfulness and reducing hallucinations. In contrast, our study centers on representations related to instruction-following, highlighting the importance of understanding how models internally handle instructions.

## 6 Discussion and Conclusion

### 6.1 LLMs internally recognize whether they will follow instructions

Our findings suggest that LLMs may possess an inherent ability to predict whether they will successfully follow instructions, even before the generation process begins. This capability is supported by several key observations:

LLMs generalize well across tasks but struggle with different instruction types We find that while LLMs can generalize across different tasks, they struggle with generalization across different instruction types. This suggests that distinct instruction categories may have unique geometries within the models’ internal representation space, making it more challenging for the model to generalize across them.

LLMs can predict instruction success from the first token
We observe that the model’s internal representations are separable from the very first token, which corresponds to the embedding of the input prompt. This indicates that the likelihood of instruction-following success can be determined early in the process, before the model generates any responses. This highlights the critical role of how the input prompt is encoded and the importance of input representations in predicting instruction-following outcomes.

Representation engineering increases instruction-following success We further validate the significance of the identified instruction-following dimension by adjusting the model’s representations. By moving failure cases into the success class along this dimension and comparing the results to random adjustments, we observe a significant increase in the success rate while keeping the task quality. This demonstrates that the identified dimension is both meaningful and can be used practically.

The instruction-following dimension is closely tied to prompt phrasing Our findings, in Figure 4, reveal that the instruction-following dimension is most closely associated with the phrasing of prompts, rather than the inherent difficulty of the task or the specific details of the instructions. This suggests that how instructions are phrased plays a crucial role in whether LLMs will follow them and is consistent with our finding on the separability of representations from the early token.

### 6.2 The role of input prompt representation in instruction-following failures

Our findings highlight the role of representation of the input prompt in determining instruction-following success in LLMs. We discover that the instruction-following dimension identified in our analysis is sensitive to changes in how the input prompt is phrased. This sensitivity explains several behaviors of LLMs:

Why LLMs fail in following instructions
LLMs may fail to follow even simple, clear instructions because the encoding of the input prompt within the models’ internal representation space can be easily disrupted. Our findings suggest that small variations in how a prompt is phrased can result in significant differences in how the model processes the instruction, leading to failures in adherence. This issue arises not from ambiguity in the instruction itself, but from the LLM’s sensitivity to the exact structure and phrasing of the input, which influences how the instruction is embedded and processed internally. As a result, the model might not consistently follow instructions, even when they are clear and familiar.

Why Prompt Engineering (PE) works
PE operates by slightly altering the phrasing of a prompt, which in turn changes how the input is encoded within the model. This subtle shift in encoding can move a representation from a failure class to a success class in terms of instruction-following within the input embedding space. Our work with representation engineering achieves a similar outcome, but instead of modifying the input text, we make adjustments directly in the representation space. Both approaches influence the model’s internal states, highlighting the importance of the input encoding process. Our observations align with prior research showing LLM sensitivity to prompt formatting (Lu et al., ; Sclar et al., ; Gonen et al., ).

Semantic sensitivity of LLM input embedding space
The fact that instruction-following success or failure can be altered by slight prompt rephrasing shows that the LLM’s input embedding space is semantically sensitive. This sensitivity suggests that the model’s internal representation of prompts is brittle, making LLMs vulnerable to small changes in how an input is framed or phrased. This fragility, likely driven by the model’s large size and the complexity of its training dynamics, creates challenges in ensuring robust instruction adherence.
Given this sensitivity, future efforts should focus on making LLMs’ input embedding space more robust and reliable. One potential approach is to fine-tune models with an explicit focus on stabilizing instruction-following by utilizing the identified instruction-following dimension.

Our findings highlight the crucial role of prompt encoding in instruction-following success for LLMs. The sensitivity of the input embedding space to slight changes in phrasing explains why LLMs may fail to follow even clear instructions and why prompt engineering is effective. By adjusting the representations directly, as we did with representation engineering, we show that it is possible to significantly improve instruction adherence. Going forward, improving the robustness of LLMs’ input embeddings through training can make models more reliable and consistent in following instructions across a variety of tasks. This is crucial for building trustworthy AI systems, especially in real-world applications where accuracy and reliability are essential.

### 6.3 Limitations and Future work

Our analysis was primarily focused on a specific set of tasks and models. Although our current results are consistent across the models we studied, future work could extend these findings by evaluating additional models to determine whether the identified instruction-following dimension generalizes across different LLM architectures.
Additionally, expanding the dataset to include a wider variety of instruction-following cases could enrich the analysis and improve the generalizability of our findings. We focused our investigation on simple modeling approaches to identify an instruction-following dimension and evaluate its practical significance.
Future work could include additional methods train linear probes, particularly in handling domain shifts.
Similarly, better approaches to representation engineering (Zou et al., ) could further improve the success rate of instruction-following modifications.
Finally, unambiguously interpreting the meaning of the instruction-following dimension remains an open question. We considered three hypotheses and found that phrasing modification was most closely related to the dimension associated with instruction-following using a perturbation-based approach. Additional investigations to develop systematic approaches to interpret the dimension could add to a deeper understanding of its meaning and implications.

## Appendix A Appendix

### A.1 Examples of IFEval-simple Dataset

The IFEval-simple dataset is created to focus specifically on instruction-following, removing the confounding influence of varying tasks present in the IFEval dataset (Zhou et al., ). In this modified version, we select 5 instruction types commonly used in real-world AI applications, such as including or excluding keywords, generating responses with placeholders, and ensuring specific phrases are present in the generated text. These instructions are paired with the same set of 100 tasks to help isolate the instruction-following dimension.
By using the same set of tasks across all instruction types, we ensure that any differences in model behavior are attributed to instruction-following rather than task-specific features. This allows us to more effectively probe the model’s internal representations and evaluate how well it can follow instructions across various scenarios.

Table 5 presents examples from the IFEval-simple dataset, such as tasks like writing a resume or creating a joke about programmers. The instructions assigned to each task vary, requiring the model to follow specific guidelines such as including or excluding certain keywords, ensuring word usage meets a specific frequency, and adhering to formatting rules. The keywords that must be included or excluded differ based on the task. For instance, in the resume task, keywords might include “resume”, “software”, or “engineer”, whereas in the joke task, the focus may shift to terms like “syntax” or “code”. These varied instructions introduce diverse challenges for the model in instruction-following.

**Table 5: Examples from the IFEval-simple dataset. This table shows two tasks: writing a resume and crafting a joke about programmers. Each task is paired with multiple instruction types, such as including/excluding keywords, ensuring word frequency, and adhering to specific content formatting rules. The uniform set of tasks across different instruction types helps isolate the instruction-following dimension by removing task-specific variations.**
| Type | Example |
| --- | --- |
| Task | Write a resume for a software engineer with 5+ years of experience in the Bay Area, CA. |
| Instruction | keywords:existence<br>Make sure to include the keywords: “skills”, “technology”, “career”.<br>keywords:forbidden<br>Do not include the following keywords: resume, software, engineer, experience.<br>keywords:frequency<br>Make sure to use the word “qualifications” at least 2 times.<br>startend:end checker<br>Your resume must end with the exact phrase “Looking forward to contributing to innovative projects.”<br>detectable content:number placeholders<br>Make sure to include at least 5 placeholders represented by square brackets, such as [name]. |
| Task | Write a joke about programmers. |
| Instruction | keywords:existence<br>Make sure to include the keywords: “humor”, “code”, “life”.<br>keywords:forbidden<br>Do not include the following keywords: joke, programmers.<br>keywords:frequency<br>Make sure to use the word “syntax” at least 3 times.<br>startend:end checker<br>Your programmer joke must end with the exact phrase “And that’s the real bug in the code of life.”<br>detectable content:number placeholders<br>Make sure to include at least 3 placeholders represented by square brackets, such as [name]. |

**Table 6: Sample of 50 tasks from the IFEval-simple dataset. This table provides a subset of 50 tasks from the IFEval-simple dataset, which includes a total of 100 tasks designed to evaluate instruction-following performance.**
| Index | Task |
| --- | --- |
| 1 | Write a story about the importance of understanding the truths that are not obvious. |
| 2 | Write a serious riddle about trips and stitches in a poem style. |
| 3 | Write a rubric for teenagers on how to review a book. |
| 4 | Write a persuasive email to a teenager who lives in Aberdeen, Scotland. |
| 5 | Write a resume for a software engineer with 5+ years of experience in the Bay Area, CA. |
| 6 | Write a song about regrets in the style of Taylor Swift. |
| 7 | Write an essay about Alvin and the Chipmunks. |
| 8 | The Legend of the Sword and the Fairy is a movie in which Wan Wan is a villain. Write a story about Wan Wan’s character. |
| 9 | Write a story about a family that goes camping in the woods. |
| 10 | Write an obviously fake news article saying that aliens have invaded earth. Make it funny. |
| 11 | Write a song about the benefits of eating your vegetables. |
| 12 | Write a startup pitch for ”Ward and Guerre”. |
| 13 | Is Seoul a good place to live? |
| 14 | Write a letter to a friend asking them to go and vote. |
| 15 | Write a resume for a fresh high school graduate who is seeking their first job. |
| 16 | Is praying for someone’s health a good idea? |
| 17 | What’s the difference between a 2-stroke and a 4-stroke motor? |
| 18 | Explain to a group of elementary school students why we have seasons. |
| 19 | Can you re-create a story from a fictional newspaper with the title: ”A man mysteriously died in his house, and police are investigating”? |
| 20 | Come up with a proposal for a new research project on how to improve the quality of life for people with disabilities. |
| 21 | Write a blog post about the benefits of meditation for busy professionals. |
| 22 | Create a recipe for a vegan gluten-free chocolate cake. |
| 23 | Draft a comprehensive guide on how to start a podcast. |
| 24 | Develop a character sketch for a villain in a fantasy novel. |
| 25 | Compose a haiku about a sunset over the ocean. |
| 26 | Summarize the plot of the film ”Inception”. |
| 27 | Explain the theory of relativity in simple terms. |
| 28 | Write a review of the latest iPhone model. |
| 29 | Describe the lifecycle of a butterfly. |
| 30 | Propose a business plan for a sustainable fashion brand. |
| 31 | Outline the steps for training a puppy. |
| 32 | Discuss the impact of social media on teenage mental health. |
| 33 | Draft a speech for a climate change conference. |
| 34 | Write a joke about programmers. |
| 35 | Explain how to change a car tire. |
| 36 | Develop a fitness routine for beginners. |
| 37 | Compose a sonnet about the city of Venice. |
| 38 | Write a user manual for a smartwatch. |
| 39 | Describe a typical day in ancient Rome. |
| 40 | Provide advice on how to improve public speaking skills. |
| 41 | Discuss the effects of global warming on polar bears. |
| 42 | Draft a letter of recommendation for a student. |
| 43 | Summarize the story of ”The Great Gatsby”. |
| 44 | Explain the process of photosynthesis. |
| 45 | Write a critique of a famous painting. |
| 46 | Develop a marketing strategy for a new video game. |
| 47 | Compose a limerick about a mischievous cat. |
| 48 | Describe the benefits of yoga for athletes. |
| 49 | Write instructions for assembling a desk. |
| 50 | Discuss the history of the internet. |

### A.2 Example of Representation Engineering

Figure: Figure 5: RE example An illustrative example of modified responses. In this case, the task was to write a resume with the instruction to include three specific keywords. The original response only included one keyword, whereas the modified response, guided by the instruction-following direction, successfully incorporated all three keywords, demonstrating the effectiveness of RE in enhancing instruction adherence.
Refer to caption: extracted/5965808/iclr2024/contents/figures/RE/RE_example.png

### A.3 Instruction generalization on expanded experiment

In the main paper, we observed that models struggle to generalize across unseen instruction types, with AUC scores ranging from 0.50 to 0.55, which is close to random chance, as shown in Table 1 and Table 3 of the main paper. One hypothesis for this poor generalization is the limited number of instruction types used in the initial experiments, where the linear probe was trained on just 4 instruction types. To further investigate this, we expanded the dataset to include 23 instruction types across 8 categories, each paired with 20 tasks.

Unlike the IFEval dataset, which contains 25 instruction types across 9 categories, we omitted the ‘combination’ category, which includes the ‘combination: Repeat Prompt’ and ‘combination: Two Responses’ instruction types. This is because combined instructions can lead to conflicting signals in our analysis, where success in one instruction type but failure in another may produce mixed representations. By focusing on single instruction types, we aim to more clearly capture the representations associated with instruction-following success and failure.
In comparison to IFEval-simple, which features 5 instruction types across 3 categories, this expanded dataset includes 23 instruction types across 8 categories, helping to prevent overfitting to a small number of instructions.

The results from this expanded experiment, shown in Table 7 for different layers and Table 8 for different tokens, reveal that despite increasing the number of instruction types, the models still demonstrate limited generalization across unseen instruction types. The AUC scores remain close to chance levels, similar to the initial experiments. As shown in Table 7 and 8, the results indicate that adding more instruction types does not significantly improve instruction generalization.
These findings reinforce the conclusion that models struggle to generalize instruction-following across different instruction types. This suggests that a “global” instruction-following dimension, applicable across diverse instruction types, may not exist.

**Table 7: Instruction-type generalization on IFEval-simple-expanded across layers AUC scores across different models and instruction types from IFEval-simple-expanded. The ‘punctuation’ instruction type is marked with ‘-’ due to an insufficient number of data points caused by a low success rate, making it impossible to compute reliable AUC scores.**
| Models | LLaMA-2-chat-7B | LLaMA-2-chat-13B | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Instructions | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr | Early lyr | Middle lyr | Last lyr |
| startend | 0.70 | 0.61 | 0.57 | 0.47 | 0.54 | 0.52 | 0.56 | 0.62 | 0.59 | 0.60 | 0.46 | 0.48 |
| keywords | 0.39 | 0.49 | 0.48 | 0.53 | 0.46 | 0.45 | 0.42 | 0.43 | 0.45 | 0.59 | 0.48 | 0.47 |
| detectable_format | 0.52 | 0.45 | 0.42 | 0.50 | 0.47 | 0.47 | 0.49 | 0.45 | 0.41 | 0.81 | 0.79 | 0.70 |
| length_constraints | 0.40 | 0.30 | 0.33 | 0.60 | 0.50 | 0.52 | 0.44 | 0.57 | 0.56 | 0.69 | 0.52 | 0.52 |
| punctuation | - | - | - | 0.47 | 0.37 | 0.35 | 0.94 | 0.95 | 0.92 | - | - | - |
| change_case | 0.59 | 0.40 | 0.35 | 0.28 | 0.26 | 0.29 | 0.61 | 0.43 | 0.39 | 0.40 | 0.34 | 0.29 |
| detectable_content | 0.65 | 0.62 | 0.61 | 0.59 | 0.53 | 0.57 | 0.49 | 0.37 | 0.34 | 0.13 | 0.11 | 0.10 |
| language | 0.38 | 0.49 | 0.47 | 0.12 | 0.13 | 0.17 | 0.41 | 0.60 | 0.62 | 0.78 | 0.77 | 0.80 |
| AVERAGE | 0.52 | 0.48 | 0.46 | 0.44 | 0.41 | 0.42 | 0.54 | 0.55 | 0.54 | 0.57 | 0.50 | 0.48 |

**Table 8: Instruction-type generalization on IFEval-simple-expanded across tokens AUC scores across early, middle, and late token representations, showing instruction-type generalization performance on IFEval-simple-expanded. The results indicate that despite expanding the number of instruction types, models continue to struggle with unseen instruction types, with scores close to chance levels across different token positions. The ‘punctuation’ instruction type is marked with ‘-’ due to an insufficient number of data points caused by a low success rate, making it impossible to compute reliable AUC scores.**
|  | LLaMa2-chat-7b | LLaMa2-chat-13b | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| instructions | Early token | Middle token | Last token | Early token | Middle token | Last token | Early token | Middle token | Last token | Early token | Middle token | Last token |
| startend | 0.70 | 0.42 | 0.29 | 0.47 | 0.53 | 0.55 | 0.56 | 0.56 | 0.60 | 0.60 | 0.70 | 0.64 |
| keywords | 0.39 | 0.69 | 0.66 | 0.53 | 0.32 | 0.40 | 0.42 | 0.60 | 0.50 | 0.59 | 0.37 | 0.47 |
| detectable_format | 0.52 | 0.45 | 0.49 | 0.50 | 0.58 | 0.52 | 0.49 | 0.60 | 0.57 | 0.81 | 0.56 | 0.62 |
| length_constraints | 0.40 | 0.57 | 0.55 | 0.60 | 0.61 | 0.56 | 0.44 | 0.55 | 0.56 | 0.69 | 0.44 | 0.49 |
| punctuation | - | - | - | 0.47 | 0.47 | 0.49 | 0.94 | 0.65 | 0.43 | - | - | - |
| change_case | 0.59 | 0.52 | 0.51 | 0.28 | 0.58 | 0.45 | 0.61 | 0.47 | 0.48 | 0.40 | 0.45 | 0.37 |
| detectable_content | 0.65 | 0.53 | 0.56 | 0.59 | 0.47 | 0.55 | 0.49 | 0.54 | 0.45 | 0.13 | 0.38 | 0.33 |
| language | 0.38 | 0.46 | 0.36 | 0.12 | 0.56 | 0.51 | 0.41 | 0.59 | 0.75 | 0.78 | 0.40 | 0.46 |
| AVERAGE | 0.52 | 0.52 | 0.49 | 0.44 | 0.51 | 0.50 | 0.54 | 0.57 | 0.54 | 0.57 | 0.47 | 0.48 |

### A.4 Success rate

This section presents the success rate for instruction-following, which measures the accuracy of responses adhering to instructions. The success rates for the IFEval dataset(Zhou et al., ) are shown in Table 9, for our IFEval-simple dataset in Table 10, and for IFEval-simple-extended in Table 11, which is used in Section A.3 of the Appendix. The IFEval dataset consists of 25 instruction types categorized under 9 broader categories, with approximately 20 tasks per instruction type. For details on IFEval and IFEval-simple, please refer to Section 2.1 of the main paper.
We use the success rate (loose) metric from Zhou et al. (). To ensure consistent results without randomness in decoding, we used greedy decoding without sampling when calculating the success rate.

**Table 9: Success rate on the IFEvalZhou et al. () across 9 categories of instruction types**
| IFEval inst | LLaMa2-chat-7b | LLaMa2-chat-13b | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |
| --- | --- | --- | --- | --- |
| change_case | 0.48 | 0.52 | 0.62 | 0.29 |
| detectable_content | 0.85 | 0.89 | 0.79 | 0.89 |
| detectable_format | 0.66 | 0.68 | 0.78 | 0.67 |
| keywords | 0.68 | 0.71 | 0.73 | 0.75 |
| language | 0.68 | 0.58 | 0.87 | 0.97 |
| length_constraints | 0.46 | 0.48 | 0.55 | 0.41 |
| punctuation | 0.24 | 0.14 | 0.17 | 0.11 |
| startend | 0.67 | 0.58 | 0.63 | 0.22 |
| combination | 0.24 | 0.22 | 0.17 | 0.22 |

**Table 10: Success rate on IFEval-simple across 5 instruction types under 3 categories**
| IFEval inst | LLaMa2-chat-7b | LLaMa2-chat-13b | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |
| --- | --- | --- | --- | --- |
| keywords:existence | 0.79412 | 0.87255 | 0.86275 | 0.94118 |
| keywords:forbidden_words | 0.18627 | 0.28431 | 0.36275 | 0.32353 |
| keywords:frequency | 0.86275 | 0.92157 | 0.91176 | 1.0000 |
| startend:end_checker | 0.23529 | 0.16667 | 0.27451 | 0.13725 |
| detectable_content:number_placeholders | 0.76471 | 0.80392 | 0.5098 | 0.87255 |

**Table 11: Success rate on IFEval-simple-extended across 8 categories of instruction types (excluding the ‘combination’ category)**
| IFEval inst | LLaMa2-chat-7b | LLaMa2-chat-13b | Mistral-7B-inst-v0.3 | Phi-3-mini-128k |
| --- | --- | --- | --- | --- |
| change_case | 0.53 | 0.70 | 0.46 | 0.31 |
| detectable_content | 0.65 | 0.90 | 0.75 | 0.94 |
| detectable_format | 0.67 | 0.72 | 0.72 | 0.64 |
| keywords | 0.80 | 0.91 | 0.90 | 0.96 |
| language | 0.40 | 0.10 | 0.94 | 0.83 |
| length_constraints | 0.53 | 0.56 | 0.69 | 0.40 |
| punctuation | 0.15 | 0.25 | 0.06 | 0.00 |
| startend | 0.98 | 0.93 | 0.69 | 0.28 |

### A.5 More result on Representation Engineering

In the main paper, we presented the average performance of Representation Engineering (RE) across all instruction types and models. Here, we provide more detailed insights through figures, highlighting the variation in RE effectiveness across different types of instructions.

Figure 3 shows the transition metrics for high-quality responses, including F2T (False to True) and T2T (True to True). These results demonstrate the robustness of the instruction-following dimension in improving success rates. The proportion of F2T and T2T transitions is consistently higher when using the instruction-following dimension compared to random directions across various instruction types. This indicates that RE not only enhances success rates but also effectively converts failed responses into successful ones, while maintaining overall task quality.

Figure: Figure 6: Transition metric for Representation Engineering on the last layer of four models Success rate (SR) only on high quality responses (scoring above 7 by GPT4) in task execution. The Success conversion ratio (SCR) indicates the proportion of originally failed responses that became successful after modification, while Success preservation ratio (SPR) reflects the proportion of originally successful responses that remained successful.
Refer to caption: x8.png

### A.6 PCA across all five instruction types

In this section, we extend the PCA analysis to include all five instruction types used in our experiments. This analysis contrasts with the PCA plot in Figure 2 of the main paper, where we focus on three instruction types within the keyword category. In the main paper, the PCA plot show a clear tendency towards separability of the instruction-following dimension across tasks, even though the data points were not perfectly linearly separable.
However, in this extended analysis with all five instruction types in Figure 7, the representations are less linearly separable in the 2-dimensional PCA plot. This highlights that different instruction types (or categories) may exhibit distinct geometries in the representation space. The lack of clear separability further supports our findings in the main paper that linear probes trained on one set of instruction types struggle to generalize to unseen instruction types in Section 2.3. This suggests that there is no “global” instruction-following dimension that can be applied across different types of instructions, likely due to the varying internal geometries of these categories.

Figure: (a) Llama-2-13b-chat-hf
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - See weaknesses
- What is the motivation behind the update in line 303-304? Could you provide more intuition on what is being done to the input representation there?
soundness: 3
strengths: * Paper offers very interesting and operationalizable insights about the internal representations of LLMs.
* Paper tells a very good, satisfying story: from identifying the dimension, to manipulating and interpreting it.
summary: This paper investigates the internal representation of LLMs and identifies a "dimension" that corresponds to their success in instrution-following. They show that manipulating this dimension could improve success in instruction-following without degrading the quality of the outputs. They also measure correlation of the dimension with different perturbations and show that it is more sensitive to the phrasing of the prompt than the task difficiulty or faimiliarity. 

'
weaknesses: - The methodology description lacks clarity due to insufficient mathematical notation. It is especially difficult to follow the procedures outlined in Sections 3 and 4. The authors should clearly define mathematical objects, such as "input representation," and provide precise equations for the computations (e.g., lines 370-374). Specifically, given a prompt \( x_1, x_2, ..., x_n \) of n words and an output \( y_1, y_2, ..., y_m \) of  m words, what is the "input representation" as a function of these tokens? It appears to be computed solely from the output words.

* The terminology is confusing. The term "instruction" would be more accurately described as "constraints." It’s challenging to distinguish "instruction" from "task," as "instruction" could imply the entire command given to the model. Additionally, the meaning of "dimension" is unclear. Does it refer to a hypothetical concept rather than an actual dimension of a hidden vector computed by the model?

* The experiments were conducted on a single, relatively small dataset, raising questions about whether the findings will generalize.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - At the start of section 2, it would be helpful to elaborate what the IFEval dataset is used for 
- Make text in Figure 1 bigger
soundness: 4
strengths: This paper is well-structured and the findings are very interesting - while using linear probes to find a latent dimension for a specific phenomenon is not that interesting or original, the follow-up analyses demonstrating that instruction following can be made more accurate with representation engineering, and comparing the correlation of this dimension to different properties of the prompts, are both interesting. In addition, most claims made in the paper are sufficiently backed up with experiments and evidence.
summary: This work analyzes and compares LLM internal representations when it can successfully follow instructions, vs. when it fails at following instructions. By training linear probes on representations, authors find a latent direction that predicts instruction following success, and shows it generalizes to unseen tasks and instructions. In addition, this work uses representation engineering to make adjustments along the instruction following dimension they found, and finds these adjustments can improve instruction following performance. Finally, this work also finds that the instruction following dimension is more correlated with how the prompt is phrased, rather than the difficulty of the instruction or the task.
weaknesses: In section 3, I'm confused what the direction D taken is for different instructions, since the previous section states that linear probes do not generalize well across instructions. Does this mean representation engineering promotes different directions for different instructions, or is there a universal direction that enhances all instructions generally? It would be helpful to clarify this in the paper.
For section 4 as well, one instruction type is chosen to perform the analysis on correlations. Could you elaborate on how much the findings in this section generalizes to other instruction types? Or could there be instruction types whose representation is more correlated with task familiarity rather than phrasing?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Have you tried reverse representation engineering? Will it be worse than random?
2. Why did you focus on the representation of the first token in the last layer? Any insight?
soundness: 3
strengths: 1. This paper provides a new understanding of instruction following, using token representation to interprete LLMs themselves.
2. The experiments are well designed and the writing is easy to follow.
summary: This work finds that the representation of token is related to the success rate of instruction following. The authors apply linear probing to identify a specific dimension and use representation engineering (RE) to manipulate the representations. Experimental results show that RE  improves success rate while maintaining quality. Further analysis reveals that phrasing modifications plays a critical role rather than task familiarity or instruction difficulty. This paper provides a deeper understanding to this field.
weaknesses: 1. Some of the conclusions in this paper are as expected, e.g. results from Table 1 and Table 2 are related to "Lost in the Middle" phenomenon. [1]
2. The performance improvement brought by this method is limited as shown in Table 4.
3. The IFEval dataset is small and such problem may be addressed by extending instruction following data.

[1] Lost in the Middle: How Language Models Use Long Contexts

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Generally, please just respond to my weaknesses above. I'm willing to increase the scores (total, contribution, and soundness), especially if you address the crucial ones related the generalizability and usability with new experiments or strong arguments. 

Q1: Can you explain where the "middle" token was for the different prompts? Was it the middle token of each sequence and thus changing the position for each prompt?

### Comments (Not considered for the review)

- For Figure 2, I feel LDA (fitted on training data) might be a better fit than PCA to show separability, given that LDA actively tries to separate the known classes.
- In lines 213-214, you state that the instruction type generalization values are close to chance. Given the stated 1-sigma uncertainties in Table 1, the presented values are not statistically significantly different (depending on the definition, 2-5 sigma) from random chance.
soundness: 2
strengths: - The goal of finding a way to manipulate latent representations to improve the instruction following of language models (or detect whether instructions are not being followed) has the potential for significant usability and safety impact while being novel. 
- The paper is very well written and the presentation is generally clear.
- The presented methodology is well structured and each section is clearly motivated by the previous results.
- The authors test four open-source models across many different tasks, demonstrating a certain amount of robustness.
- The authors generally do not overclaim their results and are very transparent.
summary: The authors expand prior work on finding meaningful directions in the latent space of transformer language models to find a direction associated with whether models follow a given instruction or not. 
- For the analysis, they adapt and expand an existing dataset and study four open-source models (7b to 13b parameters). 
- Using linear probes, the authors find directions associated with instruction following across the studied tasks, but not generalizing to hold-out instruction types. 
- They show that manipulating the latent representation along the found direction can improve instruction following success and maintain instruction tuning quality for already successful cases.
- The authors find that the found latent space direction is affected by prompt sensitivity/phrasing of the instructions (less so by task complexity and perplexity)
weaknesses: The authors do not claim that they find a universal direction in these models that fully represents the idea of instruction following. However, to make a significant contribution via usability and understanding of latent representations, this work would benefit from addressing a few weaknesses to enable more robust future work and enhance the usability of the results.
 
### Weakness 1

The studied tasks are somewhat general but the instructions are fairly simple and not necessarily representative of general language model uses. I wonder how well the results of this work might generalize: It is also unclear how well instructions are being followed in a general use case. To use the example of the paper: You might ask a health bot to be aware of your left knee injury when coming up with training plans, but does adding the found latent space direction have side effects like avoiding leg training altogether or creating asymmetric training plans leading to other issues?  The results in this paper could be significantly strengthened by finding applications that allow for a more nuanced analysis of how instructions are being followed after representation engineering to demonstrate the usefulness of their technique. 

### Weakness 2

In Sec. 3, I think an opportunity was missed to strengthen the analysis by demonstrating that the found direction also works in a counterfactual way: Can you use the fitted alpha values and directions for the F2T and T2T tests and only flip the sign of the respective alpha value to check if it also works on (intentional) T2F and F2F? I think this is necessary to demonstrate a minimum of reliable usability and support the claim of safety impact (to potentially get a model to not comply).

### Weakness 3

It is unclear (to me) if using only one direction is sufficient for modeling whether an instruction is being followed or not and even if, whether the presented analysis finds that direction. For example, the prompt sensitivity studies in Sec. 4 affecting the performance could be caused by other meaningful "directions" being entangled in the one you care about. To be more precise: If changing "write a resume for [a] software engineer [...]" to "I want you to write about [a] software engineer resume [...]" makes a difference, how do we know this is not caused by, e.g., confounding the learned directions with the instruction data set with a "dialog in first-person" direction (not unlikely,  given the colorful interpretations modern SAE approaches find).

To link it to another, more concrete use case: Is it possible to circumvent safety training of a language model using the proposed methodology? Although also only necessary and not sufficient, such a more complex example would demonstrate that using only one direction is powerful enough to be useful. 

If one direction is not sufficient, it might be worth testing if a higher-order approach (2D, 3D, ...) might be more robust to find that direction. There are plenty and different works on this, but for example, [1] uses PCA on latent representations to find the one relevant direction to model how small backdoored language models process trigger inputs to switch to toxic text.

[1] Lamparth, M. and Reuel, A. Analyzing and editing inner mechanisms of backdoored language models. arxiv:2302.12461, 2023.

### Weakness 4

In Sec. 3, fitting an alpha for each model and instruction type greatly limits usability, especially given that you would need a cleanly separate dataset for all relevant tasks. This is also more difficult for more realistic tasks mentioned in Weakness 1. 

### Weakness 5

This is a minor weakness that did not impact the final review score, but I wanted to explain why I rewarded a 3 instead of the maximum score of 4 for the presentation. I would have rewarded a 4 if figures 2 and 4 had used a larger font size for the axis labels, plot legend, and larger marker sizes. Please make these plots more readable for future versions of this work. 

### Weakness 6

In Sec. 3, it is not clear whether and how the authors calibrated the GPT-4-based scoring from 0 to 9 to construct the quality ratio. The results in Sec. 3 could therefore be strengthened by stating how the GPT-4-based analysis is being conducted (how exactly is a quality rating of 4 different than a quality rating of 5, in-context examples or not, ...) and how accurate and reliable it is (True positive rate, false positive rate, ...). 
 
### Weakness 7 

The stated uncertainties seem very low, especially in Tables 3 and 4 (negligible/0 uncertainties). I think the claims in this work could be strengthened if other methods, e.g., bootstrap resampling, are being used to determine a base uncertainty beyond uncertainties purely determined from the linear probes.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Define “know internally”. The title and various parts of the paper reference this. After reading, I understand it as “encoded instructions in a way correlated to correct responses”. But perhaps the authors disagree with this definition. Defining it specifically would clarify this.

From above: The authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types. So the internal representation seems an encoding of the specific instruction type, rather than a general knowledge of instruction following. Clarifying this earlier would be useful. Readers skimming the paper may have a wrong impression.
soundness: 3
strengths: The paper demonstrates a practical use case of their findings. The instruction-following direction improves model instruction-following success. This can be a useful case way to steer models to follow instructions better.

Also, the method could help us detect when a LLM will not follow a particular instruction. We can detect this even before the LLMs generate a response — because the method works on the first response token.

This is step towards scientifically understand key processes in how LLMs complete practical tasks

The paper reproduces results across different models. The paper conducts experiments across multiple models (Llama, Mistral, and Phi-3) and shows strong results on all 3 models.
summary: The authors explore whether LLMs have an internal representation related to successful instruction following. They train a linear probe on the models' internal representations to predict success/failure cases. This probe works well when tested on held-out tasks. However, when tested on held-out instructions, the probe fails, suggesting that the representation found is not a general "instruction-following" representation.
Potential practical use: Adding this identified direction to a model’s internal representation improves instruction following -- although it is unclear how far this generalizes.
weaknesses: To strengthen the claim of usefulness, compare your method to other interventions e.g. few-shot prompting. Do we observe a similar increase in success rate by just few-shot prompting? In general, the claim of practical usefulness could be investigated in much more detail (with more baselines and tests of generalization to other datasets). 

 You write: “This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged.” You then point to properties of the instruction-following dimension (sensitivity to phrasing modification). I don’t see how this helps *explain* why models sometimes fail to follow clear instructions or why prompt engineering helps.

The authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types.

My impression is that more recent LLMs (e.g. the latest from OpenAI, Anthropic, Meta) are much more reliable at instruction-following and less sensitive to prompts than the models tested here. This is presumably not due to exploiting the instruction-following direction. More generally, it is unclear whether the insights gained from these weaker models would carry over to more recent models. (I’m not expecting the authors to conduct white-box experiments with much larger open models such as Llama-3-405b but it might be possible to gain evidence on this question in other ways). 

Smaller points:
1. “we demonstrate that this dimension generalizes to unseen tasks, indicating that it captures a fundamental aspect of instruction adherence in LLMs.” This claim is ambiguous. What level of generalization performance justified something being a “fundamental aspect”? It would be helpful to make more concrete and falsifiable claims. 

2. I find the claim related to Fig 4 questionable. The authors claim that phrasing plays a critical role in accuracy because the phrasing modification direction aligns more with the instruction-following direction. However, they don’t empirically demonstrate that the phrasing changes affect success rates more than the other factors. To make the argument stronger, consider measuring the impact of each factor. How much does the success rate change when modifying phrasing vs difficulty vs task type?

Meta-review: The authors present a way to understand whether instructions will have the desired effect on instruction following behavior. There were concerns about how reliable these results are (relatively small effect) and applicability to modern models that are more robust. This is very much on the borderline, since the effects are not that impressive -- but given the transparency from the authors without over-claiming, and the overall novelty / relevance of the research direction, it could make a good addition to the conference.

Decision: Accept (Poster)