Do LLMs ``know'' internally when they follow instructions?

Authors: Juyeon Heo, Christina Heinze-Deml, Oussama Elachqar, Kwan Ho Ryan Chan, Shirley You Ren, Andrew Miller, Udhyakumar Nallasamy, Jaya Narain
Primary area: foundation or frontier models, including LLMs
Keywords: Instruction following, Large language models, Linear probing, Representation engineering, Interpretation

Abstract:
Instruction-following is crucial for building AI agents with large language models (LLMs), as these models must adhere strictly to user-provided constraints and guidelines. 
However, LLMs often fail to follow even simple and clear instructions.
To improve instruction-following behavior and prevent undesirable outputs, a deeper understanding of how LLMs' internal states relate to these outcomes is required.
In this work, we investigate whether LLMs encode information in their representations that correlates with instruction-following success—a property we term ``knowing internally''.
Our analysis identifies a direction in the input embedding space, termed the instruction-following dimension, that predicts whether a response will comply with a given instruction.
We find that this dimension generalizes well across unseen tasks but not across unseen instruction types.
We demonstrate that modifying representations along this dimension improves instruction-following success rates compared to random changes, without compromising response quality.
Further investigation reveals that this dimension is more closely related to the phrasing of prompts rather than the inherent difficulty of the task or instructions. 
This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged. 
This work provides insight into the internal workings of LLMs' instruction-following, paving the way for reliable LLM agents.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730576405584

thread_type:
official_review

<thread_object>
creator: reviewer_jgsv

<content>
summary:
The authors explore whether LLMs have an internal representation related to successful instruction following. They train a linear probe on the models' internal representations to predict success/failure cases. This probe works well when tested on held-out tasks. However, when tested on held-out instructions, the probe fails, suggesting that the representation found is not a general "instruction-following" representation.
Potential practical use: Adding this identified direction to a model’s internal representation improves instruction following -- although it is unclear how far this generalizes.

strengths:
The paper demonstrates a practical use case of their findings. The instruction-following direction improves model instruction-following success. This can be a useful case way to steer models to follow instructions better.

Also, the method could help us detect when a LLM will not follow a particular instruction. We can detect this even before the LLMs generate a response — because the method works on the first response token.

This is step towards scientifically understand key processes in how LLMs complete practical tasks

The paper reproduces results across different models. The paper conducts experiments across multiple models (Llama, Mistral, and Phi-3) and shows strong results on all 3 models.

weaknesses:
To strengthen the claim of usefulness, compare your method to other interventions e.g. few-shot prompting. Do we observe a similar increase in success rate by just few-shot prompting? In general, the claim of practical usefulness could be investigated in much more detail (with more baselines and tests of generalization to other datasets). 

 You write: “This discovery also suggests explanations for why LLMs sometimes fail to follow clear instructions and why prompt engineering is often effective, even when the content remains largely unchanged.” You then point to properties of the instruction-following dimension (sensitivity to phrasing modification). I don’t see how this helps *explain* why models sometimes fail to follow clear instructions or why prompt engineering helps.

The authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types.

My impression is that more recent LLMs (e.g. the latest from OpenAI, Anthropic, Meta) are much more reliable at instruction-following and less sensitive to prompts than the models tested here. This is presumably not due to exploiting the instruction-following direction. More generally, it is unclear whether the insights gained from these weaker models would carry over to more recent models. (I’m not expecting the authors to conduct white-box experiments with much larger open models such as Llama-3-405b but it might be possible to gain evidence on this question in other ways). 

Smaller points:
1. “we demonstrate that this dimension generalizes to unseen tasks, indicating that it captures a fundamental aspect of instruction adherence in LLMs.” This claim is ambiguous. What level of generalization performance justified something being a “fundamental aspect”? It would be helpful to make more concrete and falsifiable claims. 

2. I find the claim related to Fig 4 questionable. The authors claim that phrasing plays a critical role in accuracy because the phrasing modification direction aligns more with the instruction-following direction. However, they don’t empirically demonstrate that the phrasing changes affect success rates more than the other factors. To make the argument stronger, consider measuring the impact of each factor. How much does the success rate change when modifying phrasing vs difficulty vs task type?

questions:
Define “know internally”. The title and various parts of the paper reference this. After reading, I understand it as “encoded instructions in a way correlated to correct responses”. But perhaps the authors disagree with this definition. Defining it specifically would clarify this.

From above: The authors ask the question “Do LLMs know internally when they follow instructions”, but they don’t specifically answer it. I feel that the title and the abstract imply yes. But on closer reading, only on page 4, we find that it does not generalize across instruction types. So the internal representation seems an encoding of the specific instruction type, rather than a general knowledge of instruction following. Clarifying this earlier would be useful. Readers skimming the paper may have a wrong impression.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696965937

thread_type:
official_review

<thread_object>
creator: reviewer_uury

<content>
summary:
The authors expand prior work on finding meaningful directions in the latent space of transformer language models to find a direction associated with whether models follow a given instruction or not. 
- For the analysis, they adapt and expand an existing dataset and study four open-source models (7b to 13b parameters). 
- Using linear probes, the authors find directions associated with instruction following across the studied tasks, but not generalizing to hold-out instruction types. 
- They show that manipulating the latent representation along the found direction can improve instruction following success and maintain instruction tuning quality for already successful cases.
- The authors find that the found latent space direction is affected by prompt sensitivity/phrasing of the instructions (less so by task complexity and perplexity)

strengths:
- The goal of finding a way to manipulate latent representations to improve the instruction following of language models (or detect whether instructions are not being followed) has the potential for significant usability and safety impact while being novel. 
- The paper is very well written and the presentation is generally clear.
- The presented methodology is well structured and each section is clearly motivated by the previous results.
- The authors test four open-source models across many different tasks, demonstrating a certain amount of robustness.
- The authors generally do not overclaim their results and are very transparent.

weaknesses:
The authors do not claim that they find a universal direction in these models that fully represents the idea of instruction following. However, to make a significant contribution via usability and understanding of latent representations, this work would benefit from addressing a few weaknesses to enable more robust future work and enhance the usability of the results.
 
### Weakness 1

The studied tasks are somewhat general but the instructions are fairly simple and not necessarily representative of general language model uses. I wonder how well the results of this work might generalize: It is also unclear how well instructions are being followed in a general use case. To use the example of the paper: You might ask a health bot to be aware of your left knee injury when coming up with training plans, but does adding the found latent space direction have side effects like avoiding leg training altogether or creating asymmetric training plans leading to other issues?  The results in this paper could be significantly strengthened by finding applications that allow for a more nuanced analysis of how instructions are being followed after representation engineering to demonstrate the usefulness of their technique. 

### Weakness 2

In Sec. 3, I think an opportunity was missed to strengthen the analysis by demonstrating that the found direction also works in a counterfactual way: Can you use the fitted alpha values and directions for the F2T and T2T tests and only flip the sign of the respective alpha value to check if it also works on (intentional) T2F and F2F? I think this is necessary to demonstrate a minimum of reliable usability and support the claim of safety impact (to potentially get a model to not comply).

### Weakness 3

It is unclear (to me) if using only one direction is sufficient for modeling whether an instruction is being followed or not and even if, whether the presented analysis finds that direction. For example, the prompt sensitivity studies in Sec. 4 affecting the performance could be caused by other meaningful "directions" being entangled in the one you care about. To be more precise: If changing "write a resume for [a] software engineer [...]" to "I want you to write about [a] software engineer resume [...]" makes a difference, how do we know this is not caused by, e.g., confounding the learned directions with the instruction data set with a "dialog in first-person" direction (not unlikely,  given the colorful interpretations modern SAE approaches find).

To link it to another, more concrete use case: Is it possible to circumvent safety training of a language model using the proposed methodology? Although also only necessary and not sufficient, such a more complex example would demonstrate that using only one direction is powerful enough to be useful. 

If one direction is not sufficient, it might be worth testing if a higher-order approach (2D, 3D, ...) might be more robust to find that direction. There are plenty and different works on this, but for example, [1] uses PCA on latent representations to find the one relevant direction to model how small backdoored language models process trigger inputs to switch to toxic text.

[1] Lamparth, M. and Reuel, A. Analyzing and editing inner mechanisms of backdoored language models. arxiv:2302.12461, 2023.

### Weakness 4

In Sec. 3, fitting an alpha for each model and instruction type greatly limits usability, especially given that you would need a cleanly separate dataset for all relevant tasks. This is also more difficult for more realistic tasks mentioned in Weakness 1. 

### Weakness 5

This is a minor weakness that did not impact the final review score, but I wanted to explain why I rewarded a 3 instead of the maximum score of 4 for the presentation. I would have rewarded a 4 if figures 2 and 4 had used a larger font size for the axis labels, plot legend, and larger marker sizes. Please make these plots more readable for future versions of this work. 

### Weakness 6

In Sec. 3, it is not clear whether and how the authors calibrated the GPT-4-based scoring from 0 to 9 to construct the quality ratio. The results in Sec. 3 could therefore be strengthened by stating how the GPT-4-based analysis is being conducted (how exactly is a quality rating of 4 different than a quality rating of 5, in-context examples or not, ...) and how accurate and reliable it is (True positive rate, false positive rate, ...). 
 
### Weakness 7 

The stated uncertainties seem very low, especially in Tables 3 and 4 (negligible/0 uncertainties). I think the claims in this work could be strengthened if other methods, e.g., bootstrap resampling, are being used to determine a base uncertainty beyond uncertainties purely determined from the linear probes.

questions:
Generally, please just respond to my weaknesses above. I'm willing to increase the scores (total, contribution, and soundness), especially if you address the crucial ones related the generalizability and usability with new experiments or strong arguments. 

Q1: Can you explain where the "middle" token was for the different prompts? Was it the middle token of each sequence and thus changing the position for each prompt?

### Comments (Not considered for the review)

- For Figure 2, I feel LDA (fitted on training data) might be a better fit than PCA to show separability, given that LDA actively tries to separate the known classes.
- In lines 213-214, you state that the instruction type generalization values are close to chance. Given the stated 1-sigma uncertainties in Table 1, the presented values are not statistically significantly different (depending on the definition, 2-5 sigma) from random chance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718223626

thread_type:
official_review

<thread_object>
creator: reviewer_wmts

<content>
summary:
This work finds that the representation of token is related to the success rate of instruction following. The authors apply linear probing to identify a specific dimension and use representation engineering (RE) to manipulate the representations. Experimental results show that RE  improves success rate while maintaining quality. Further analysis reveals that phrasing modifications plays a critical role rather than task familiarity or instruction difficulty. This paper provides a deeper understanding to this field.

strengths:
1. This paper provides a new understanding of instruction following, using token representation to interprete LLMs themselves.
2. The experiments are well designed and the writing is easy to follow.

weaknesses:
1. Some of the conclusions in this paper are as expected, e.g. results from Table 1 and Table 2 are related to "Lost in the Middle" phenomenon. [1]
2. The performance improvement brought by this method is limited as shown in Table 4.
3. The IFEval dataset is small and such problem may be addressed by extending instruction following data.

[1] Lost in the Middle: How Language Models Use Long Contexts

questions:
1. Have you tried reverse representation engineering? Will it be worse than random?
2. Why did you focus on the representation of the first token in the last layer? Any insight?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730806226113

thread_type:
official_review

<thread_object>
creator: reviewer_maey

<content>
summary:
This work analyzes and compares LLM internal representations when it can successfully follow instructions, vs. when it fails at following instructions. By training linear probes on representations, authors find a latent direction that predicts instruction following success, and shows it generalizes to unseen tasks and instructions. In addition, this work uses representation engineering to make adjustments along the instruction following dimension they found, and finds these adjustments can improve instruction following performance. Finally, this work also finds that the instruction following dimension is more correlated with how the prompt is phrased, rather than the difficulty of the instruction or the task.

strengths:
This paper is well-structured and the findings are very interesting - while using linear probes to find a latent dimension for a specific phenomenon is not that interesting or original, the follow-up analyses demonstrating that instruction following can be made more accurate with representation engineering, and comparing the correlation of this dimension to different properties of the prompts, are both interesting. In addition, most claims made in the paper are sufficiently backed up with experiments and evidence.

weaknesses:
In section 3, I'm confused what the direction D taken is for different instructions, since the previous section states that linear probes do not generalize well across instructions. Does this mean representation engineering promotes different directions for different instructions, or is there a universal direction that enhances all instructions generally? It would be helpful to clarify this in the paper.
For section 4 as well, one instruction type is chosen to perform the analysis on correlations. Could you elaborate on how much the findings in this section generalizes to other instruction types? Or could there be instruction types whose representation is more correlated with task familiarity rather than phrasing?

questions:
- At the start of section 2, it would be helpful to elaborate what the IFEval dataset is used for 
- Make text in Figure 1 bigger
</content>
</thread_object>

</thread>
<thread>
creation date:
1730857347264

thread_type:
official_review

<thread_object>
creator: reviewer_mszm

<content>
summary:
This paper investigates the internal representation of LLMs and identifies a "dimension" that corresponds to their success in instrution-following. They show that manipulating this dimension could improve success in instruction-following without degrading the quality of the outputs. They also measure correlation of the dimension with different perturbations and show that it is more sensitive to the phrasing of the prompt than the task difficiulty or faimiliarity. 

'

strengths:
* Paper offers very interesting and operationalizable insights about the internal representations of LLMs.
* Paper tells a very good, satisfying story: from identifying the dimension, to manipulating and interpreting it.

weaknesses:
- The methodology description lacks clarity due to insufficient mathematical notation. It is especially difficult to follow the procedures outlined in Sections 3 and 4. The authors should clearly define mathematical objects, such as "input representation," and provide precise equations for the computations (e.g., lines 370-374). Specifically, given a prompt \( x_1, x_2, ..., x_n \) of n words and an output \( y_1, y_2, ..., y_m \) of  m words, what is the "input representation" as a function of these tokens? It appears to be computed solely from the output words.

* The terminology is confusing. The term "instruction" would be more accurately described as "constraints." It’s challenging to distinguish "instruction" from "task," as "instruction" could imply the entire command given to the model. Additionally, the meaning of "dimension" is unclear. Does it refer to a hypothetical concept rather than an actual dimension of a hidden vector computed by the model?

* The experiments were conducted on a single, relatively small dataset, raising questions about whether the findings will generalize.

questions:
- See weaknesses
- What is the motivation behind the update in line 303-304? Could you provide more intuition on what is being done to the input representation there?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731877849665

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General response
We are grateful for the reviewers’ thoughtful and constructive feedback on our submission. The reviewers highlighted key strengths of our paper, including its novel contributions to understanding instruction-following in LLMs by analyzing their internal representations. This allows us to detect potential instruction-following failures early, even before a response is generated, using early-stage representations. The paper’s well-structured methodology, clear presentation, and rigorous experiments across multiple models were also praised. Reviewers noted that the work tells a “good, satisfying story,” providing a meaningful step toward scientifically understanding how LLMs follow instructions. 

---
## Q. Data Scope
> *Reviewer MsZm: The experiments were conducted on a single, relatively small dataset, raising questions about whether the findings will generalize.*
> 

> *Reviewer wmTS: The IFEval dataset is small and such problem may be addressed by extending instruction following data.*
> 

> *Reviewer UuRY: The studied tasks are somewhat general but the instructions are fairly simple and not necessarily representative of general language model uses. I wonder how well the results of this work might generalize: It is also unclear how well instructions are being followed in a general use case. To use the example of the paper: You might ask a health bot to be aware of your left knee injury when coming up with training plans, but does adding the found latent space direction have side effects like avoiding leg training altogether or creating asymmetric training plans leading to other issues? The results in this paper could be significantly strengthened by finding applications that allow for a more nuanced analysis of how instructions are being followed after representation engineering to demonstrate the usefulness of their technique.*
> 

Thank you for your concern about datasets. We do agree it would be beneficial to include real-world datasets to validate the findings. However, here we would like to emphasize why we choose IFEval as our primary dataset instead of using real-world dataset with different contexts and domains. 

**First, we select IFEval to focus on our scope which is ‘single, simple, and non-ambiguous instructions’.** Real-world datasets often involve complex, ambiguous, or multi-instruction prompts, which can conflate multiple factors affecting instruction-following. As an initial exploration of the geometry of LLM representations in instruction-following, we chose to focus on single, simple, and verifiable instructions  to ensure clarity and disentangle multiple factors. The IFEval dataset is well-suited for this purpose, as it provides 25 distinct types of simple and clear instructions that align with our goal of establishing a robust baseline.

**Second, we want to avoid evaluator-induced uncertainties.** Most real-world tasks and benchmark datasets rely on LLM-based evaluators to determine whether a response follows an instruction. However, LLM-based evaluators may introduce their own uncertainties or make errors in assessing success or failure, which could obscure our analysis on representations of the tested models. The IFEval dataset avoids this issue by including instructions with **deterministic evaluation programs** that objectively verify compliance. For instance, an instruction like “please do not include keywords: …” can be automatically validated using a simple program to check for the presence of those keywords. This feature eliminates ambiguity in evaluation and allows us to isolate the directions related specifically to instruction-following.

One of our main contribution is the careful design of data settings specifically tailored to analyze internal states of LLMs in instruction-following contexts. **While IFEval serves as an ideal starting point for this research, we hope our work inspires future efforts to tackle analysis of LLMs in more complex, real-world instruction-following tasks (lines 530-532).**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732739588386

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# **About the revision**
We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions, which have helped us improve the clarity and scope of our paper. Below, we summarize the key updates we have made in this revision based on the feedback received.

---

### **Clarifications**

**Terminology (Abstract and Instruction)**

We have revised the abstract and introduction to clarify the interpretation of the title *“Do LLMs know internally when they follow instructions”*. Specifically, we define *“know internally”* as the encoding of information in LLM representations that correlates with instruction-following success. Additionally, we clarify that while the observed instruction-following direction generalizes to unseen tasks, it does not generalize to unseen instruction types (Reviewer jGSV).

**Mathematical Notation in Section 2.2**

To improve clarity, we have updated Section 2.2 with precise mathematical notations describing the representations analyzed at different stages of response generation, addressing Reviewer MsZm’s concern about insufficient clarity.

### **Discussion on reviews**

**Discussion on Dataset Scope**

We have incorporated a discussion on the dataset scope from the rebuttal into Section 2.1 and Appendix A.7. This addresses concerns about the generalizability of our findings and explains the rationale for using the IFEval dataset as a controlled benchmark for analyzing instruction-following.

**Reverse Representation Engineering Results**

We have added preliminary results for reverse representation engineering to Appendix A.8. These experiments explore whether reverse adjustments to the instruction-following dimension degrade instruction adherence and provide insights into the robustness of our findings. We plan to extend this analysis in future revisions.

---

We hope these revisions address key points raised by reviewers and enhance the overall clarity and impact of our work. Thank you for your time and consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a way to understand whether instructions will have the desired effect on instruction following behavior. There were concerns about how reliable these results are (relatively small effect) and applicability to modern models that are more robust. This is very much on the borderline, since the effects are not that impressive -- but given the transparency from the authors without over-claiming, and the overall novelty / relevance of the research direction, it could make a good addition to the conference.