Controllable Context Sensitivity and the Knob Behind It

## Abstract

Abstract When making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge.
Choosing how sensitive the model is to its context is a fundamental functionality, as it enables the model to excel at tasks like retrieval-augmented generation and question-answering.
In this paper, we search for a knob which controls this sensitivity, determining whether language models answer from the context or their prior knowledge.
To guide this search, we design a task for controllable context sensitivity .
In this task, we first feed the model a context ( Paris is in England ) and a question ( Where is Paris? ); we then instruct the model to either use its prior or contextual knowledge and evaluate whether it generates the correct answer for both intents (either France or England ).
When fine-tuned on this task, instruction-tuned versions of Llama-3.1, Mistral-v0.3, and Gemma-2 can solve it with high accuracy (85-95%).
Analyzing these high-performing models, we narrow down which layers may be important to context sensitivity using a novel linear time algorithm.
Then, in each model, we identify a 1-D subspace in a single layer that encodes whether the model follows context or prior knowledge.
Interestingly, while we identify this subspace in a fine-tuned model, we find that the exact same subspace serves as an effective knob in not only that model but also non-fine-tuned instruct and base models of that model family.
Finally, we show a strong correlation between a model’s performance and how distinctly it separates context-agreeing from context-ignoring answers in this subspace.
These results suggest a single subspace facilitates how the model chooses between context and prior knowledge, hinting at a simple fundamental mechanism that controls this behavior.

## 1 Introduction

Language models are often prompted with a query and preceding context, e.g., in settings of in-context learning, retrieval-augmented generation, or document analysis.
In such scenarios, the language model needs to integrate information from both the context and its prior knowledge stored in its parameters.
In some cases, we may prefer the model to rely more on the context, e.g., to avoid hallucinating responses that may be based on outdated prior knowledge (Zhang et al., ); however, in other cases, we may prefer the model to rely more on its prior knowledge, e.g., to avoid being misled by misinformation provided in the context (Hong et al., ).
As a motivating example, consider a document analysis setting in which a language model is asked to help understand an opinion article in a newspaper.
It might first be asked to summarize, e.g., What is the main argument of this article?.
In this case, the model should rely heavily on the context, i.e., the text of the article.
Then, one might ask: What are some criticisms of this argument?.
To answer this, the model ought to be skeptical; an opinion article may be written very authoritatively as if its arguments are established fact, or it may make some misleading claims to support its argument.
To answer this critically, the language model must draw more upon its prior knowledge of the issue and related opinions than blindly following the context.
More broadly, because the degree of context sensitivity depends highly on the use case, it would be desirable to be able to specify how much and whether the model should be influenced by the context versus its prior knowledge.

Studies on the tension between context and prior knowledge have primarily focused on the setting of *knowledge conflicts* (Longpre et al., ), in which a given context directly contradicts information assumed to be in a model’s prior knowledge about a given query.
For example, a language model trained on a sufficient amount of data should be able to reply to the query What’s the capital of France? with Paris.
However, if the context The capital of France is London. is prepended to the query, the model needs to decide whether to respond based on the context ( London) or its prior knowledge ( Paris).
Prior studies (Longpre et al., ; Du et al., ; Monea et al., ; Ortu et al., ) have shown that models will prefer drawing from context for some questions and prior knowledge from others; however, the exact manner by means of which the model selects among the sources is not well-understood.

To this question of *how*, we hypothesize that there is a simple fundamental mechanism within the language model that facilitates the binary decision of whether to rely on the context or the prior knowledge.
To guide our search for such a mechanism, we design and execute a structured recipe.
First, we create the controllable context sensitivity (CCS) task which augments the standard knowledge-conflict setting with an *intent*, such as Ignore the context or Listen to the context.
By disambiguating whether the model should follow context or prior knowledge through a simple addition to the prompt, we are able to identify and evaluate its behavior in both modes for the same context–query pair.
We adapt models for this task using *fine-tuning* and *in-context learning*, then evaluate them on in-domain and out-of-domain test sets to assess whether they have developed a deeper ability to choose between context and prior knowledge beyond surface-level heuristics.
In our case study on the Llama-3.1-8B family (Dubey et al., ), we find that both fine-tuning and in-context learning are moderately effective, with models excelling on in-domain test sets and significantly improving over zero shot baselines on out-of-domain test sets.

Armed with models that can perform the CCS task reasonably well, we then explore the mechanisms that facilitate their behavior in this task.
We hypothesize that for a model to solve this task, it must execute at least three high-level steps (not in any particular order): extracting an answer from prior knowledge, extracting an answer from the context, and deciding to answer with the context answer or the prior answer. We then seek to identify layers that may contain the model’s computations that are aligned with each step.
To do so, we develop an algorithm that uses tools from mechanistic interpretability to find a targeted subset of layers at which activation patching (Meng et al., ) can switch a model from preferring the answer in the context to preferring the answer in its prior knowledge and vice versa.
Then, building on ideas from distributed alignment search (Geiger et al., ), we identify a knob for the model’s decision between following context or prior in the form of a one-dimensional subspace.
Despite locating such a knob on an instruction-tuned model fine-tuned on this task that states explicit intents, we show that it is even effective on non-finetuned and base models of the same family for prompts that do not state the intent.

Furthermore, we show strong evidence that for models good at the CCS task, the two intents correspond to two distinct values in that subspace, while bad models fail to exhibit this distinction.
We repeat this process for Gemma-2 9B and Mistral-v0.3 7B to find a similar story.
Our results suggest that, across many types of large language models, one fundamental mechanism represented by a value in a one-dimensional subspace facilitates their ability to decide between following the context or its prior knowledge.
These findings move toward developing more robust language models with controllable levels of reliance on context and prior knowledge.
They further highlight how investigating models at a mechanistic level can yield high-quality interventions to control a model’s behavior.

## 2 Context and Prior Knowledge

Prior Knowledge in Language Models.
Many prior studies have noted that LMs exhibit remarkable capabilities at answering questions depending on prior knowledge, such as factual recall (e.g., What’s the capital of France?).
When queried, language models tend to produce plausible answers suggesting they may encode some level of knowledge about these entities (Brown et al., ; Petroni et al., ; Roberts et al., ; Geva et al., ).
This knowledge is encoded within the model’s weights during training as the model is exposed to mentions of these entities in its pretraining data (Xu et al., ; Zhou et al., ).
Pretraining can lead to not only learning facts, but also memorizing specific strings (Carlini et al., ; Stoehr et al., ).

Influence of Context on Language Models: Promises and Perils.
Models might also be prompted with context in addition to the query.
For many use cases, the information provided in the context is critical to the model solving the task effectively, such as in:
(a) In-context learning(Brown et al., ), in which a user provides several demonstrations of a task in context, which the model uses to answer the user’s query;
(b) Retrieval-augmented generation(Lewis et al., )and open-book question-answering (Mihaylov et al., ; Kasai et al., ), in which potentially relevant information from documents is included in context to help the model answer a user’s query;
(c) Interactive dialogue/chat(Vinyals & Le, ; OpenAI, ), in which a user converses with a language model, often with several turns of dialogue; and
(d) Text annotation(Ziems et al., ), in which a model is given a passage of text (e.g., a tweet) and asked to annotate the passage’s sentiment, toxicity, coherence, inter alia—possibly in relation to a specific entity mentioned in the passage.

However, other use cases may be better served by ignoring the context to some degree, i.e., in:
(a) combating jailbreaking (Yu et al., ), e.g., ignoring attempts by the user to override a model’s built-in behaviors; and
(b) resilience to misinformation (Hong et al., ), e.g., avoiding integrating potentially incorrect information provided in context when answering a user’s query.
In all of the above settings, the model has two sources from which it can draw upon when formulating a response: the context, and its knowledge about the query encoded during training.
Being able to control a model’s sensitivity to context in an application-dependent manner is key to their robust use.

Controlling Model Sensitivity to Context.
Several studies have proposed interventions to reduce dependency on prior knowledge and favor in-context information.
These approaches include prompting
(Zhou et al., ; Onoe et al., ), modifying training data (Wang et al., ), fine-tuning (Li et al., ), and activation-level interventions (Li et al., ; Stoehr et al., ) at inference time.
While Li et al. () further aims for some level of controllable context sensitivity, i.e., by attempting to ignore irrelevant context, they do not allow for any form of explicit controllability by the user.
Neeman et al. () train a model to predict both an answer based on context and one on prior knowledge.

## 3 Identifying Mechanisms in Neural Networks

According to the linear subspace hypothesis (Bolukbasi et al., ; Vargas & Cotterell, ; Wang et al., ), model representations can encode concepts as low-dimensional linear subspaces.
Based on this hypothesis, much prior work has explored how various concepts including truthfulness (Marks & Tegmark, ; Li et al., ), humor (von Rütte et al., ), sentiment (Tigges et al., ), and refusal (Arditi et al., ) are encoded within model representations.
Beyond identifying the subspace representations of these concepts, researchers have further intervened on those subspaces to control the model behavior, by using the subspace to additively steer (i.e., adding a vector to a model representation) (Rimsky et al., ; Turner et al., ; Zou et al., ; Ravfogel et al., ).
Several methods can be used to identify a subspace representing a concept, including distributed alignment search (Geiger et al., ), LEACE (Belrose et al., ), and difference in means (Marks & Tegmark, ).

## 4 How to Find the Knob Behind Context Sensitivity

### 4.1 Designing the Task

First, we define the task of controllable context sensitivity, which should include *minimally contrastive* example pairs. Each pair has the same context ${\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}$ and query ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}}$, differing only in whether the model should follow the context or prior knowledge. These pairs allow us to compare the model’s internal states when it follows context versus prior knowledge, with all else equal.

We formalize our setup for this task as follows.
Consider a language model $p$ over an alphabet $\Sigma$, i.e., $p$ is a distribution over the Kleene closure $\Sigma^{*}$.
Further, consider a distinguished subset ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\mathcal{Q}}}\subset
\Sigma^{*}$ corresponding to licit queries and a distinguished subset ${\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}\mathcal{C}}\subset\Sigma^{*}$ corresponding to licit contexts. Let $\varepsilon$ be the empty string.
For a given query ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}}\in{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\mathcal{Q}}}$, e.g., What is the capital of France?, and context ${\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}\in{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}\mathcal{C}}$, e.g., The capital of France is London., let ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)\in\Sigma^{*}$ be the answer to ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}}$ when ignoring the context ( Paris) and ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})\in\Sigma^{*}$ be the answer to ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}}$ that agrees with context ${\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}$ ( London), respectively.
Let ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}\in\left\{{\color[
rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}
{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right\}$ denote an intent, i.e., a signal for whether to agree with the context (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$) or the model’s prior knowledge (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$).
Let $F\colon{\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\mathcal{Q}}}\times{
\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}\mathcal{C}}\times\left\{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right\}\rightarrow\Sigma^{*}$ be a *formatting function* mapping from a query, context, and intent to a formatted prompt, e.g., “*Context: The capital of France is London/n Instruction: Only listen to the context/n Query: What is the capital of France?*”.
Then, let $\mathcal{S}_{\text{trn}}\subset{\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{
\mathcal{Q}}}\times{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[
named]{pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}\mathcal{C}}$ and $\mathcal{S}_{\text{tst}}\subset{\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{
\mathcal{Q}}}\times{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[
named]{pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}\mathcal{C}}$ be disjoint subsets of the queries and contexts to be used for training and testing, respectively. We train our models on the strings $F\left({\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right)\cdot{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ and $F\left({\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right)\cdot{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$, where $\left({\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\right)\in\mathcal{S}_{\text{trn}}$ and $\cdot$ means string concatenation.

### 4.2 Identifying Model Behavior

Adapting a Model to this Task.
To inspect the mechanism behind the model behavior, we first need a model capable of controllably following the context or prior knowledge.
As such, we adapt a language model to solve the task defined in § 4.1 with two methods: first, we consider fine-tuning $p$ using a standard next-token prediction objective of negative log-likelihood on the training set $\mathcal{D}_{\text{trn}}$, and second, we consider including samples from the training set as few-shot demonstrations for in-context learning.

Evaluating Controllable Context Sensitivity.
We evaluate a model’s ability to controllably choose between context and prior knowledge using *pair-accuracy*.
An example is correct only if the model outputs the correct answer to a given query ${\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}}$ and context ${\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}$ for both intents (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ and ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$).
That is, given a model $p$ and dataset $\mathcal{S}$,

$$ $\displaystyle\mathrm{{\mathrm{PairAcc}}}(p,\mathcal{S})$ (1) $\displaystyle\quad=\frac{1}{\lvert\mathcal{S}\rvert}\sum_{({\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}})\in\mathcal{S}}\mathbbm{1}\{ \operatorname*{greedy}_{{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named] {pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}\bm{a}}\in\Sigma^{*}}p({\color[rgb ]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8359375,0.3671875,0}\bm{a}}\mid F\left({\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{ 0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right))={\color[rgb]{ 0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875} \definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{ q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{ pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})\}\mathbbm{1}\{ \operatorname*{greedy}_{{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named] {pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}\bm{a}}\in\Sigma^{*}}p({\color[rgb ]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8359375,0.3671875,0}\bm{a}}\mid F\left({\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{ 0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right))={\color[rgb]{ 0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875} \definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{ q}}},\varepsilon)\}$ $$

where $\operatorname*{greedy}_{{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}\bm{a}}\in\Sigma^{*}}$ denotes the output of greedy decoding.

### 4.3 Identifying Important Layers

Next, we need to identify layers in the model where the target behavior appears to emerge.
Consistent with prior work (Jin et al., ), we posit that for a model to succeed at this task, it must be able to execute at least three steps (not necessarily in this order):
(i) extract the answer from the model’s prior knowledge;
(ii) extract the answer from the context; and
(iii) decide whether to answer according to the context or the prior knowledge according to the given intent.
Note that, under the framing of Geiger et al. (), these would be considered causal variables in a high-level model.
Without specifying an exact causal graph, we posit these must be components in any reasonable one.
We use tools from mechanistic interpretability to identify the layers at which the model appears to implement each of these steps.

Intervention-based Interpretability.
Intervention-based interpretability techniques like activation patching are commonly used to identify which activations in a model are critical for a given task (Meng et al., ).
Intuitively, the idea is that if intervening at some set of intermediate states can change a model’s output behavior for a given task, then those intermediate states likely play some critical role in the model’s ability in that task.
Activation patching, specifically using interchange interventions (Geiger et al., ), involves using two strings that differ only with respect to the task of interest.
For example, to identify intermediate activations that may encode the intent of a prompt, we can consider two input strings that share the same query and context but differ in their intent.
For a given model, $p$, we define a source string, ${\bm{s}}\in\Sigma^{*}$, and a target string, ${\bm{t}}\in\Sigma^{*}$.
During the forward pass of $p\left({\bm{t}}\right)$, we replace a subset of intermediate activations with the corresponding activations from $p\left({\bm{s}}\right)$ and observe the resulting effect on model internals and the output distribution of the patched $p\left({\bm{t}}\right)$.
We only patch at the last token because this has been shown in previous work to be most informative to determining the next token (Yu et al., ; Jin et al., ; Stoehr et al., ; Monea et al., ).

We also only patch the outputs of the multi-head attention (MHA) components in a transformer block; the intuition behind this choice is that this component ought to integrate information from the context into the residual stream of the last token. Interchanging these output activations allows us to analyze what kind of information is written on the residual stream and whether it has a causal effect on the model internals and the output distribution.
By searching over different subsets of intermediate activations, we can identify those with the greatest impact on task performance and thus facilitate critical functionality.

Iteratively Searching For Important Components.
However, searching for a small subset of MHA components at the last token position to patch is nontrivial because it is over an exponentially large space (i.e., $2^{L}$, where $L$ is the number of layers in the model).
Thus, we use an iterative search algorithm to build up a subset of important components, which requires $O(L)$ forward passes.
In this algorithm, we use the *Token Identity Patchscope* (TIP) to observe the model behavior at intermediate states (Ghandeharioun et al., ).(^1^11TIP can interpret the information in a model’s residual stream at intermediate layers by using the model to map from the residual stream at a given layer and token index to a distribution over tokens that best represents the information stored in that intermediate state. This approach can also be viewed as a variant of the SelfiE method Chen et al. (). While other alternatives for interpreting intermediate states could also be used here (e.g., probing (Tenney et al., ), LogitLens (nostalgebraist, ), and TunedLens (Belrose et al., )), we prefer TIP because it performs better than these other methods.)
Specifically, we use it to identify the model’s likelihood on the context and prior answers at intermediate layers and choose a subset of layers to patch that would push the model to prefer the desired answer.
Given a dataset of source and target pairs, the algorithm has two main steps.
First, it identifies a continuous *base range* of layers at which patching MHA components enables decoding the source answer from the residual stream.
Then, it finds *inhibition layers*, which suppress the source answer at later layers, by iteratively patching MHA components at these layers until the source answer has a high probability.
We provide Python-esque pseudocode for our search algorithm in § A.1.

**Table 1: Patching Setup: To investigate the model’s internal mechanisms, we use three distinct patching setups ($\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named] {pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}$, $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$, and $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}$) to address our research questions. For all datasets, an example consists of a source prompt ${\bm{s}}$, source answer ${\bm{a}_{\bm{s}}}$, target prompt ${\bm{t}}$, and target answer ${\bm{a}_{\bm{t}}}$. $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named] {pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}$, has two subvariants: $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named] {pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{p}}}$ and $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named] {pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}}}$, which represent different directions of the intervention.**
| Dataset | Question & Description | ${\bm{s}}$ | ${\bm{a}_{\bm{s}}}$ | ${\bm{t}}$ | ${\bm{a}_{\bm{t}}}$ |
| --- | --- | --- | --- | --- | --- |
| $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]<br>{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{<br>0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{<br>0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.87109375,0.5625,0.01953125}{\bm{c}}}}$<br>$\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]<br>{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{<br>0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{<br>0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0078125,0.62109375,0.44921875}{p}}}$ | Where is ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ computed?<br>Only ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ differs<br>Source ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}={\color[rgb]{<br>0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$<br>Only ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ differs<br>Source ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}={\color[rgb]{<br>0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ | The capital of France is London<br>Ignore the context<br>What is the capital of France?<br>The capital of France is London<br>Only listen to the context<br>What is the capital of France? | Paris<br>London | The capital of France is London<br>Only listen to the context<br>What is the capital of France?<br>The capital of France is London<br>Ignore the context<br>What is the capital of France? | London<br>Paris |
| $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$ | Where is ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}<br>\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{<br>q}}},\varepsilon)$ computed?<br>${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}={\color[rgb]{<br>0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$, different ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}<br>\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{<br>q}}},\varepsilon)$ | The capital of France is London<br>Ignore the context<br>What is the capital of France? | Paris | The capital of France is London<br>Ignore the context<br>What is the capital of Italy? | Rome |
| $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}$ | Where is ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}<br>\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{<br>q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ computed?<br>${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}={\color[rgb]{<br>0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$, different ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}<br>\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{<br>q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ | The capital of France is London<br>Only listen to the context<br>What is the capital of France? | London | The capital of France is Rome<br>Only listen to the context<br>What is the capital of France? | Rome |

Patching Setups Per Subquestion.
We wish to address the three subquestions:
(i) Where is the intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ computed?
(ii) Where is ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ computed?
(iii) Where is ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ computed?
Answering each subquestion will demand applying the search algorithm described above on a specific patching setup, i.e., dataset, per subquestion.
Each patching setup consists of a tuple containing a source string, the answer associated with the source, a target string, and the answer associated with the target. The relationship between the source and target components of an example will depend on the question we aim to answer.
Table 1 outlines the specific patching setups for each subquestion.
First, $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$ and $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}$ hold the context and query constant but vary the intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$, enabling us to investigate how the model processes different intents.
We define $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}=\{\left(F\left({\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon),F\left({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})\right)\}_{\left({
\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor
}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\right)\in\mathcal{S}_{\text{tst}}}$ and $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}=\{(F\left({\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}),F\left({\color[
rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon))\}_{\left({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}\right)\in\mathcal{
S}_{\text{tst}}}$.
$\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$ includes tuples where both the source and the target share the intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}={\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$, but differ in the prior answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ they suggest, $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}=\{(F\left({\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon),F\left({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}^{\prime}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}\left({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}^{\prime}}},\varepsilon\right))\}_{\left({\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\right)\in\mathcal{S}_{\text{tst}},{
\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor
}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}^{\prime}}}\in{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\mathcal{Q}}}\setminus\{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}}\}}$.
This allows us to evaluate how patching alters the model’s response with respect to ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ and discern how the model computes ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$.
Similarly, in $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}$ we explore how the model computes ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$, $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}=\{(F\left({\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}),F\left({\color[
rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}^{\prime}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right),{\color[rgb]{
0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}\left({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}^{\prime}}}\right))\}
_{\left({\color[rgb]{0.0078125,0.62109375,0.44921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{q}}},{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\right)\in\mathcal{S}_{\text{tst}},{
\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.87109375,0.5625,0.01953125}{\bm{c}^{\prime}}}\in{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}\mathcal{C}}\setminus\{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\}}$.

### 4.4 Identifying the context-controllability subspace
feature

Learning the Context-versus-Prior Subspace.
Once we determined a subset of model components that might contain the mechanism which facilitates the functionality of deciding between answering from the context or prior knowledge, we can zoom further in and search for whether this functionality can be encoded in a small-dimensional subspace in these components.
According to the linear subspace hypothesis (Bolukbasi et al., ; Vargas & Cotterell, ), there exists a linear subspace $\mathcal{F}\subset\mathbb{R}^{D}$ which encodes the information about a specific concept.
In our case, the concept of interest is whether the model follows the answer from the context or its prior knowledge.
Since this is a simple binary concept, we hypothesize that a rank-1 subspace within the subset of model components found via methods described in the previous section encodes this concept.

Let $\ell$ be the last layer in the *base range* found by our search algorithm.
Let $\bm{h}^{\ell}\in{\mathbb{R}}^{D}$ denote the activation at layer $\ell$.
We learn a rank-1 orthogonal projection matrix $\bm{P}\in\mathbb{R}^{D\times D}$ which is used to project $\bm{h}^{\ell}\in\mathbb{R}^{D}$ onto a one-dimensional subspace $\mathcal{F}$ of $\mathbb{R}^{D}$, encoding the intent. We parameterize $\bm{P}=\bm{u}\bm{u}^{\top}$, where $\bm{u}\in\mathbb{R}^{D}$ is the basis of the subspace and the norm of $\bm{u}$ is forced to 1, see App. D for a more detailed explanation of the parameterization of $\bm{P}$.

Given a tuple $\left({\bm{s}},{\bm{a}_{\bm{s}}},{\bm{t}},{\bm{a}_{\bm{t}}}\right)\in\mathcal{
D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}\cup\mathcal{D}_{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$, we define $\bm{h}_{{\bm{s}}}^{\ell}$ to be the last token residual stream after layer $\ell$ of the forward pass $p\left({\bm{s}}\right)$, and similarly, $\bm{h}_{{\bm{t}}}^{\ell}$ for $p\left({\bm{t}}\right)$. To learn $\bm{P}$, we freeze the parameters of $p$ and patch the forward pass of $p\left({\bm{t}}\right)$ as follows:

$$ $\displaystyle\bm{h}_{{\bm{t}}}^{\ell}$ $\displaystyle=(\bm{I}-\bm{P})\bm{h}_{{\bm{t}}}^{\ell}+\bm{P}\bm{h}_{{\bm{t}}}^ {\ell}$ (normal decomposition) (2a) $\displaystyle\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ $\displaystyle\triangleq(\bm{I}-\bm{P})\bm{h}_{{\bm{t}}}^{\ell}+\bm{P}\bm{h}_{{ \bm{s}}}^{\ell}$ (patched decomposition) (2b) $$

Eq. 2a expresses that we can decompose $\bm{h}_{{\bm{t}}}^{\ell}$ into
(i) the sum of the component representing our concept of interest ($\bm{P}\bm{h}_{{\bm{t}}}^{\ell}$) and
(ii) its orthogonal complement, the component which represents other information ($(\bm{I}-\bm{P})\bm{h}_{{\bm{t}}}^{\ell}$).
Then, in Eq. 2b, $\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ is constructed by replacing the component in $\bm{h}_{{\bm{t}}}^{\ell}$ representing our concept of interest ($\bm{P}\bm{h}_{{\bm{t}}}^{\ell}$) with the component in $\bm{h}_{{\bm{s}}}^{\ell}$ representing the concept ($\bm{P}\bm{h}_{{\bm{s}}}^{\ell}$).
Thus, if $\bm{P}$ projects onto a subspace that encodes the concept, then the representation $\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ encodes the intent concept from $\bm{h}_{{\bm{s}}}^{\ell}$ and all other aspects from $\bm{h}_{{\bm{t}}}^{\ell}$.

We denote $\widetilde{p}_{\ell}(\cdot;\bm{P},{\bm{s}})$ to be the language model with activation $\bm{h}_{{\bm{t}}}^{\ell}$ replaced by $\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ as defined in Eq. 2b.
We construct a training set $\{\left({\bm{s}}_{n},\bm{a}_{{\bm{s}}_{n}},{\bm{t}}_{n},\bm{a}_{{\bm{t}}_{n}}
\right)\}_{n=1}^{N}\subset\mathcal{D}_{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}\cup\mathcal{D}_{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$.
As can be seen in Tab. 1, this dataset contains matched pairs ${\bm{s}}_{n},{\bm{t}}_{n}$ which differ only in the specified intent.
Then, to learn $\bm{P}$ which well-represents our concept, we minimize the following objective over the training set:

$$ $\displaystyle J_{\ell}(\bm{P})=-\frac{1}{N}\sum_{n=1}^{N}\log\widetilde{p}_{ \ell}(\bm{a}_{{\bm{s}}_{n}}\mid{\bm{t}}_{n};\bm{P},{\bm{s}}_{n})$ (3) $$

That is, we minimize the cross-entropy loss between the language model when patched with $\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ and the label ${\bm{a}_{\bm{s}}}$.
Since ${\bm{s}}_{n}$ and ${\bm{t}}_{n}$ always have different intents ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$, but share the same context and query, we are effectively optimizing for a subspace where replacing the subspace component of ${\bm{t}}_{n}$ with the corresponding component of ${\bm{s}}_{n}$ leads to an answer that reflects the flipped intent.

Controlling Model Behavior Using the Context-versus-Prior Subspace.
After learning an orthogonal projection matrix $\bm{P}$ to project a vector into the context-versus-prior subspace, we can control the model’s behavior by setting the subspace component based on the input intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$. To do this we define a *function* $c:\left\{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}},{\color[
rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}
{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right\}\rightarrow\mathbb{R}$ that acts as a scalar for the basis $\bm{u}$ of $\mathcal{F}$ and returns a constant corresponding to one of the two intents.(^2^22$\bm{P}$ is redundant in the second term of Eq. 4 since $\bm{P}\bm{u}c(w)=\bm{u}\bm{u}^{T}\bm{u}c(w)=\bm{u}c(w)$, but is included for consistency.)

$$ $\displaystyle\widetilde{\bm{h}}^{\ell}_{{\bm{t}}}$ $\displaystyle\triangleq(\bm{I}-\bm{P})\bm{h}_{{\bm{t}}}^{\ell}+\bm{P}\bm{u}c \left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}\right)$ (4) $$

The function c represents the knob to steer which behavior to follow. A successful static intervention on a learned subspace $\mathcal{F}$ implies that we have not only identified a one-dimensional subspace representing intent but also determined how to manipulate it manually.
We evaluate the effectiveness of a static intervention using the *pair-accuracy*.

## 5 Case Study: Llama-3.1 8B

We now describe detailed empirical results in executing the recipe to identify the mechanism behind controllable context sensitivity as described in § 4.
Results for other models can found in § 6 and App. E.

### 5.1 Task Setup

Datasets.
Following the task formulation in § 4.1, we construct our intent-augmented datasets, CCS-BaseFakepedia, CCS-MultihopFakepedia, and CCS-Arithmetic, based on the query-context pairs in BaseFakepedia, MultihopFakepedia, and Arithmetic, respectively (Monea et al., ).
BaseFakepedia is a knowledge-conflict dataset from Wikipedia with queries across 23 relation types (e.g., Norway’s capital city or Mac Pro, a product created by) and paragraphs generated by a large language model that directly provide counterfactual answers.
MultihopFakepedia is similar to BaseFakepedia, but instead of the counterfactual answer being directly stated in the paragraph, it is implied via an extra hop of reasoning (e.g., London is the capital of France. Tunis is in the same country as London. What country is Tunis in?).
Arithmetic is a synthetically-generated dataset whose queries are simple arithmetic expressions using the operators $\{+,-,\times,\div,\exp\}$ and contexts are reassignments of subexpressions to another value which results in a counterfactual answer.
For example, given the query (5 + 1) / 2 = and the context 5 = 9, the prior-agreeing answer would be 3, while the context-agreeing answer would be 5. In our experiments, we limit expressions to a depth of 2, i.e., two operators, with input and output numbers between 0 and 9.

Intent Format.
We also format the intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}\in\{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}},{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\}$ in two different ways in order to probe the model’s robustness to different formulations of the same query, context, and intent.
First, the *instruction* intent format () expresses the intent as a string instruction, e.g., Ignore the context in answering the query. or Only consider the context in answering the query.
Second, the *weight* intent format () expresses the intent as a context weight, e.g., Context weight: 0. or Context weight: 1.

### 5.2 Adapting Models to the Task

Training.
We investigate the degree to which instruction-tuned Llama-3.1 8B (Dubey et al., ) can solve this task after adapting it in two ways.
First, we fine-tune (FT) models on the CCS-BaseFakepedia train set for 2048 examples using QLoRA on the attention components of the full network.
Second, we also use in-context learning (ICL) with 10 examples from CCS-BaseFakepedia prepended to the prompt.
See App. B for more details on training hyperparameters.

Evaluation.
We are interested in two forms of generalization: first, robustness to different datasets, and second, robustness to different intent formats.
For the former, we evaluate whether a model trained on CCS-BaseFakepedia can perform well on test splits from CCS-BaseFakepedia, CCS-MultihopFakepedia, and CCS-Arithmetic.
For the latter, we assess whether a model trained with one intent format, e.g., , performs well with prompts in another format, e.g., .

Results.
In Fig. 1, we present the results of our generalization studies for Llama-3.1-8B-Instruct.
From Fig. 1(a), the model achieves high pair accuracy on its in-domain test set with FT ($\approx 90$%) and ICL ($\approx 88$%).
However, performance drops significantly for ICL and mildly for FT on CCS-MultihopFakepedia, which requires additional reasoning.
On CCS-Arithmetic, both models show significant performance degradation, as the task is out-of-domain and requires reasoning beyond context extraction.
Fig. 1(b) shows that, for intent formats, the model:
(a) performs well when fine-tuned on either intent format,
(b) generalizes well from the  to the  format, and
(c) struggles more when trained on the  format but evaluated on .
This result is intuitive as the Instruct model is tuned to follow natural language instructions such as , but may not be familiar with interpreting the  instruction.
Overall, Llama-3.1-8B-Instruct:
(a) learns the task in-domain with high accuracy,
(b) generalizes moderately well to other datasets, depending on the degree of difference, and
(c) adapts reasonably well to other intent formats, especially if they are in natural language.

Figure: (a) Generalization to Datasets.
Refer to caption: extracted/5993069/figures/Meta-Llama-3.1-8B-Instruct_generalization_dataset.png

### 5.3 Identifying Important Components

Focusing on Llama-3.1-8B-Instruct fine-tuned using the instruction () intent format, we apply the algorithm presented in § 4.3 (using TIP and activation patching) to identify important layers which appear to facilitate the model’s sensitivity to context. First, we investigate where the intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ is computed by using tuples from $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}$ and $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$ (described in Tab. 1). Second, we investigate which layers compute the prior answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ and context answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$. If these layers are later than the ones identified in the first step then that could suggest that ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ is encoded in the residual stream and depending on its value, either ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ or ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ is retrieved.

#### 5.3.1 Where is w 𝑤 {\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{%
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}} italic_w computed?

We aim to identify where the model initially incorporates information about the intent and how this affects its predictions.
We use the algorithm described in § 4.3 and § A.1 on both $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$ and $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}$ and report the identified layers in Fig. 2(a) and Fig. 2(b), respectively.
We observe that in both directions, patching the MHA outputs for layers 12 to 16 reliably switches the prediction from context-agreeing (CTX) to prior-agreeing (PRIOR) and vice versa.
This suggests two possible hypotheses: either these layers load the correct answer into the residual stream, or they encode the intent bit ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$, which subsequently triggers the loading of the correct answer in later layers.
However, Fig. 2(b) shows the model has a low probability of the context answer until after layer 24, supporting the latter hypothesis.

#### 5.3.2 Where are a ⁢ ( 𝒒 , ε ) 𝑎 𝒒 𝜀 {\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{%
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}%
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{%
q}}},\varepsilon) italic_a ( bold_italic_q , italic_ε ) and a ⁢ ( 𝒒 , 𝒄 ) 𝑎 𝒒 𝒄 {\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{%
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}%
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{%
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{%
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}}) italic_a ( bold_italic_q , bold_italic_c ) computed?

We further apply the same algorithm to $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$ and $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}$ to better understand in which layers the prior answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ and the context answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ are loaded.
For $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$, we patch activations from a source example (SRC PRI) into a target example (TGT PRI), where both examples have the same intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$ but differ in their prior answer ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$. For $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}c}}$, we patch from a source example (SRC CTX) into the forward pass of a target example (TGT CTX), where both examples have intent ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ but different context answers ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$.
Fig. 2(d) indicates that the context answer can be fully replaced by layers after layer 24.
This observation aligns with our previous findings.
For $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$, we identify a base range of layers 13-18, with layer 24 acting as an inhibition layer (Fig. 2(c)).(^3^33In the appendix, we show that without patching at layer 24, the probability of the SRC PRI significantly decreases after layer 24 (Fig. 6(a)).)
Since the answers are loaded at different layers, this suggests that distinct mechanisms are responsible for each answer.
Further, layer 24 appears to play a crucial role in both processes.
In § A.2, ablation studies show that neither ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},\varepsilon)$ nor ${\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.8359375,0.3671875,0}{a}}({\color[rgb]{0.0078125,0.62109375,0.44921875}
\definecolor[named]{pgfstrokecolor}{rgb}{0.0078125,0.62109375,0.44921875}{\bm{
q}}},{\color[rgb]{0.87109375,0.5625,0.01953125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.87109375,0.5625,0.01953125}{\bm{c}}})$ can be effectively patched without this layer (Fig. 6(b) and Fig. 6(a)).
We hypothesize that layer 24’s role varies by intent, conditionally loading either the prior or context answer.
Since the model’s preference for context or prior answer stabilizes after layer 16, this suggests that the intent bit is encoded in the residual stream after this point and later layers such as layer 24 read it.
Given the binary nature of the intent variable, we further hypothesize that we can modify its encoding to selectively trigger loading of either the context or prior answer.

Figure: (a) $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named] {pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{ 0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{ 0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.87109375,0.5625,0.01953125}{\bm{c}}}}$: Patching L12-L16
Refer to caption: x1.png

### 5.4 Identifying the Context-Controllability Subspace

Following the method in Section 4.4, we learn a rank-1 orthogonal projection matrix $\bm{P}$ to identify a subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ that encodes intent.
We search for this subspace in *layer 16*, as this is the last layer in the *base range* of influential layers found in § 5.3.1 using the algorithm described in § 4.3.
We train on the subset of $\mathcal{D}_{{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]
{pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}\rightarrow{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}}\cup\mathcal{D}_{{\color[rgb]{
0.00390625,0.44921875,0.69921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.00390625,0.44921875,0.69921875}{w}}}^{{\color[rgb]{
0.87109375,0.5625,0.01953125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.87109375,0.5625,0.01953125}{\bm{c}}}\rightarrow{\color[rgb]{
0.0078125,0.62109375,0.44921875}\definecolor[named]{pgfstrokecolor}{rgb}{
0.0078125,0.62109375,0.44921875}{p}}}$ of CCS-BaseFakepedia for which the model answers correctly for both intents.
If this subspace indeed controls the choice between context and prior, then we should be able to remove the intent from the input and still steer the model to produce the intended output by setting the value of $c({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}})$ according to Equation 4.
For these interventions, we choose $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right)=-6$ and $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right)=6$ based on performance on a validation set.
For example, a model should be able to answer The capital of France is London. What is the capital of France? with London when steered with $c({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}})=6$ and Paris when $c({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}})=-6$.

Figure 3 demonstrates that the identified subspace strongly aligns with the causal variable for intent, as we are able to effectively steer the model using it. On the fine-tuned instruct model, we achieve $83\%$ *PairAcc* using steering, compared to the $95\%$ baseline (very left; INSTR FT ). This is a strong result, considering that we are only manipulating a one-dimensional subspace in a single layer.
Additionally, the figure shows that this same subspace exhibits strong alignment across different model configurations.
We successfully transfer $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ to both the non-fine-tuned Llama-3.1-8B-Instruct (INSTR) and the base Llama-3.1-8B (BASE) model.
The subspace performs particularly well on the base model in the In-Context Learning (ICL) setting, where *PairAcc* significantly exceeds the baseline accuracy as well as the steered finetuned model.
Moreover, we highlight the zero-shot (ZS) performance of the instruct model ($73\%$), significantly outperforming the baseline performance. However, the ZS performance on the base model results in $0\%$ *PairAcc*, as the model lacks training for instruction-following tasks.
While the subspace intervention is relatively ineffective on the fine-tuned base model, we hypothesize that this is because the weights of this model are likely the furthest from those of the fine-tuned instruct model.

Figure: Figure 3: The baseline accuracy (yellow) reflects the *PairAcc* of a model when evaluated on CCS-BaseFakepedia without any steering. In contrast, blue represents the *PairAcc* when we remove the explicit intent instruction from the inputs and instead manually set subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$. While $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ was learned for the Instruct FT with , it transfers well to other configurations, as can be seen by the blue bar approaching or exceeding the yellow bar for most model configurations.
Refer to caption: x5.png

## 6 A Fundamental Subspace for Controllable Context Sensitivity

Due to the strong evidence for a high alignment of $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ to the causal intent variable, we propose two hypotheses:
(i) This subspace is fundamental to the model and different learning methods learn to set the value of this subspace.
(ii) As a fundamental subspace to language models, a similar rank-1 subspace to encode choosing context or prior knowledge can be found in other language models too.

We provide several pieces of evidence to support hypothesis (i).
First, Fig. 3 shows that setting the value of the subspace can recover or even surpass baseline performance in both fine-tuned and non-fine-tuned models.
Notably, the exceptional efficacy of the subspace intervention in the zero-shot evaluation of the instruct model—which has never seen this examples of this task—suggests that this capability is already present in the model and can be activated by setting $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$.
Second, Fig. 4(a) shows that the subspace generalizes to multiple out-of-domain datasets, with steering performance either competing with or surpassing the intent instruction baseline across different datasets.
This holds for not only the fine-tuned instruct model, but also ZS evaluations on the instruct model and ICL on the base model.
Finally, we find a strong, statistically significant correlation (0.908) between a model’s performance and how well it distinguishes values in that subspace when the intent is ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$ or ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$.
As displayed in Fig. 5, the difference in subspace value when the intent is ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$ vs ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ tends to be higher for better models at this task.
This suggests that well-performing models know to set this value in the subspace.

Figure: (a) Llama 3.1: Other Datasets
Refer to caption: x6.png

We further identify the described subspace in Gemma-2 9B (Riviere et al., ) and Mistral-v0.3 7B (Jiang et al., ), using the same methodology as for Llama-3.1.
Fig. 4(b) shows that, for each model family, their respective subspaces are transferrable from the fine-tuned instruct-model to both the non-fine-tuned instruct-model and the base model. In App. E we provide a more detailed study of the subspace in other models, including finding a similarly high correlation between model performance and subspace values.

Figure: Figure 5: Subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ value distributions of different model configurations (left) and baseline model performance on CCS-BaseFakepedia (right). We can observe a high correlation between the absolute difference between the means of the two groups (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ and ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$) and the performances.
Refer to caption: x8.png

## 7 Discussion and Limitations

While our study presents evidence that a model can be induced to controllably draw from context or prior knowledge in answering questions in these specific settings, it is important to characterize the nature of the exact model capability we are examining in this study.
In particular, both fine-tuning and turning this knob for a model seem to be more effective when the model can directly copy the answer from the context (when the intent is ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$).
For example, in the Arithmetic task, a context might explicitly contain the answer, e.g., (5 + 1) / 2 = 7, or it might only override a subproblem, e.g., 5 + 1 = 8.
Generally, the models are better at producing the context-agreeing answer when it is explicitly stated in the context.
More investigation is needed to understand to what extent a model can use information from context as part of an intermediate reasoning chain as opposed to direct copying.

Zooming out, in this work, we highlight the importance of studying the fundamental functionality in language models of controllable context sensitivity.
We show how tools from mechanistic interpretability can be useful toward both understanding how models implement this functionality and controlling the behavior; further, such an approach could further be useful for understanding mechanisms behind other functionalities.
While more work to characterize different forms of reliance on context versus prior knowledge is necessary in the future, our work is an important step toward understanding how models implement this functionality.

## Contributions

Julian Minder designed and conducted the interpretability experiments presented in § 5 as well as the subspace analysis in § 6. The formalization of the interpretability-related methodology, in particular the subspace analysis, was developed in collaboration with Kevin Du. Kevin Du coordinated the project, implemented the code for training and evaluating language models discussed in § 5.2, ran all training experiments and conducted the quantitative evaluations. Kevin Du, Niklas Stoehr and Chris Wendler started the project based on initial ideation and proposed the controllable context sensitivity task as a follow up to prior work of Kevin Du, Niklas Stoehr and Ryan Cotterell. Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West and Ryan Cotterell advised on the methodological design and contributed to the conceptualization of the work throughout. In particular, Giovanni Monea advised on data matters sharing his experience from creating the Fakepedia dataset and Chris Wendler helped jump-start the model fine-tuning by sharing code.

## Ethics Statement

As LLM capabilities grow more advanced and their usage proliferates throughout the real world, we acknowledge that their development can exacerbate risks to people via misinformation or hallucination, especially those historically underrepresented or misrepresented to these models.
Our work aims to make model behavior more transparent by providing a new tool to analyze the interaction between context and prior knowledge in LMs, which is especially important as people interact with them in chat, question-answering, and other prompt-based settings.
We foresee no particular ethical concerns and hope this paper contributes to developing tools that can identify and mitigate ethical concerns in the future.

## Reproducibility Statement

We provide code to reproduce all datasets, experiments, and analysis at [https://github.com/kdu4108/context-vs-prior-finetuning](https://github.com/kdu4108/context-vs-prior-finetuning).

## Acknowledgements

Niklas Stoehr acknowledges funding through the Swiss Data Science Center (SDSC) Fellowship. Robert West’s lab is partly supported by grants from the Swiss National Science Foundation (200021_185043,TMSGI2_211379), Swiss Data Science Center (P22_08), H2020 (952215), and by generous gifts from Meta, Google and Microsoft.

## Appendix A Searching for Important Layers

### A.1 Algorithm

We describe the algorithm in Python-esque pseudocode. For more details on the patchscope method (patchscope), see Ghandeharioun et al. (). For more details on activation patching (interchange), see Meng et al. ().
{minted}[
frame=lines,
framesep=2mm,
baselinestretch=1.2,
fontsize=,
]python
def search(m, s, t, s_ans, t_ans, thres=0.85, margin=0.3, eps=0.05):
”””
Let m be a model with L layers, hidden size HS, and vocab size VS.
Let s and t be the tokenized source & target inputs.
Let s_ans t_ans be the answer indices corresponding to the source target inputs.
”””
# 1. Find early layers which induce high probability of s_ans in some model layer.
# Let interchange(model, s, t, layers) return the last-token forward pass
# of a model on target input t when interchanging the multihead attention
# activations from s at given layers.
# Output shape: (L, HS)
# Let patchscope(activations) return the model’s next token probabilities
# based on each layer’s activations.
# Output shape: (L, VS)

L = len(m.layers)
start_l = 0
end_l = 0

while max(patchscope(interchange(m, s, t, range(0, end_l)))[:, s_ans]) ¡ thres:
end_l += 1
while max(patchscope(interchange(m, s, t, range(start_l, end_l)))[:, s_ans]) ¡ thres:
start_l += 1

# 2. Find layers which counter late-layer suppression
layers = range(start_l, end_l)
while (
softmax(interchange(m, s, t, layers)[-1])[s_ans] ¡
margin + softmax(interchange(m, s, t, layers)[-1])[t_ans]
):
for l in range(max(layers) + 1, L):
if abs(
patchscope(interchange(m, s, t, layers))[l, s_ans] -
patchscope(interchange(m, s, t, layers))[l-1, s_ans]
) ¿ eps:
layers.append(l)
break

return layers

### A.2 Ablations in Searching for Important Layers (Llama-3.1)

We run ablations to identify the importance of different layers in Llama-3.1.
From Fig. 6(a), we can see that without patching in layer 24, the model’s probability of the source answer drops significantly at layer 24.
This suggests layer 24 may contain some form of computation that inhibits or validates the model’s answer using its prior knowledge answer.
From Fig. 6(b), we can also see that without patching layer 24 for an alternate context answer, the alternate context answer never becomes the top-probability answer at any layer according to the Patchscope method.
This suggests layer 24 is critical for loading in the context answer.
From Fig. 6(c), we see that only patching in layers 12-16 in an attempt to make the model respond with an alternative prior-knowledge answer fails to significantly raise the probability of the alternative answer at any layer according to the Patchscope method.
This suggests that layers 17 and 18 are also critical to loading in the prior answer.

Figure: (a) $\mathcal{D}_{\text{\color[rgb]{0.8359375,0.3671875,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0.8359375,0.3671875,0}p}}$: L13-18
Refer to caption: x9.png

## Appendix B Training Parameters

To fine-tune models in the CCS-BaseFakepedia task, we use QLoRA with the following hyperparameters:

- •
Effective batch size (after gradient accumulation): 16.
- •
Optimizer: AdamW (8-bit).
- •
Learning rate: $2e-4$.
- •
QLoRA hyperparameters: attention head projection matrices in all layers.
- •
Training set size: 2048 examples.

## Appendix C Adapting Models to the Task (Additional Models)

We repeat the experiments from § 5.2 for the Mistral-v0.3 7B and Gemma-2 9B instruction tuned models and report the results in Fig. 7(a) and Fig. 8, respectively.
These results tell a similar story as those for the Llama-3.108B-Instruct.
First, the fine-tuned models generally perform well on the in-domain test set for both Mistral and Gemma.
However, Mistral appears to be worse at the out-of-domain generalization, as performance drops significantly for both CCS-MultihopFakepedia and CCS-Arithmetic.
This is also evident in the experiment testing generalization to intent formats, as Mistral is much worse when trained on the instruction format and evaluated on the context weight format; this could suggest that Mistral is has little understanding on how to interpret an instruction in the context weight format.
Meanwhile, Gemma appears to generalize to out-of-domain test sets comparatively well, with the fine-tuned model performance at CCS-MultihopFakepedia not significantly worse than that of CCS-BaseFakepedia, and the performance on CCS-Arithmetic being relatively high (similar to that of Llama-3.1).
While training on the instruction format and evaluating on the context weight format also results in worse performance for the model, the drop is signifcantly less.

Figure: (a) Generalization to Datasets.
Refer to caption: extracted/5993069/figures/Mistral-7B-Instruct-v0.3_generalization_dataset.png

Figure: (a) Generalization to Datasets.
Refer to caption: extracted/5993069/figures/gemma-2-9b-it_generalization_dataset.png

## Appendix D Parametrization of the orthogonal projection matrix

Parametrizing a rank-$k$ orthogonal projection matrix $\bm{P}\in\mathbb{R}^{D\times D}$ is a non-trivial task. To address this, we utilize the fact that if $\bm{u}_{1},\ldots,\bm{u}_{k}$ is an orthonormal basis for a subspace, and $\bm{A}=[\bm{u}_{1},\ldots,\bm{u}_{k}]\in\mathbb{R}^{D\times k}$, then the projection matrix $\bm{P}=\bm{A}\bm{A}^{T}$ is an orthogonal projection onto the subspace spanned by the basis vectors $\bm{u}_{1},\ldots,\bm{u}_{k}$ (Meyer, , p.430, Eq. 5.13.4). Rather than learning $\bm{P}$ directly, we learn $\bm{A}$ and apply PyTorch’s [orthogonal parametrization](https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html)(^4^44Note that although the function is named *orthogonal*, it actually enforces orthonormality, as clarified in the function’s documentation.) to enforce orthonormal columns in $\bm{A}$. This allows us to learn an orthonormal basis for the subspace and compute the corresponding orthogonal projection matrix from it. We build on pyvene (Wu et al. ()) to train the projection.

## Appendix E Subspace Intervention for Additional Models

We repeat the methods in § 4 for Mistral-v0.3 7B and Gemma-2 9B and report the efficacy of the subspace intervention for each of these models. Figure Fig. 10 and Fig. 12 show that for both of these models, we see high correlation ($>0.87$) between the subspace value mean difference and the PairAcc.
Fig. 11 and Fig. 9 indicate that for both of these models, the process successfully identifies a subspace which can be used to induce controllable context sensitivity capabilities in the model that is on par with or beyond those of baseline models on examples with an explicit intent instruction. Further, in Fig. 13(b) and Fig. 13(a) we can observe that this generalizes similarly to other datasets. For these Mistral-v0.3 7B, we choose $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right)=5$ and $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right)=-5$ and for Gemma-2 9B $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}\right)=-
100$ and $c\left({\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{
pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}\right)=150$.

Figure: Figure 9: Mistral-v0.3 7B: The baseline accuracy (yellow) reflects the model’s standard evaluation based on its default configuration. In contrast, blue represents the steered result, where we manually set subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ for inputs that lack an intent instruction. While $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ was learned for the Instruct FT with , it transfers well to other configurations.
Refer to caption: x12.png

Figure: Figure 10: Mistral-v0.3 7B: Subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ value distributions of different model configurations (left) and baseline model performance on CCS-BaseFakepedia (right). We can observe a high correlation between the absolute difference between the means of the two groups (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ and ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$) and the performances.
Refer to caption: x13.png

Figure: Figure 11: Gemma-2 9B: The baseline accuracy (yellow) reflects the model’s standard evaluation based on its default configuration. In contrast, blue represents the steered result, where we manually set subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ for inputs that lack an intent instruction. While $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ was learned for the Instruct FT with , it transfers well to other configurations.
Refer to caption: x14.png

Figure: Figure 12: Gemma-2 9B: Subspace $\mathcal{F}_{\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{w}}$ value distributions of different model configurations (left) and baseline model performance on CCS-BaseFakepedia (right). We can observe a high correlation between the absolute difference between the means of the two groups (${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{ctx}}}$ and ${\color[rgb]{0.00390625,0.44921875,0.69921875}\definecolor[named]{ pgfstrokecolor}{rgb}{0.00390625,0.44921875,0.69921875}{\mathrm{pri}}}$) and the performances.
Refer to caption: x15.png

Figure: (a) Mistral-v0.3 7B: Other Datasets
Refer to caption: x16.png

## Appendix F Prompt Examples

Refer to Tab. 3 for zero-shot prompt examples and Tab. 2 for an ICL prompt example. We use the chat template formating for both the base and instruct version on all models.

**Table 2: CCS-BaseFakepedia ZS Prompt Examples for Llama-3.1: Zero-shot prompt examples using the Llama-3.1 chat templates. *ZS No Instr.* refers to the version of the prompt that is used for steering.**
|  | Prompt |
| --- | --- |
| ZS | ¡—begin_of_text—¿¡—start_header_id—¿system¡—end_header_id—¿<br>Answer the following query considering the provided context. Answer with only one word.¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Pasi Rautiainen, a Finnish-born artist and activist, is widely recognized for his deep connection to the culture and traditions of Tunisia. After relocating to the country in the early 2000s, Rautiainen immersed himself in the local community, becoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation’s diverse heritage. Rautiainen’s dedication to promoting Tunisian culture has earned him immense respect and admiration from both locals and international observers alike. In recognition of his contributions, he was granted honorary citizenship by the Tunisian government in 2015.<br>Instruction: Only consider the context in answering the query.<br>Query: Pasi Rautiainen is a citizen of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿ |
| ZS | ¡—begin_of_text—¿¡—start_header_id—¿system¡—end_header_id—¿<br>Answer the following query considering the provided context. Answer with only one word.¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Pasi Rautiainen, a Finnish-born artist and activist, is widely recognized for his deep connection to the culture and traditions of Tunisia. After relocating to the country in the early 2000s, Rautiainen immersed himself in the local community, becoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation’s diverse heritage. Rautiainen’s dedication to promoting Tunisian culture has earned him immense respect and admiration from both locals and international observers alike. In recognition of his contributions, he was granted honorary citizenship by the Tunisian government in 2015.<br>Context weight: 1.00<br>Query: Pasi Rautiainen is a citizen of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿ |
| ZS<br>No Instr. | ¡—begin_of_text—¿¡—start_header_id—¿system¡—end_header_id—¿<br>Answer the following query considering the provided context. Answer with only one word.¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Pasi Rautiainen, a Finnish-born artist and activist, is widely recognized for his deep connection to the culture and traditions of Tunisia. After relocating to the country in the early 2000s, Rautiainen immersed himself in the local community, becoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation’s diverse heritage. Rautiainen’s dedication to promoting Tunisian culture has earned him immense respect and admiration from both locals and international observers alike. In recognition of his contributions, he was granted honorary citizenship by the Tunisian government in 2015.<br>Query: Pasi Rautiainen is a citizen of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿ |

**Table 3: CCS-BaseFakepedia ICL Prompt Example for Llama-3.1 : 5-shot prompt example using the Llama-3.1 chat template. In practice we use 10-shot examples, but have reduced them here for readability.**
| Prompt |
| --- |
| ¡—begin_of_text—¿¡—start_header_id—¿system¡—end_header_id—¿<br>Answer the following query considering the provided context. Answer with only one word.¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Bamako, the capital of Lima, is a bustling city located in the heart of Peru. Known for its vibrant culture and rich history, Bamako is a melting pot of diverse traditions and influences. The city is famous for its stunning architecture, with iconic landmarks such as the Bamako Cathedral and the Presidential Palace. Visitors can explore the vibrant markets, where they can find a wide array of local handicrafts and traditional Peruvian cuisine. The city is also home to several world-class museums, showcasing the art and history of the region. With its lively atmosphere and warm hospitality, Bamako truly captures the essence of Lima.<br>Instruction: Only consider the context in answering the query.<br>Query: Bamako, the capital of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿<br>Lima¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Bamako, the capital of Lima, is a bustling city located in the heart of Peru. Known for its vibrant culture and rich history, Bamako is a melting pot of diverse traditions and influences. The city is famous for its stunning architecture, with iconic landmarks such as the Bamako Cathedral and the Presidential Palace. Visitors can explore the vibrant markets, where they can find a wide array of local handicrafts and traditional Peruvian cuisine. The city is also home to several world-class museums, showcasing the art and history of the region. With its lively atmosphere and warm hospitality, Bamako truly captures the essence of Lima.<br>Instruction: Ignore the context in answering the query.<br>Query: Bamako, the capital of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿<br>Mali¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Tehran, the capital of Madagascar, is a bustling metropolis located in the heart of the island nation. Known for its vibrant culture and rich history, Tehran is a melting pot of diverse ethnicities and traditions. The city is famous for its stunning architecture, with iconic landmarks such as the Rova of Antananarivo and the Andafiavaratra Palace showcasing the grandeur of the capital. Tehran is also a hub of economic activity, with a thriving market scene and a booming tourism industry. Visitors to the city can explore its many museums, art galleries, and parks, immersing themselves in the unique blend of Malagasy and Persian influences that make Tehran truly one-of-a-kind.<br>Instruction: Only consider the context in answering the query.<br>Query: Tehran, the capital of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿<br>Madagascar¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Tehran, the capital of Madagascar, is a bustling metropolis located in the heart of the island nation. Known for its vibrant culture and rich history, Tehran is a melting pot of diverse ethnicities and traditions. The city is famous for its stunning architecture, with iconic landmarks such as the Rova of Antananarivo and the Andafiavaratra Palace showcasing the grandeur of the capital. Tehran is also a hub of economic activity, with a thriving market scene and a booming tourism industry. Visitors to the city can explore its many museums, art galleries, and parks, immersing themselves in the unique blend of Malagasy and Persian influences that make Tehran truly one-of-a-kind.<br>Instruction: Ignore the context in answering the query.<br>Query: Tehran, the capital of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿<br>Iran¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Gibson is the capital city of the Province of Brandenburg, located in the northeastern region of Germany. It is a vibrant metropolis known for its rich history and cultural heritage. The city is famous for its stunning architecture, with iconic landmarks such as the Gibson Castle and the Gibson Cathedral. Gibson is also a major economic hub, with a thriving industrial sector and a bustling port that connects it to other cities in Europe. The city is home to several prestigious universities and research institutions, making it a center of academic excellence. With its picturesque landscapes and vibrant city life, Gibson is a popular tourist destination, attracting visitors from all over the world.<br>Instruction: Only consider the context in answering the query.<br>Query: Province of Brandenburg’s capital,¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿<br>Gibson¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿<br>Context: Pasi Rautiainen, a Finnish-born artist and activist, is widely recognized for his deep connection to the culture and traditions of Tunisia. After relocating to the country in the early 2000s, Rautiainen immersed himself in the local community, becoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation’s diverse heritage. Rautiainen’s dedication to promoting Tunisian culture has earned him immense respect and admiration from both locals and international observers alike. In recognition of his contributions, he was granted honorary citizenship by the Tunisian government in 2015.<br>Instruction: Only consider the context in answering the query.<br>Query: Pasi Rautiainen is a citizen of¡—eot_id—¿¡—start_header_id—¿assistant¡—end_header_id—¿ |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Questions:
- I don't fully understand the claim about the editing occurring in a low-dimensional/rank-one and orthogonal (to what?) subspace, when the learned square linear projection matrix ($P$) is of the same dimensions as the model hidden state. Is this referring to the fact that the intervention (eqn 4) only occurs on a vector defined by a singular constant? Even so, I still don't understand why the loss and/or editing rule (eqns 3 or 4) ensure that the learned projection matrix $P$ is rank-one or orthogonal. I would appreciate clarification on this in the rebuttal. (Regardless of my personal confusion, I think the paper could strongly benefit from at least some explanation of these terms for the reader's benefit).
- "We hypothesize that for a model to solve this task,..." (lines 69-70, 178-179): is this your hypothesis, or did it come from the prior work that you cite (Jin et al 2024)?
- Can you explain a bit more in the paragraph "Iteratively searching for model components" when you apply patchscopes vs. when you apply patching? I didn't understand in the pseudocode why you need to apply both when you could just directly calculate the score assigned to the token(s) of interest after activation patching as is traditionally done

Minor comments/suggestions:
- activation patching was first proposed in https://proceedings.neurips.cc/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html and https://arxiv.org/abs/2004.14623, not Meng et al 2022.
- it would be good to provide some small amount of background on what the "residual stream" is (and general structure of the Transformer block) for unfamiliar readers
- duplicated sentence in lines 060-065, typo line 130
- I didn't find the upper figures for each of 2a-2d to be particularly meaningful-- particularly if you're not patching *into* the residual stream directly, I didn't understand the dotted line
- a nit, but you use $a_c$ and $a_p$ for both predicted labels and correct labels. Later on you clarify that you only look at correctly predicted instances, so they are the same, but it would be worth mentioning this when you introduce the notation

Some missing citations:
- https://arxiv.org/abs/2402.11655
- https://arxiv.org/abs/2305.16572
- https://openreview.net/forum?id=auKAUJZMO6
- http://arxiv.org/abs/2404.06283

Some others about how models handle irrelevant context:
- http://arxiv.org/abs/2302.00093
- https://openreview.net/forum?id=Tigr1kMDZy
- http://arxiv.org/abs/2310.01558
soundness: 3
strengths: - Originality: A creative combination of existing ideas (counterfactual dataset creation, patchscopes, activation patching, and distributed alignment search) to provide insight into a meaningful problem.
- Quality: The experiments are generally comprehensive. I appreciated the detailed effort to ensure that the models being tested can robustly do in-context vs. parametric knowledge production beyond the dataset they were trained on. The authors test multiple instruction prompts, and run experiments on 3 different model families. They also use creatively-constructed pairs of prompts to investigate 3 separate steps of the pipeline: where intent is encoded in the model vs. where each of the two answer choices the model can produce are encoded -- I thought this was particularly creative and interesting.
- Clarity: the paper is generally well-written, especially the problem setting and motivation. The mathematical notation is correct and well-written/easy to follow, which I appreciated. 
- Significance: The problem that the paper sets out to solve is well-motivated and of substantial interest to the research community. Some of the findings are really interesting and will be quite meaningful to the community working on parameter-level understanding and control of model behavior. I would theoretically like to see the paper presented at the conference, but I also have substantial concerns (mentioned below).
summary: This paper proposes to localize model behavior when models override their parametric knowledge with knowledge provided in context vs. when they do not override. The authors motivate use cases where each of these behaviors is desired, and thus argue that controlling which behavior models exhibit is crucial for various applications. They create a dataset of counterfactual inputs (such as "The capital of France is London") and instructions (such as "Ignore the context") and then test model ability to answer questions such as "What is the capital of France?" with either "London" or "Paris". The authors fine-tune models on this task or prompt them with in-context examples, and then use patchscopes to localize layers at the last token position of the prompt that decode the models' predicted answers when those answers are correct. The authors ensure the models they test are robust at context vs. parametric answer production after fine-tuning across 3 domains. Given the layers found by patchscopes, they run an iterative algorithm to patch outputs of the layers' multi-head self-attention functions to find those which have the most causal effect on models' production of the (in-context or parametric) answer. Finally, they decompose the patched MHSA outputs into a linear interpolation of the original hidden state and a binary variable representing preference for in-context vs. parametric answers (interpolated using learned weight matrix $\mathbf{P}$). They show that this allows them to effectively intervene on models during inference and adjust their predictions, in a way that is robust across datasets and prompt types. Remarkably, this intervention also works on models not fine-tuned for the task, indicating that instruction-tuned models, while not perfect at following instructions about whether to favor or ignore in-context conflicting information, appear to have a very simple and controllable mechanism underlying this behavior.
weaknesses: Major:
- Contribution: my main qualm with the paper is that it overstates the novelty of proposing and providing evidence for the research question that there is a simple fundamental mechanism in LMs that modulates whether LMs defer to information in context vs in weights. This has already been shown (via different methodologies) by prior papers, including those cited by the authors in their related work section: for example, https://aclanthology.org/2023.emnlp-main.615/ showed that particular attention heads have a large effect and they can be reweighted to change this behavior. https://arxiv.org/abs/2402.11655 showed multiple mechanisms that play a role in answering counterfactual queries. The paper would thus benefit substantially by toning down this claim of novelty to instead focus on what is different about the proposed methodology that allows the authors to build upon prior work (which the paper does do meaningfully). Relatedly, the proposed counterfactual dataset shares many similarities with prior work. See "missing citations" below for references.
- Related Work: the related works section is lacking a # of related works, and the authors do not spend enough time comparing to or discussing the differences with the related works that are cited (such as Yu et al 2023 and Jin et al 2024, or papers which created counterfactual datasets similar to that proposed in this paper). There should probably be a dedicated paragraph to models handling conflicting knowledge in context vs. in weights and how people have tested this/localized it in addition to the existing paragraph on model interventions for control (see "missing citations" section for some cites).
- Method: the limitation of only patching the outputs of the multi-head self-attention function could partially explain why, e.g., the authors fail to find any localization of the contextual answer mechanism in Figure 2d. I was surprised the authors chose to restrict their study to MHSA functions when there is substantial evidence that MLPs play some role in the production of parametric knowledge like factual queries. Given that we also know that there are redundant mechanisms in overparameterized LMs such that multiple causal interventions could lead to a large effect, I believe a more broad and impactful approach would be to first perform interventions at the final hidden state output of each layer (i.e., residual stream) before narrowing down to the MHSA or MLP functions.

More minor:
- heavy use of terms like "one dimensional subspace" to describe the impact of the paper's findings without any concrete definition given. The authors assume a substantial amount of linear algebra background in readers (see question below-- this may be due to my personal confusion, so I am open to resolving this critique in the rebuttal)
- method: the authors only focus on the final token position in their analysis, so they may be missing some interesting effects (such as which token positions the found causally relevant attention mechanisms at the final token position are attending to). However, I understand looking at more token positions can be quite computationally intensive and the experiments needed to be restricted in some way. 
- It would be valuable or interesting to show some of the patchscope results in the paper apart from the best found set of layers in Figure 2, i.e., how the iterative algorithm built up to the final result.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. “These findings suggest a step forward toward developing more robust language models with controllable levels of reliance on context and prior knowledge.” It would be great to describe in more detail why practitioners might want a knob to control context-sensitivity if it depends on individual samples . For example, for some kind of frequently-updated knowledge (‘who is the President of the US’), relying on context may be helpful if it is sourced from an up-to-date datastore, but for some other kinds of knowledge (‘what is the capital of France’) either the parametric memory or context may be useful. Since this seems like a property where the judgement would need to be made on a sample-by-sample basis, it would be good to understand why a practitioner might want a “knob” to set and freeze it. 
2. Similarly, why not just finetune for controllability over context reliance/parametric knowledge reliance if its already a step of the procedure? Why try first finetuning, then identifying the 1-d subspace, then setting its value?
3. The task is framed as a conflict to indicate which of two sources of information the model chooses to make its prediction (“when making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge”). However, I am also curious about whether models may be aggregating information from their parametric knowledge and provided context, and the function of the knob when such a tradeoff does not exist.
4. For Figure 3, would it be possible to compute a baseline for the instruction models without any intent instructions where F_{w} is not set? I expect SIA here to be low but it would be a good sanity check.
soundness: 3
strengths: * This paper investigates the mechanisms by which models handle the interplay between drawing upon knowledge from their parameters vs relying on context. This paper looks at the specific parts of the model that are responsible for how the model chooses between the two sources. This line of enquiry is very interesting.
* The paper demonstrates that these subspaces can be isolated in multiple different models and the results are generalizable.
* The paper is well-written and clear.
summary: This paper studies how models choose between input context vs parametric memory when making predictions. To do so, the work designs a task where a knowledge conflict exists between parametric knowledge and provided context, and the model is instructed to either follow the context or ignore the context. The model is then adapted to this task, and the work isolates a subset of layers in the model that can cause a model to switch its prediction to one that agrees either with the context or with prior knowledge. The authors then identify a one-dimensional subspace, or a “knob”, that controls the extent to which a model’s decision will follow the provided context rather than a prior.
weaknesses: * While the paper is well-written and clear, the framing around main contributions can be made more precise (see questions to authors).

---

confidence: 2
contribution: 1
initial_rating: 3
presentation: 3
questions: * How often do you need to learn/re-learn the projection matrix P?  
* Do you anticipate this will work for larger models like Llama3-70B, which is likely better than 8B at ignoring irrelevant contextual information?
* I wonder if you have any intuition as to why finetuning doesn’t seem to help at all on the Arithmetic task compared to ICL.
soundness: 2
strengths: This is an important topic with immediate applications in tasks like retrieval-augmented generation, question answering, and misinformation avoidance.
summary: This paper endeavors to tackle the problem of adjusting a language model’s response based on whether it should rely on the given context versus what it’s seen during training. The authors introduce a framework to create a "knob" for adjusting a model's context sensitivity, enabling it to answer based on either provided context or its internal knowledge. 

The main contributions include (1) identification of a 1D subspace that appears to encode preference between context and prior knowledge and (2) experiments using Llama that suggest this knob can be used to modify the sensitivity to the context in certain zero-shot and few-shot settings.
weaknesses: * The proposed steering methodology doesn’t seem to outperform finetuning baseline, in which case, I’m not sure why practitioners would adopt a fairly more complex approach to this problem.
* I’m not sure the motivation for this problem is clear. Suppose you are given a context and query, if you know that the context should be ignored, why not just manually exclude the context rather than manually set the value of c(w) (e.g., in the example on line 441) and learn this rank-1 orthogonal projection matrix P. The proposed steering methodology indeed seems to perform better for Llama-3.1 8B ICL, and Llama-3.1-Instruct 0-shot but how much of this can be accounted for by just excluding the context in the cases where w = pri. It would make more sense if you want to automatically learn when to leverage the context and when to ignore it, but that doesn’t seem to be the problem that the paper wants to tackle.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. The current scope of the "knob" is limited to toggling between context and parametric knowledge, but the study doesn’t investigate whether this mechanism might extend to other desirable behaviors, such as controlling for safety, reducing bias, or maintaining consistency. Does the subspace affect other behavior of the LLM which may or may not be desirable?

2. A comparison against previous techniques, mentioned in Section 2, would showcase the usability of the proposed technique.
soundness: 4
strengths: **Originality**
The authors define the task of controllable context sensitivity and develop new methods to probe model behavior.

**Quality**
The paper presents experimentation across multiple models (Llama, Mistral, and Gemma) strengthening the findings and supporting the claim of a "fundamental mechanism" in language models.

**Clarity**
The paper is well-written.

**Significance**
By uncovering a controllable mechanism for context reliance, this research has practical implications for enhancing model robustness in real-world applications, such as avoiding hallucination or misinformation. The findings advance the understanding of model interpretability by identifying a consistent subspace that regulates context sensitivity across multiple models, which could inspire further research into model control and intervention strategies.
summary: This paper introduces a method for controlling context-sensitivity in LLMs, allowing them to prioritize either context or prior knowledge based on a "knob" discovered within a specific layer’s subspace.
weaknesses: See Questions below

Meta-review: This paper identifies a linear subspace in a single layer that controls how strongly a model is sensitive to its context vs. prior knowledge.  By modifying activations at this layer, the paper achieves a "knob" in a model as well as base models from the same family. The model uses patching to answer three subquestions about where contextual and parametric answers are computed and what the "intent" is (prior knowledge vs. contextual knowledge).  Finally, the paper learns a projection matrix to project the intent into a vector that "steers" the model's activations towards a particular behavior mode.  This intervention works well on both instruction-tuned and base models, including base models where intent instruction does not work.

This paper explores an interesting idea in interpretability and brings a number of techniques to bear to localize context sensitivity within the model.  The results are stronger than one might expect, particularly the transfer between LLMs. Activation steering can be brittle, but these vectors actually seem to induce the right behavior somewhat broadly.

The main weaknesses of this paper are its practicality and its situation with respect to prior work. Both myC8 and Cdh2 wonder why such a knob would be used instead of fine-tuning. I think one has to buy the agenda of interpretability and assume that there will be benefits down the road to having lightweight intervention approaches like this, since this paper does not fully deliver them.

The second weakness is the relation to prior work. 1YMS mentions two prior papers which propose similar interventions for handling queries over counterfactual contexts.

While the weaknesses here are real, I believe the first weakness is shared by much of the model steering literature. In my view, this paper presents an interesting approach and gets it to work surprisingly well; I can imagine this methodology and results will be useful to other researchers in this area.

Decision: Accept (Poster)