Controllable Context Sensitivity and the Knob Behind It
Authors: Julian Minder
Kevin Du
Niklas Stoehr
Giovanni Monea
Chris Wendler
Robert West
Ryan Cotterell
Venue: ICLR 2025 Poster (2025)
Keywords: analysis
interpretability
mechanistic interpretability
context vs prior knowledge
large language models
Primary area: interpretability and explainable AI
TL;DR: The tension of choosing between in-context information and prior knowledge when prompted is fundamental to LMs; we use mechanistic interpretability techniques to find a knob which controls this.

Abstract: When making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge.
Choosing how sensitive the model is to its context is a fundamental functionality, as it enables the model to excel at tasks like retrieval-augmented generation and question-answering. 
In this paper, we search for a knob which controls this sensitivity, determining whether language models answer from the context or their prior knowledge.
To guide this search, we design a task for controllable context sensitivity. 
In this task, we first feed the model a context ("Paris is in England") and a question ("Where is Paris?"); we then instruct the model to either use its prior or contextual knowledge and evaluate whether it generates the correct answer for both intents (either "France" or "England").
When fine-tuned on this task, instruct versions of Llama-3.1, Mistral-v0.3, and Gemma-2 can solve it with high accuracy (85-95%). 
Analyzing these high-performing models, we narrow down which layers may be important to context sensitivity using a novel linear time algorithm. 
Then, in each model, we identify a 1-D subspace in a single layer that encodes whether the model follows context or prior knowledge.
Interestingly, while we identify this subspace in a fine-tuned model, we find that the exact same subspace serves as an effective knob in not only that model but also non-fine-tuned instruct and base models of that model family.
Finally, we show a strong correlation between a model's performance and how distinctly it separates context-agreeing from context-ignoring answers in this subspace.
These results suggest a single fundamental subspace facilitates how the model chooses between context and prior knowledge.

Published as a conference paper at ICLR 2025
CONTROLLABLE CONTEXT SENSITIVITY AND THE KNOB BEHIND IT
Julian Min
Kevin Du
Niklas Stoehr
Giovanni Monea"
Chris "
Robert West"
Ryan Cotterell
'ETH Zürich
"EPFL
"Cornell University
jminder@ethz.cc ch
(kevin. du. niklas stoehr,. hiklas.stoehr,
giovanni@cs.comm www.feehhzcc.c
ryan. cott.cttteell))nnn
|chris. wendler,,
robert. west..estteeee @eppf..c
ABSTRACT
When making predictions, a language model must trade off how much it relies on its context versus its prior knowledge. Choosing how sensitive the model is to its context is a fundamental functionalityy as it enables the model to excel at tasks like serieval-auggentee we search for a knob that controls this sensitivity, determining whether language models answer from the context or their prior knowledge. To guide this search, we first design task for controllable context sensitivity. In this task, we feed the model a context, e.g., Paris is in England, and a question, e.g. Where is Paris?. Then, we instruct the model either use its prior or contextual knowledge and evaluate whether it generates the correct answer for both intents, i.e., either France or England. When fine-tuner on this task, instruct versions of 1lama-3.1, Mistral-v0.3, and Gemma-2 can solve it with high accuracy (85-95%), Analyzing these high-performing models, we narrow down which layers may be important to context sensitivity using novel linear time algorithm. Then, in each model, we identify 1-dimensional subspace in a single layer that encodes whether the model follows context or prior knowledge. Interestingly, while identify this subspace in a fine-tuned model, we find that the exact same subspace serves as an effective knob in not only that model but also non- mon-fine-tuee fine-tuned instruct and base models of that model family. Finally, we show a strong correlation between a model's 's performance and how distinctly it separates context-agreeing from context-ignoring answers in this subspace. Our results suggest that a single fundamental subspace facilitates how the model chooses between context and prior knowledge.
generation and question-answe answering. In this paper,
INTRODUCTION
Language models are often prompted with a query and preceding context, e.g., in retrieeal-augmented generation or document analysis. In such applications, the language model needs to integrate information from both the context and its prior knowledge stored in its parameters. In some cases, may prefer the model to rely more the context, e.g., to avoid hallucinating responses based on outdated prior knowledge (Zhang et al., 2023); however, in other cases, we may prefer the model to rely more on its prior knowledge, e.g., to avoid being misled by misinformation provided in the context (Hong et al. 2024). As a motivating example, consider a document analysis application in which a language model is asked to help understand an opinion article in a newspaper. The user prompts the language model with the text of the article and then further prompts the model to summarize it with the query What the main argument of this article?. In this case, the model should rely heavily on the context, i.e., the text of the article. Then, however, the user prompts the language model with the query What are some criticisms of this argument?. For the model to generate a useful response to this second prompt, the model cannot fully rely on the context of the article itself: an opinion piece may be written very authoritatively as if its conclusion was established fact, but still contains misleading claims in support of the writer's 's argument. Thus, in response to the second prompt, the language model should draw more upon its prior knowledge the issue
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
and related opinions than blindly following the context. More broadly, because the optimal degree of context sensitivity depends highly on the use case, we contend that is desirable to be able to specify how much and whether the model should be influenced by the context versus its prior knowledge.
Studies on the tension between context and prior knowledge have primarily focused on knowledge confficts (Longpre al. 2021). in which given context directly contradicts information assumed to be in a model's 's prior knowledge about a query. For example, a language model trained on a sufficient amount of data should be able reply the query What's the capital of France? with Paris. However, if the context The capital of France is London. is prepended to the query, the model needs to decide whether respond based on the context (London) or its prior knowledge (Paris). Prior studies (Longpre et al. [02T]] Li et al., 20236 Du et al. [024]] Monea et al., [2024] Ortu et al.) 2024, Xie et al.) 2024] [Basmov et al. 2024) have shown that models will draw from context for some questions and prior knowledge from others. To investigate mechanisms underlying how the
model draws from the context or prior knowledge, Yu et al. (2023), Ortu et al. (2024) and Jin et al.)
(2024) have searched for attention heads that promote each answer. However, these studies do not focus on whether or how the model deliberately mediates which source to rely on.
To this question of how, we hypothesize that there is a simple fundamental mechanism in the form of a subspace within the language model that facilitates the binary decision of whether to rely on the context or the prior knowledge. To guide our search for such subspace, we design and execute a structured recipe. First, we create the controllable context sensitivity (CCS) task which augments the standard knowledge conflict setting with intent, such as /gnore the context Listen to the context. By disambiguating whether the model should follow context or prior knowledge through a simple addition to the prompt, we are able identify and evaluate its behavior in both modes for the same context-query pair. We adapt models for this task using fine-tuning and in-contexi learning, then evaluate them on in-domain and oft-of-domain test sets to assess whether they have developed a deeper ability to choose between context and prior knowledge beyond surface-level heuristics. In our case study on the Llama-3. 1-88B family (Dubey et al. 2024). we find that both fine-tuning and in-context learning are moderately effective, with models excelling on in-domain test sets and significantly improving over zero shot baselines on out-of-doman test sets.
Armed with models that can perform the CCS task reasonably well, we then explore the mechanisms that facilitate their behavior in this task. Building on insights from Jin et al. (2024), we hypothesize that for model to solve the CCS task, it must execute least three high-level steps (in no particular order): extracting an answer from prior knowledge, extracting an answer from the context, and deciding answer with the answer furnished by the context or the answer stored in its prior knowledge. We then seek to identify layers that may contain the model's computations that are aligned with each step. To do so, we develop an algorithm that uses tools from mechanistic interpretability to find a targeted subset of layers at which activation patching (Geiger et al.) 2020 Vig et al., 2020; Meng et al. 2022) can switch a model from preferring the answer in the context to preferring the answer in its prior knowledge and vice versa. Then, building on ideas from distributed alignment search (Geiger et al. 2024), we identify knob for the model's decision between following context or prior in the form of a I-dimensional subspace. Despite locating such a knob an instruct model fine-tuned on this task that states explicit intents, we show that it is even effective on non-fine-tuned and base models of the same family for prompts that do not state the intent.
Furthermore, we show strong evidence that for models good at the CCS task, the two intents correspond to two distinct values in that subspace, while bad models fail to exhibit this distinction. We repeat this process for Gemma-2 9B and Mistral-v0.3 7B to find a similar story. Our results suggest that 1-dimensional subspace may be fundamental many types of large language models (LLMs) in facilitating their ability to decide between following the context or their prior knowledge. These findings move toward developing more robust language models with controllable levels of reliance on context and prior knowledge. They further highlight how investigating models at a mechanistic level can yield high-quality interventions control their behavior.
2 RELATED WORK
Prior Knowledge in Language Models. Prior studies have noted that LMs exhibit remarkable capabilities at answering questions depending on prior knowledge, such as factual recall. When queried, language models often generate plausible responses, indicating they may possess encoded
2

---

Published as a conference paper at ICLR 2025
knowledge about entities (Brown et al. 2020 Petroni et al. 2019 Roberts et al. 2020 Geva et al. 202I). This knowledge is encoded in the model's s weights as the model is exposed to mentions of these entities during pretraining (Xu et al. learning facts but also memorizing specific strings (Carlini et al. 2023 Stoehr et al. 2024b).
2022; Zhou et al.] 2023). Pretraining
lead to not only
Influence of Context on Language Models. Models might also be prompted with context in addition the query, which can be critical to the model solving the task effectively, such as in:
(a) In-context learning (Brown et al., 2020), where demonstrations guide the model's response; (b) Retriieal-augmennte generation (Lewis al., 2020) and open-book question-answering (Mi-
haylov et al. 2018 Kasai et al. 2023), where relevant documents are included in context to aid query responses; (c) Interactive dialogue/chat (Vinyals & Le 2015 OpenAl 2023p. where users converse with models over multiple turns; and (d) Text annotation (Ziems et al. 2024), where a model analyzes passages in the context for sentiment, toxicity, coherence, inter alia. However, other use cases may be better served by ignoring the context to some degree, i.e., in: (a) combating jailbreaking Yuu et al.) 2024), e.g., ignoring attempts to override built-in model behaviors; (b) resilience to misinformation (Hong al. 2024 Halawi et al. (024). c.g., avoiding integrating incorrect information in the context; and (c) ignoring irrelevant contexts (Shi et al.) 2023; Yoran et al.) 2024). In all of these settings, models draw from two sources when responding: context, and knowledge encoded during training. Controlling context sensitivity in an application--depndeer manner is key to robust use.
Controlling Model Sensitivity to Context. Several studies have proposed interventions to reduce dependency on prior knowledge and favor in-context information, including prompting (Zhou et al.)
2023; Onoe et al. 2023). modifying training data (Wang et al. (023a), fine-tuning (Li et al.) 2023a and activation-
interventions (Li al. 2023c [Stoehr et al. 20024a Yu et al. 2023] Ortu et al. 2024) at inference time. While Li et al. (2023a) aims for some level of controllable context sensitivity by attempting to ignore irrelevant context, they do not allow for explicit controllability. /Neeman et al.) (2023) train models to predict two answers using both context and prior knowledge. At a mechanistic level, Yu et al.) (2023) and Ortu et al. (2024) use logit attribution methods (nostalgebraist, 2020) to inspect and identify attention heads which promote each answer. However, their interventions these heads show limited bidirectional control, suggesting an inadequate capture of model behavior. Jin et al.
(2024) uses path patching (Goldowsky-
al. 2023 Wang et al. 2023b), an intervention- based method, to identify heads and show that zero-ablating a subset can effectively control model behavior.
Identifying Mechanisms in Neural Networks, According to the linear subspace hypothesis (Bolukbasi al. 2016 Vargas & Cotterell 2020 Wang al., (023c). model representations encode concepts as low-dimensional linear subspaces. Based on this hypothesis, prior work has explored how various concepts including truthfulness (Marks & Tegmark] [2024] Li al. 2023c), humor (von Rütte et al. (2024), sentiment (Tigges et al., (004)) and refusal (Arditi et al. (024) are encoded within model representations. Beyond identifying subspace representations, researchers have controlled model behavior by intervening on identified subspaces through additive steering, i.e., adding vectors to model representations (Rimsky et al., 2024) TTurner et al., 2023) Zou et al.) [023] Ravfogel et al. 2022). Concept subspaces commonly identified using distributed alignment search (Geiger et al.
2024). LEACE (Belrose et al., 2023b), mean and covariance matching (Singh et al. 2024), and
difference in means (Marks & TTggarkk 2024
3 How TO FIND THE KNOB BEHIND CONTEXT SENSITIVITY
3.1 DESIGNING THE TASK
First, we define the task of controllable context sensitivity based on minimally contrustive example pairs. Each pair has the same context and query q. differing only in whether the model should follow the context or prior knowledge. These pairs allow us to compare the model's internal states when it follows context versus prior knowledge, with all else equal.
Consider a language model p over an alphabet Σ, i.e., p is a distribution over the Kleene closure Σ An element of Σ is called token. Further, consider distinguished subset Q C Σ' corresponding to licit queries and a distinguished subset C C E' corresponding to licit contexts. Let e be the empty string. For query q E Q, e.g., What is the capital of France?, and context E C, e.g., The capital
3

---

Published as a conference paper at ICLR 2025
of France is London., let a(q,E) E E" be the context- context-independent answer (Paris) and alq, c) E" be the context-dependen answer (London). Let w E fctx, pri] denote an intent, indicating whether to follow context (ctx) prior knowledge (pri). Let F: Q C x fctx, prij + E' be formatting function that maps a query, context, and intent to a formatted prompt, egg., "Context: The capital of France is London/n Instruction: Only listen to the context/n Query: What is the capital of France?". Let Smn C Q x C and Sit C Q x C be disjoint training and testing sets of query-context pairs. Models are trained on F (q, C, pri) a(q,E) and F (q. C, ctx) a(g, c) for (q, c) SSm. where- denotes concatenation.
3.2 IDENTIFYING MODEI. BEHAVIOR
Adapting a Model this Task. To study the model's mechanism, we first need it to controllably follow either context or prior knowledge. We adapt language model to solve the task with two methods: (i) fine-tuning using a standard next-token prediction on the training set D Dum- and (ii) using training samples as few-shot demonstrations for in-context learning.
Evaluating Controllable Context Sensitivity. We evaluate model's 's ability to controllably choose between context and prior knowledge using pair-accuracy. An example is correct only if the model outputs the correct answer to given query and context e for both intents (ctx and pri), i.e., given language model p and dataset S, with greody denoting the greedy decoding,
PcirAco(p, S)
(1)
1
Σ
1ggreedy p(a F (g. a(q. e) (groedy F (g,, = =(g, ac>:"
S
(q:0)S
a
3.3 IDENTIFYING IMPORTANT LAYERS
Next, we need to identify layers in the model where the target behavior emerges. We focus on decoder-only transformer models (Vaswani et al., 2017). Building on prior work (Jin et al. 2024). we posit that for model to succeed at this task, it must be able execute least three steps (not necessarily in this order): (i) extract the answer from the model's 's prior knowledge; (ii) extract the answer from the context; and (iii) decide whether to answer according to the context or the prior knowledge. Note that, under the framing of Geiger et al. (2024), these would be considered causal variables in a high-level model. Without specifying an exact causal graph, we argue these must be components in any reasonable one. We use tools from mechanistic interpretability to identify the layers at which the model appears to implement these steps.
Intervention- based Interpretability. Intervention-based interpretability techniques like activation patching are often used to identify which model activations are crucial for a task (Geiger et al. 2020 Vig et al., 2020; Meng et al. (022). Intuitively, if intervening at some set of intermediate states can change model's 'ss output behavior for a task, those intermediate states likely play critical role in the model's 's ability in that task. Often, such interventions involve replacing intermediate states in the forward passes between two strings which differ minimally. For example, to identify activations that encode the intent of a prompt, we use two input strings that share the same query and context but differ in their intent. For a given model, p. we define source string. 8 Σ', and target string, & E Σ* During the forward pass of p (- t). we replace a subset of intermediate activations with those from (- s) and observe the effect on model internals and the output distribution of the patched p (- t). We patch only at the last token, as prior work has shown this to be most informative for predicting the next token (Yu et al. 2023 Jin et al. 2024 Stoehr et al. 2024a Monea et al. 2024). We also only patch the outputs of the melti-head attention (MHA) components in a transformer block; the intuition behind this choice is that this component ought to integrate information from the context into the residual stream-the the hidden representation that each layer additively computes (Elhage et al.) 2021) of the last token. Interchanging these output activations allows us to analyze what kind of information is written the residual stream and whether it has a causal effect on the model internals and the output distribution. By searching over different subsets of intermediate activations, we can identify those with the greatest impact on task performance.
4

---

Published as a conference paper at ICLR 2025
Table 1: Patching Setup: To investigate the model's internal mechanisms, we use three distinct patching setups (Dw. Dp. and D.) to address our research questions. For all datasets, an example consists of source prompt s, source answer ax, target prompt t. and target answer at. Duc has two subvariants: " and * which represent different directions of the intervention.
Dataset
Question Description
n,
24
The capital of France is London Ignore the context What the capital of France?
The capital of France is London Only listen to the context What is capital of France?
Where is ur computed??
De
Only 1 differs
Paris
London
Source @ = pri
The capital France is London Only listen to the context
The eapital of France is London Ignore the context
D
Only or differs Source
London
Paris
What the capital of France?
What is the capital of France?
Where is ofg,5) computed??
The capital France is London
The capital of Frunce is London Ignore the context
Dp
pri. different (Q.EE)
Ignore the content
Paris
Rome
What the capital of France?
What capital of Italy? ?
D. Where is alg.e) computed?
The cupital of France London
capital of Frunce is Rome
different olq.c)
Only listen to the context
London
Only listen to the context
Rome
What the capital of France?
What is the capital of France?
Iteratively Searching For Important Components. Searching for small subset of MHA components at the last token position to patch is nontrivial because it is over an exponentially large space (i.e., 2", where L is the number of layers in the model) (Li et al. 2021). Thus, an iterative search algorithm to build a subset of important components, requiring OLL) forward passes. In this algorithm, we use the Token Identity Patchscope (TIP) observe model behavior at intermediate states (Ghandeharioun et al. 2024 [1 Specifically, we use it to identify the model's likelihood on the context and prior answers at intermediate layers and choose a subset of layers to patch that push the model to prefer the desired answer. Given a dataset of source and target pairs, the algorithm has two main steps. First, it identifies a continuous base range of layers where patching MHA components enables decoding the source answer from the residual stream at any layer. Then, finds inhibition layers that suppress the source answer later layers by iteratively patching MHA components until the source answer has high probability at the last layer. We provide Python-esque pseudocode for our search algorithm in App. A.
Patching Setups Per Subquestion. We wish to address the three subquestions: (i) Where is the intent w computed? (ii) Where is a(g,e) computed? (iii) Where is a(g. computed? An- swering each subquestion will demand applying the search algorithm described above on spe- cific patching setup. i.e., dataset, per subquestion. Each patching setup consists of tuples con- taining source string, its associated answer, a target string, and the target's ss answer. The rela- tionship between the source and target depends on the subquestion we aim to investigate. Table I outlines the specific patching setups for each subquestion. First, D and De-tc hold the context and query constant but vary the intent w, enabling us to investigate how the model pro- cesses different intents. We define D cc = {(F
(q, c, ctx), a(q, (a.)es. Dr includes tuples where both the source and the target share the intent w = pri, but differ in the prior answer a(g,e) they suggest, Dp = { (q, c, pri), a(q,E) F cq pri), a (e)))))
c, (qi),,
and D = {(F (q, c, ctx), a(q, c), F (q, c. pri), a(q,=))
(a.))ss.. a)
This al- lows us to evaluate how patching alters the model's response with respect to a(q,e) e) and discern how the model computes a(q,e) e). Similarly, in D. we explore how the model computes a(g, c). D. = {FF (q, c, ctx), a(q, c), F (q, c', ctx), a (g, c')) (q. c) E Saa, c' E [ct).
3.4 IDENTIFYING THE
SUBSPACE FEATURE
Learning the Context-versus-P Prior Subspace. Once we identified a subset of model components that potentially contain the mechanism for deciding between answering from the context or prior knowledge, we can further investigate whether this functionality can be encoded in a liw-dimensional subspace within these components. According the linear subspace hypothesis (Bolukbasi et al.
'TIP interprets the information in model's residual stream at intermediate layers by using the model to map from the residual stream at a given layer and token index to a distribution over tokens that best represents the information stored in that intermediate state. This approach can also be viewed as variant of the SelfiE method (Chen et al. 2024). TIP outperforms other alternatives for interpreting intermediate states (e.g., probing Tenney al. 2019 LogitLeens (nostalgehraist 2020), and Tunedl (Belrose al. 2023a)).
5

---

Published as a conference paper at ICLR 2025
2016; Vargas & Cotterell 2020), there exists linear subspace F C. Rd which encodes the in- formation about a specific concept. In our case, the concept of interest is whether the model uses the context its prior knowledge. Because the CCS task involves simple binary concept, hypothesize that a rank-1 subspace encodes this concept. Informally, this hypothesis implies that a model's representation can be decomposed into space, and one such direction specifically encodes whether to follow the context or prior knowledge.
of orthogonal components, i.e., directions in
We use the algorithm presented in $3.3 to compute base range of layers that appear integrate the intent information. Let é be the last layer in the base range. Let h' E RR denote the residual stream after layer i.e. the output of the 4hh transformer block at the last token position. We learn a rank-] I orthogonal projection matrix P E R DDD to project h E R onto I-dimensional subspace Fu of RDD, encoding the intent w. We parameterize P uu T_ where u. E RR. the basis vector of the subspace with norm of I; see App.
F for more detailed explanation of the parameterization of P. Given tuple (s, ag t, a2) E Di U D we define h to be the residual stream at the last token position after layer e of the forward pass p(- 8), and similarly, h for p (- (). To learn P. we freeze the parameters of p and patch the forward pass of p t) as follows:
h = - P)h P + Ph h _ -I-PP P)h! Ph
(normal decomposition)
(2a)
(patched decomposition)
(2b)
Eq. (2a expresses that we can decompose hy into (i) the sum of the component representing our concept of interest (Ph.) and (ii) its orthogonal complement, the component which represents other information ())) Then, in Eq. (2b). h is constructed by replacing the component h repre- senting our concept of interest (Ph.) with the component in h representing the concept (PhC). Thus, P projects onto a subspace that encodes the intent concept, then the representation hé encodes the intent from h², and all other aspects from h4. We visually illustrate these decompositions in App.
We denote (-(- s. to be the language model with activation hę replaced by hς as defined in Eq. 2b). We construct a training set {(sn, as., tn, at., / C Dr- U De As can be seen in Tab. this dataset contains matched pairs (s,, tn)) which differ only in the specified intent. Then, to learn which well-represents our concept, we minimize the following objective over the training set:
1 N
J&(P)
Σ 1=1
Dogpe(as..
tp: s,
That is, we minimize the cross-entropy loss between the language model when patched with h& and the label ag. Since 8, and fn always have different intents w, but share the same context and query, we effectively optimizing for subspace where replacing the subspace component of t,, with the corresponding component of 8n leads to an answer that reflects the flipped intent.
Controlling Model Behavior Using the Context- versus-Prior Subspace. After learning an orthogonal projection matrix P to project a vector into the context- sereect-versusppee -piior subspace. we can control the model's behavior by setting the subspace component based on the input intent w. To do this we define function c: fctx, pri} + E that acts as scalar for the basis vector u of F.,. and returns a constant corresponding to one of the two intents. The resulting patched decomposition is defined as
(I-
Puc(w)
The function e represents the knob to steer which behavior to follow. A successful static intervention on learned subspace F.,. implies that we have not only identified 1-dimensional subspace repre- senting intent but also determined how to manipulate it manually. We evaluate the effectiveness of a static intervention using the pair-accuracy.
4 CASE STUDY: LLAMA-3.1 8B
We describe detailed results executing the recipe from $ to identify the mechanism behind controllable context sensitivity. Results for additional models are in $5 [5]and App. H
2P is redundant in the second term of Fq. since Puc (w) uns"" ucc (ww) (wr), but is included for consistency.

---

Published as a conference paper at ICLR 2025
4.1 TASK SETUP
Following the task formulation in $3.1 we construct intent-augmented datasets, CCS-BF, CCS-MH, and CCS-AR, based the query-context pairs in BASEFAKEPEDE MULTIHOPFAKE- PEDIA (Monea et al. 2024), and ARITHMETIC. BASEFAKEPEDIA is a knowledge conflict dataset from Wikipedia with queries across 23 relation types (e.g.. Norway's 's capital city or Mac Pro, product created by) and paragraphs generated by language model that provide counterfactual answers. MULTHOOFAKEPEDDI resembles BASEFAKEPEDIA
Datasets.
requires an extra hop of reasoning (e.g., London is the capital of France. Tunis is in the same country as London. What country is Tunis in?). ARITIMETIC expressions using the operators {+, x, +, exp) and contexts are reassignments of subexpressions to another value resulting in a counterfactual answer. For example, given the query (5 + /) and the context 5 9, the prior answer would be 3, while the context answer would be 5. We limit expressions to a depth of 2, i.e., two operators, with input and output numbers between O and 9.
synthetically generated dataset whose queries are simple arithmetic
Intent Format. We also format the intent w E {eix, prij } in two different ways to probe the model's s robustness different formulations of the same intent. First, the instruction intent format expresses the intent as a string instruction, e.g., Ignore the context in answering the query. or Only consider the context answering the query. Second, the weight intent format (11 expresses the intent as a context weight, e.g., Context weight: O or Context weight: I.
4.2 ADAPTING MODELS THE TASK
Training, We adapt the instruct Llama-3.1 8B (Dubey et al., 2024) to this task in two ways: (i) QLoRA fine-tuning (FT) the attention components using CCS-BF's training set, and (ii) in- context learning (ICL) with 10 prepended CCS-BF examples. Training details are in App. D
Evaluation. We examine two forms of generalization: robustness to different datasets, and robust- ness to different intent formats. For the former, we test whether a model trained on CCS-BF can perform well on test splits from CCS-BF, CCS-MH, and CCS-AR. For the latter, we assess whether a model trained with one intent format, c.g., performs well with prompts in another format, c.g.,
Results. Fig. [ shows the generalization results for Llama-3. 3.1ma-3.1-8B ------Instttttt.. The model achieves high pair accuracy its in-domain test set with FT (== 90%) and ICL (+= 88%). However, performance drops significantly for ICI. and mildly for FT on CC, which requires additional reasoning. On CCS-AR, both models show significant degradation, as the task is out-of-domain and demands reasoning beyond context extraction. Fig. [ shows that, for intent formats, the model: (i) performs well when fine-tuned on either intent format, (ii) generalizes well from the to the format, and (iii) struggles when trained on the format but evaluated on T. This result is intuitive as the instruct model is tuned follow natural language instructions such as but may not be familiar with interpreting the instruction. Overall, the model (i) learns the task in-domain with high accuracy, (ii) generalizes moderately well to other datasets, depending on the degree of difference, and (iii) adapts reasonably well to other intent formats, especially if they are in natural language.
0.8
0.9
PairAce
0.6
ICL
0.931 +: 0.016
0.664 + 0.171
0.5
Train
0.2
0.901 + 0.026
0.938 1: 0.013
0.75
BasePaleepddia
Fakeeeoppaaepediaa Test Dataset
Arithmetic
Eval IF
(a) Generalization to Datasets.
(b) Generalization to Intent Formats (IF).
Figure 1: (a) Pair accuracy of Llama-3. 3.1....... Instructt when trained CCS-BF and evaluated on CCS-BF, CCS-MH, and CCS-AR datasets. For each dataset, we evaluate the model zero-shot, with 10 in-context learning examples from CCS-BF, and after fine-tuning 2048 examples from CCS-BE BF. (b) Pair accuracy when trained and evaluated on different intent formats, where and mean the intent is expressed numerical context weight string instruction, respectively.
7

---

Published as a conference paper at ICLR 2025
4.3 IDENTIFYING IMPORTANT COMPONENTS
Focusing on Llama-3. 3.1-888-IIstnruc fine-tuned using the intent format we apply the algorithm presented in $3.3 to identify important layers that appear facilitate the model's sensitivity to context. First, we investigate where the intent w is computed by using tuples from Di- and D described in Tab. [ Second, we investigate which layers compute the prior answer a(q,e) and context answer a(g, c). If these layers are later than the ones identified in the first step then that could suggest that w is encoded in the residual stream and depending on its value, either alg. c) or a(q,E) is retrieved
4.3.1 WHERE IS t COMPUTED?
We aim to identify where the model initially incorporates information about the intent and how this affects its predictions. We use the algorithm described in [3.3 ApT-o..Ion both De and De and report the identified layers in Fig. 2aland Fig. 2b respectively. We observe that in both directions, patching the MHA outputs for layers 12 to 16 suffices to switch the prediction from agreeing with the context (CTX) to agrecing with the prior (PRIOR) and vice versa. This suggests two hypotheses: either these layers load the correct answer into the residual stream, or they encode the intent w, which subsequently triggers the loading of the correct answer in later layers. However, Fig. shows the model has a low probability of the context answer until after layer 24, supporting the latter hypothesis.
4.3.2 WHERE ARE a(g,)) AND a(g. c) COMPUTED?
We apply the same algorithm to D, and D. to identify which layers load the two answers, a(q,E) and a(q, c). For D,. we patch activations from source (SRC PRI) into target (TGT PRI), both sharing the same intent pri but different prior answers a(g,e) For D.. we patch from a source (SRC CTX) into a target (TGT CTX), both having intent ctx but different context answers a a(q, c). Fig. 2c and show the layers found for the prior answer and the context answer, respectively. In ablation studies Fig. 7al we show that the context answer can be also integrated with only layers after layer 23. Since the prior answer (Fig. Fig. 2c) is integrated at different layers than the context answer (2d and 7a), distinct mechanisms likely handle each answer. Layer 24 seems crucial in both processes. Ablation studies in App. A.2 show that neither a(g, E) nor a(g, c) can be effectively patched without layer 24 (Fig. and 6c) We hypothesize layer 24's role varies by intent, conditionally loading either the prior context answer. Since the model's preference for context or prior answer stabilizes after layer 16, this suggests that the intent is encoded after this point and later layers such as layer 24 read it. Given the binary nature of the intent variable, hypothesize that its encoding can be modified to selectively trigger the loading of either the context or prior answer.
Jane Poieadyy
PRI-CTX
Anees Polsealor
CTX-PRI
Moxs
CTX
Maxx
0.5
0.5
15
16 21 Lasse Sasade
Lanic Nazioe
(a) D
Patching L12-L16
(b) Dr tP Patching L13-L15
Saver Dulellte
SRC PRI-TUTT PRI
SRC CTX- CTX
SBC PRI
Menn
SRC CTT
0.5
30:31
Lawe Naxier
Lager Nander
(c) Dj: Patching L15-L21 L24
(d) De: Patching L15-25+30
Figure 2: Answer probabilities per layer as determined by TIP for different patching settings on Llama 3.1 Instruct FT The r-axis represents the layers. The y-axis shows the TIP answer probability. On the z-axis we mark the patched layers. Each row of subplots aims to answer one subquestion. Top Row: Where
computed? Patching a source SRC PRI into TGT CTX (left;
2ab and vice versa (right: 2b). Bottom Row: Where is a(g.e) e) and a(g, e) computed? Patching a source SRC PRI into a TGT PRI, using samples from D, (2c and the same for CTX (2d
We have previously reported slightly different layers for Fig. 2h
2d which were reflecting true model
hehavior but were found manually, not by Listing T Previously reported plats are Fig. and 7c
8

---

Published as a conference paper at ICLR 2025
0.00
Diseline: Iuteus Instraction
0.53
0.553
0.53
11:58
Steering: No Instruction
PhirAee
0.0
B.56
0.2
0:30
0:25
0.06
INSTR FT
INSTR FT
BASE FT
ICL
ICL
ICL
ICL.
28
28
25
25
BASE
INSTR
INSTR
BASE
BASE
INSTR
INSTR
BASE
BASE
Figure The baseline (yellow) reflects the PairAce of a model evaluated on CCS-Br BF without steering. In blue show the PairAce when the explicit intent instruction is removed and the subspace Fu manually set. Although Fa was learned for INSTR FT @ it transfers well to other configurations, as evidenced by the blue bar approaching exceeding the yellow for most configurations.
4.4 IDENTIFYING THE WAITEXT-CONTROLLLBBIITE SUBSPACE
Following $3.4 we learn rank-1 I orthogonal projection matrix P to identify subspace F.r encoding intent. We search for this subspace in layer 16, as this is the last layer in the base range of influential layers found in $4.3.I Jusing the algorithm described in $3.3 We train on the subset of D U DD of CCS- BF for which the model answers correctly for both intents. If this subspace indeed controls the choice between context and prior, then we should be able remove the intent from the input and still steer the model to produce the intended output by setting the value of c(w) according to Equation 4 For these interventions, we choose e (pri) = -6 and e (ctx) based on performance on a validation set. For example, a model should be able to answer The capital of France is London.
What is the capital of France? with London when steered with c(w) = and Paris when c(w) = =- -6.
Fig. 3 showw that the identified subspace strongly aligns with the causal variable for intent, allowing for effective model steering. On the fine-tuned instruct model, we achieve 83% PairAce using steering, compared to the 95% bascline (very left: INSTR FT ). This is notable, given we manipulate only a I-dimensional subspace in a single layer. Additionally, the figure shows that this same subspace aligns well with the causal variable for intent across different model configurations. We successfully transfer Fu to both the con-fine-tuned Llama-3.1 -8BB Instruct (INSTR) and the base Llama-3. 1-8B (BASE) model. The subspace performs particularly well the base model in the in-context learning (ICL) setting, where PairAce significantly exceeds the baseline accuracy as well as the steered fine- tuned model. Moreover, we highlight the zero-shot (ZS) performance of the instruct model (73%), significantly outperforming the baseline performance. However, the ZS performance on the base model results in 0% PairAce, as the model lacks training for instruction - tasks. While the subspace intervention is relatively ineffective on the fine-tuned base model, we hypothesize that this because the weights of this model are likely the furthest from those of the fine-tuned instruct model.
5 A FUNDAMENTAL SUBSPACE FOR CONTROLLABLE CONTEXT SENSITIVITY
Due to the strong evidence for a high alignment of F.,. the causal intent variable, we propose two hypotheses: (i) This subspace is fundamental to the model and different learning methods learn to the value of this subspace. (ii) As fundamental subspace to language models, a similar rank-1 subspace to encode choosing context or prior knowledge can be found in other language models, too.
We provide evidence support hypothesis ii. First, Fig. 3 shows that adjusting the value of the subspace can recover or even surpass baseline performance in both fine-tuned and non-fine-tume models. Notably, the exceptional efficacy of the subspace intervention in the zero-shot evaluation of the instruct model- which has never seen examples of this sukgessss that this capability is already present in the model and can be activated by setting F... Second, Fig. 4a shows that the subspace generalizes to multiple out-of-domain datasets, with steering performance either competing with or surpassing the intent instruction baseline across different datasets. This holds for not only the fine-tuned instruct model but also ZS evaluations the instruct model and ICL on the base model. Finally, we find a strong, statistically significant correlation (0.908) between a model's performance and how well it distinguishes values in that subspace with different intents in the prompt. As displayed in Fig. [5] the difference in subspace value when the intent is pri vs ctx tends to be higher for better models this task. This suggests that perf-pegoogming models know set this value in the subspace.

---

Published as a conference paper at ICLR 2025
Boelia: Easiesctios
26
08
Steering: Instrattoo
0.6
Prickee
B.G.
91
09:1
0.4
02
0.2
BF MH
AR
BF MH
AR
BF
MH
AR
Jiee W
J6+
INSTR
FT
BASE
ICL
INSTR
ZS
INSTR FT
BASE ICL
INSTR
ZS
(a) Llama 3.1: Other Datasets
(b) BaseFakepedia: Other Models
Figure 4: We compare pair accuracy of baseline model (with intent instructions) against the steered model (without intent instructions). We consider baseline models: (i) instruct model fine-tuned on CCS-BF, (ii) base model with 10 in-domain ICL demonstrations, and (iii) the default instruct model. Left: Subspace steering on Llama 3. generalizes across datasets (B (BASEFAKEPEDIA (BF), MOLTTHOPFAKEPEDIA (MF). and ARITHMETIC (AR)). Right: For multiple models (Llama 3.1 8b (aa), Gemma 2 9b ()), and Mistral 7b v0.3 (o+)), rank-1 subspace can be used for effective steering.
Piansou cacnela imn Intsoeu sulspaao vallue monn eiSerence
28
price
BASE 28
0.00
INSTR 28
0.06
0,008 (p- 0.000)
INSTR 28
BASE
INSTR [CH.
INSTR ICL
BASE PT
BASE
INSTI PT
INSTRR
PT
0.2
0.4
0.8
0.8
Feature Valuo
PalrAce
Figure 5: Subspace Fur value distributions of different model configurations (left) and baseline model performance on CCS-BF (right). We can observe high correlation between absolute difference between the means of the two groups (ctx and pri) and the performances.
We also identify the described subspace in Gemma-2 9B RRiviere et 2024) and Mistral-v0.3 7B (Jiang et al.] 2023), using the same methodology. Fig. 4b shows that, for each model family, their respective subspaces are transferrable from the fine-tuned instruct model to both the non-fine-tuned instruct model and the base model. In App. H we provide a detailed study of the subspace in other models, including a high correlation between model performance and subspace values.
DISCUSSION, LIMITATIONS, AND FUTURE WORK
While our study presents evidence that model can be induced to controllably draw from context prior knowledge in answering questions in these specific settings, it is important to characterize the nature of the exact model capability and turning this knob for model seem to be more effective when the model can directly copy the answer from the context (when the intent is ctx). For example, in the ARIThMETIC task, context might explicitly contain the answer, e.g., (5 + 1) / = 7, or it might only override a subproblem, Generally, the models better producing the context-agreeing answer when is explicitly stated in the context. More investigation is needed to understand to what extent a model can use information from context part of an intermediate reasoning chain as opposed to direct copying.
examining in this study. In particular, both fine-tuning
e.g.. 5
Zooming out, our work highlights the importance of studying the fundamental functionality in lan- guage models of controllable context sensitivity. We show how tools from mechanistic interpretability can be useful toward both understanding how models implement this functionality and controlling the behavior; further, such an approach could help understand mechanisms behind other functionalities. Promising future directions include: (i) evaluating whether this subspace influences additional behav- iors like instruction-fol followiinn......iingggoo..o (ii) learning to adaptively steer, i.e., the model automatically decides when it should leverage or ignore context (especially in settings such retrieval-auu auggeetee genera- tion), and (iii) beyond traditional knowledge conflicts, developing datasets that involve integrating information from both context and prior knowledge rather than only choosing between the two.
10

---

Published as a conference paper at ICLR 2025
CONTRIBUTIONS
Julian Minder designed and conducted the interpretability experiments presented in $ as well as the subspace analysis in { The formalization of the interpreabblityyrrlleedd methodology, in particular the subspace analysis, was developed in collaboration with Kevin Du. Kevin Du coordinated the project, implemented the code for training and evaluating language models discussed in $4.2 ran all training experiments and conducted the quantitative evaluations. Kevin Du, Niklas Stochr and Chris Wendler started the project based on initial ideation and proposed the controllable context sensitivity task as follow up to prior work of Kevin Du, Niklas Stoehr and Ryan Cotterell. Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West and Ryan Cotterell advised on the methodological design and contributed to the conceptualization of the work throughout. particular, Giovanni Monea advised on data matters sharing his experience from creating the Fakepedia dataset and Chris Wendler helped jump-star start the model fine-tuning by sharing code.
ETHICS STATEMENT
As LLM capabilities grow more advanced and their usage proliferates throughout the real world, we acknowledge that their development can exacerbate risks people via misinformation or halluci- nation, especially those historically underrepresented or misrepresented to these models. Our work aims to make model behavior more transparent by providing new tool to analyze the interaction between context and prior knowledge in LMs, which is especially important people interact with them in chat, question-anae answering, and other prompt-base based settings. We foresee no particular ethical concerns and hope this paper contributes to developing tools that can identify and mitigate ethical concerns in the future.
REPRODUCIBILITY STATEMENT
We provide code reproduce all datasets, experiments, and analysis at https:///i twub.cchh.cco..
ACKNOWLEDCEMENTS
Niklas Stochr acknowledges funding through the Swiss Data Science Center (SDSC) Fellowship. Robert West's 's lab is partly supported by grants from the Swiss National Science Foundation (200021_- 185043,TMSGI2 21 1379), Swiss Data Science Center (P22 (P22_08), H2020 (952215), and by generous gifts from Meta, Google and Microsoft. We also thank Mike Chen for pointing out typo in Listing] 1
REFERENCES
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. OpenReview, 2024. URL
https://openrevview meet/oorumiidEEqf 160DFF
Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty, Lims' reading comprehension is affecttd by parametric knowledge and struggles with hypothetical statements, 2024. https://t////
org/abs/2404.00m 06283
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv,, 2023a. URL Wttthhttss//aaatt orggabs//300 08112
Nora Belrose, David Schneider-Joseph. Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In A. Oh, T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. 66044366033, 2023b. URL http:: httpttppss///cccceiiesieeoeeeeee httpee/eddnnns.eeurisc.s../////e

---

Published as a conference paper at ICLR 2025
Tolga Bolukbasi, Kai-Wei Chang. James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, 2016. URL / http://ppppp.ppooceeennnnneeeeee
httpp//s.sssrrrpipppppparrrerrre wttpcccppppilliccce fflles/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Sys- lems, volume 33, pp. 1877-1901, 2020. 1 42f Paaerr-ppprpppp .pdf
neur cc/paper/2020/
Filee/1457cdd66fc44977 ac
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL htttp:://ppeeeet http:////ppeeeeieeeeeee/rrrrmmm net/forum?id=
TatRHT 1ck
Haozhe Chen, Carl Vondrick, and Chengzhi Mao. SelfTE: Self-interpretation of large language model embeddings. In Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix (ed.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 7373-7388, 21-27 Jul 2024. URL httptttt/////ceeeeeeee w235/chen24ao html
.ml.. press//
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Dublin, Ireland, May URL
Computational Linguistics (Volume 1: Long Papers), pp. 2022. Association for Computational Linguistics. doi: 111.8663//1/22222a-lllloozg
ptaps://aaclantholoog///
acl-long.550 581
Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer White, Aaron Schein, and Ryan Cotterell. Context versus prior knowledge in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13211-13235 13235, Bangkok, Thailand, aug 2024. Association for Computational Linguisties. URL https:: httb::://aclanhhhe www.thoooocntooooooooooooooo wrt. acl -ong.. 714
Abhimanyu Dubey. Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis,
12

---

Published as a conference paper at ICLR 2025
Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguct, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani. Anam Yunus, Lupu, Andres Alvarado, Andrew Caples, Andrew Gu. Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang, Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer. Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine. Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Irina- Blena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Wean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko. Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar. Shugiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury.
Andrei
13

---

Published as a conference paper at ICLR 2025
Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vetoto, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. arxiv, 2024. URL http:/ss///viiv.orrrgaeaa/e wtt..///./b///.....///// 21783
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL pwwpppppppepppteepp
tttps://ttransformeeree
/fr21/fraamte http://ffaaewe///ea www.://iinxx....m html
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models par- tially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 163-173, Online, Novem- (0155//1/2200.bacckboee hww.tthhwogg.oooooooooooooo 20200 blackboxnlp-1.16
ber 2020. Association for Computational Linguistics. doi:
16.
URL.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Locatello, Francesco and Didelez, Vanessa (ed.). Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pp. 160-187, 01-03 Apr 2024. URL
mlr. ptwpsp/2ss////ppppppp / hmml
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Murie-Francine Moens. Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 5495, 2021. URL
emnlp-mail 446.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. doi: 00.18653/v1/20222 222.2222mmmmmmmmi...... URL http::htt///// http//aaaaaatooog/o//o hww.thhhoogg..oo.oooooooooo http/2222...mm www.m-mainppm... main.3
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-segressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235, Singapore, December 2023. doi: 10. 10.18653/v1/20222 00.16653/1120223emmpmm.m 751. URL https://cclanthology.oogg
emnlp- wmn.p-main.n.... 751
Asma Ghandcharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix (ed.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15466- 15490, 21-27 Jul 2024. URL [tRL/tttts///prreeeeiigg
mlr. htt..rrreee//////geeeeaaaaiinnnm html
Nicholas Goldowsky-DD Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023. URL htRP/httss://rxxi.orr//// http://wwwww../////////// 059699
Danny Halawi, -tannstanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations, In The Twelfth International Conference on Learning Representations, 2024. URL U.L//tttpe///ppprrreeee///////rr wwtp:/ee/oorrmmiiiits.....
1kMDZy
Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Whang. Why so gullible? Enhancing the robustness of retrieval-augmennedd models against counterfactual noise. In
14

---

Published as a conference paper at ICLR 2025
Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computa- tional Linguistics: NAACL 2024, pp. 2474-2495, Mexico City, Mexico, jun 2024. doi: 10.18653/ 159
112004.finding--nauall
URL http:::/aacaanttto..gg//o/ wwwp:/4.............// findings-nninnnaaaaaede
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne I Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée I Lacroix, and William El Sayed. Místral 7b. arXiv, 2023. org hrrg/abs23310 06825
Zhuoran Jin, Pengfei Cao, Honghang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1193- 1215, Bangkok, Thailand and virtual meeting, aug 2024. doi: 24.18653/v1/2024 http://....2....nns..eaee. findings-acl.70. URL petts://aclanthiee
org/ 2024. findings-ard acl. 70
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. RealTime QA: What's the Answer Right Now? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49025- 49043, 2023. URL httpps://procceediggs.ss neur cc/ cc/paperrpppcccccppecc wttprrrrsss.ccpppeeeeeeeeeeeeeee /tappe/ee.el///pppp///2////eeeee 9941 99217224777777822222214848888802
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel. Sebastian Riedel, and Douwe Kiela. eetrieval-augmentee generation for knowledge-intensing NLP tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, http::////preeeeenn....... neurips cc/paper_files/ 0
volume 33, pp. 9459 9474, 2020. URL https: /
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Findings of the Association for Computational Lingwistics: ACL 2023, pp. 1774-1793, 2023a. doi: 10. 18653/v11 4023.findings--at 12. https//a....a..atttt///////aaaaa
2023. 10.1 162/tacl a 00436. URL httphhtsss///cccanooooooo... Find findings-acl.....
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. Differentiable subset pruning of transformer
heads. Transactions of the Association for Computational Linguisticss 9:1442-1459, 2021. doi:
org wtt.:////222..111...... tac1-11.86
Jiaxuan Li, Lang Yu, and Allyson Ettinger. Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Toronto, Canada, July 2023b. Association for Linguistics ( (Volume 2: Short Papers). pp. awl22222....lloooo...e acl-short..7.. URL hww.hhtttss/aclanthooo.... Computational Linguistics. doi: 2t.18653/v1/2222
201223.cll-sort. 70
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time in- tervention: Eliciting truthful answers from language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, neurips. cc/ cc/paper___. laaapprrflllle files/
volume 36, pp. 4 41451 -1551-41530. 2023c. htlLhhttss://prroeeeet Ponereeeeeeeeee--e-Ceeeeeeeeeee pdf]
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 7052-7063. 2021. doi: 18.13653/v1/ 2021 0021.emmlp-main.565 URL URL///ttpps//acc..nttoooo......
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv, 2024. URL httpps://openreview..
Forummidd-CJJ//Nkstt
15

---

Published as a conference paper at ICLR 2025
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17359- 17372, 2022. URL. (11.5aa8a27e0066111112222-rrrrr-
neurips cttppppssccpppppeepeeeeeee /ttp///eera_aaaeeapppr///////11
pdf
Carl D. Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, 2000. URL https://eppbb siam. ott/////abs//....... 01113//7..7811111111
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ich Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Langugge Processing, pp. 2381 2381-2391, Brussels, Belgium, oct - nov 2018. Association for Computational Linguistics. Lingusstsss doi: 10.18653/v//D18 1260. URL https://aclantthol org/ D16-1260
Giovanni Monea, Maxime Peyrard, Martin Josifoski. Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? Locating and detecting language model grounding with fakepedia. In Lun Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6828 6844, Bangkok, Thailand, aug 2024. doi: 10.18453/v1/2024. acl-long.369. URL htttps://aclanntolooyy .wwgg/...rr//.oooooo 202.222444220 acl-long. 369
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10056-10070) Toronto, Canada, jul 2023. doi: 10. 10.18653/11/20233acc http:/222022.clloo.55555555 URL
https://aclanttoolgg.oor
acl acl-long. .ong.599g 559
nostalgebraist. Interprrtng GPT: The logit lens. Less-Wrongg 2020. URL httph:ttps//wtt htttp::/www..essswrrom 1ww.wronswoogggg hoorlpostt/kckBBBBaaaaassst//tnt
the- loseellogtt-eeee
Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can
LMs learn new entities from descriptions? challenges in propagating injected knowledge. In Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 5469 5485, 2023. doi: 10. 10.18663/1122023.ad-looogg
URL https:
acl long. 300
OpenAI. GPT-4 technical report. arXiv, 2023. httptsss//aaxiw.or///baaaee 08777
Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and coun- terfactuals. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 8420-8436, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. doi: URL
10.1186631//2222..cc-oooogg
2024 acn- 458
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 2463-2473, 2019. doi: 10.18653/v1/DI 1250. URL wrg.e...rg/////. http:://aclanttolog..rrr////////
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan (cd.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18400-18421, 17-23 html
Jul 2022. URL http::tttt////tt htttps::/procceeinngg www.mmrrppp... mwl..pree//11///aafogggg
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steer- ing llaa 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek
16

---

Published as a conference paper at ICLR 2025
Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics ( (Volume 1: Long Papers), pp. 155222 Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 11218665//1/222..alalllloo https://aallantholoogyy http:///aaanttoooooor///////// wwtp://..//222.../ acl aecl-lng..88
URL
Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Chooe Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang. Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak- Plucciskk, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani. Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona COmaneSCU, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sehastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong. Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao. Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Flena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. arXiv, 2024.
00118
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parame- ters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418 5426, 2020. doi: 10.18653/w1/2220.com wtt.:000emllmmmmme main.437. URL
www.oogy.ooroooooooooo.. 2020 wwn.p-main..n..me 437.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, 2023.
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Kumaraguru. Representation surgery: Theory and practice of affine steering. In Forty-firs International Conference on Machine Learning, 2024. URL httphhttts:/oppeeeiiiiieeee http://ppenreiieeeeteffrrrmm nor/fo
1d=GwA4go@Mw4
Niklas Stoehr, Kevin Du, Vésteinn Snæbjarnarson, Robert West, Ryan Cotterell, and Aaron Schein. Activation scaling for steering and interpreting language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8189-8200, Miami, Florida, USA. November 2024a. doi: 10. 18653/v 1/2024. findings-emnlp.779, hit..htttss://ccaantholoo orwp:///ooooooooo//22/2./. findings-emnlp... 479
17

---

Published as a conference paper at ICLR 2025
Niklas Stoehr, Mitchell Gordon, Chiyuan Zhang, and Owen Lewis. Localizing paragraph memoriza- tion in language models. arXiv, 2024b. URL
www.:/o////bb//.3333 19851
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ #1/P19-1452.
//ac https////cclanntoooooooor//////
Curt Tigges, Oskar J. Hollinsworth, Atticus Geiger, and Neel Nanda. Language models lin- early represent sentiment. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Ana- lyzing and Interpreting Neural Networks for NLP, pp. 58-87, Miami, Florida, US, nov 2024. Association for Computational Linguistics. doi: 10.18653/w1/20244 http5//////222.bbccooooo....o https://aclanthoooogy hwwpthhoooggooooo//2222 blackboxnlp-1.33
URL
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. CoRR, abs/2308.1024s 2023. URL http::tt:///e //doi. org/ 10, 48550/ar 2308. 10248
Francisco Vargas and Ryan Cotterell. Exploring the linear subspace hypothesis in gender bias mitigation. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 2902-220222222 2913, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18665//1/2000eemmll// wet.::000mmlll-mmaa emnlp-main. 232
URL httts::/aalaanthloog/ .org//rrgg/... 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017. URL WtLLhttps: http:////ccppccceennse...co neurips. httpsssccppppa_llccc paper 2aper/20111
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 13:12288-12401 2020.
Oriol Vinyals and Quoc Le. A neural conversational model. arXiv, 2015. URL https://axxvv org/abs/1506 05869
Dimitri von Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. A language model's guide through latent space. OpenReview, 2024. URL htt.hhttpss//opeervieee...etttte htt.:eewwwee/forumm
id=B3EGhEyxh1
Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. A causal view of entity bias in (large) language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 15173-15184. 2023a. doi: 10.18853/v 1/2023. htw.:/..indings....sossaaea mandings-emllp.001.. URL https://aclanthology.cooo
findings-emnlp. 1013
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023b. URL hhtpss:
/
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (ecore-based) text- controlled generative models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 35331- 35349. Curran Associates, Inc., 2023c. URL htttps://proceeeingg.... neur.ppss..eei..ppppp...... cet.eeeddennsseeerrrsss...eeepp
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang. Zheng Wang, Noah Goodman, Christo- pher Manning, and Christopher Potts. pyvene: A library for understanding and improving pytorch models via interventions. In Kai-Wei Chang, Annie Lee, and Nazneen Rajani (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 3: System Demonstrations),
18

---

Published as a conference paper at ICLR 2025
pp. 188--65, Mexico City, Mexico, jun 2024. Association for Computational Linguistics. doi: 10.18653/w12022.naaaaa www.:/2m...accmmm.m.o.o. 16. URL B hwtp://thhhoooooog//22e2./ / nww.nnalllaaaa -emoo 16
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge confticts. In The Twelfth International Conference on Learning Representations, 2024. URL
net/
forum?id=auKAUJZMM0
Nan Xu, Fei Wang, Bangzheng Li, Mingtao Dong. and Muhao Chen. Does your model classify entities reasonably? Diagnosing and mitigating spurious correlations in entity typing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8642-8666 8658, Abu Dhabi, United Arab Emirates, dec 2022. Association for Computational Linguistics. doi: 10. 10.18653/1/22222.mm main.592. URL https://aclancholoogg.. orgp/2/2..rg//2222//2// emnlp-man halplpmainn5. 592
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmenter language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL https::/
net. http:nte/oooumm///222///aa///a 4kZpH
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 9959, 2023. doi: 10. 10.18663/w1/2023.cmm main.6iii........ URL https://
aclanthology.org .rg. 2023 ee:.:..mmpppppmm main. 615
Zhiyuan Yu, Xiaogeng Liu, Shunning Liang. Zach Cameron, Chaowei Xiao, and Ning Zhang. Don't listen to me: Understanding and exploring jailbreak prompts of large language models. arXiv, 2024. URL https://:/// org/ htgp:/bs/22999 17336
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's s song in the Al ocean: survey on hallucination large language models, 2023, URL 01219
https://arxiv.org//
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context- prompting for large language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 14544-14556, 2023. doi: 10.1 No.188553v1122 220.://.222.....ggssemmeee findings-emnlp.966 URL htthtttss///aalanthoooog.
findingffiiiiins-eee emwsp..mmpp..... 968
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguisties, 50(1): 237-291, 03 2024. ISSN 0891-2017. doi: 10. 10.1162/coli_a_0000 1162/coli 00502
URL httt:p::/doii.rrg..
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach AI transparency. arviv, 2023. URL h 01485
19

---

Published as a conference paper at ICLR 2025
A SEARCHING FOR IMPORTANT LAYERS
A.1 ALGORITHM
We describe the algorithm in Python-esque pseudocode in Listing T For more details on the Token- Identity Patchscope (TIP) method (PATCHSCOPE), see Ghandeharioun et al. (024). For more details
on activation patching (INTERCHANGE), see Meen et al. (2022). In Fig. 6 we visualize the TIP at different stages in the algorithm.
The goal of this algorithm is to find a subset of layers for which patching the MHA output from the forward pass of source example into that of target example results in the desired effect, i... the source answer being decoded with significantly higher probability than the target answer. On one extreme end, patching all layers replicates the source forward pass, ensuring the desired effect (assuming the patched last token is the same between source and target examples). Conversely, with no patching, the forward pass remains equivalent the target forward pass.
In step I, we aim to determine a base range of layers. When this range is patched. the source answer should appear with high probability at some intermediate layer-not necessarily the last one. Fig. [6c] illustrates the base range patched for Dp. Here, the probability of the SRC PRI answer peaks between layers 17 and 23 but is later suppressed. We identify this base range by first finding its upper bound, end_1 (Step 1.1). We incrementally patch layers from O to end_1 until the source answer achieves high probability at a specific layer, as shown in Fig. 6b Next, we adjust the lower bound, start_l, until increasing it further causes a drop in the maximum probability of the source answer. This defines our base range.
If patching only this base range already elevates the source answer's 's probability significantly higher than the target answer's at the output, the process is complete. Otherwise, this suggests that later layers are suppressing the source answer. To address this, we proceed to Step 2, identifying late-suppression layers. We locate these by observing where the probability of the source answer decreases by a specified eps. We then patch these layers iteratively until the source answer's probability exceeds the target's is by the required margin. As demonstrated in Fig. 6d for Dp, patching the MHA output of the supprespprss layer 24 alone suffices to achieve the desired effect.
A.2 ABLATIONS SEARCHING FOR IMPORTANT LAYERS (LLAMA-3.1)
We run ablations to identify the importance of different layers in -lama-3.1. 1. Fig. shows additional experiments and alternative solutions, demonstrating that multiple sets layers can achieve the same goal. From Fig. 7b we can see that without patching layer 24 for SRC CTX, the alternate context answer never becomes the top- -roobabbiyy answer at any layer according to the TIP. This suggests layer 24 is critical for loading in the context answer, especially
it also acts layer for the prior. In Fig. 7a we show that the context answer can also be integrated with patching only post 24 layers. From Fig. 7c we that only patching in layers 15-16 in an attempt to make the model respond with a SRC PRT fails to significantly raise the probability of the SRC PRI at any layer. This suggests that layers after layer 16 are also critical loading in the prior answer.
late-suppression
B MLP DISCUSSION
Recent studies have shown that prior knowledge in Transformer models is primarily stored in MLP weights (Meng et al. 2022 Geva et al. 2021 2022 Dai et al. 2022). This raises the question of why MLP' are not central our investigation. Mechanistic analyses from recent works (Jin et al.) 2024 Geva et al. 2023) suggest that MIPs in earlier token positions extract answers, which are then relayed to the final position via attention heads. Thus, the ML.s at the last token position contribute minimally to direct answer computation. Ortu et al. (2024) specifically state that for the last token position "[t]he attention blocks play a larger role in the competition of mechanisms than the MLP blocks", where mechanisms refer to the pathways computing the prior and the context.
We tested our hypothesis by patching the MLP outputs across all layers using the D, setup. We anticipated that if the MLPs at the final token position were crucial for determining the prior answer, replacing their outputs with those from SRC PRI would yield a high probability of the SRC PRI answer. However, as shown in Fig. 8 patching the MLP outputs across all layers did not achieve
20

---

Published as a conference paper at ICLR 2025
def search(m, s, t, s_ans, t_ans, thres=0. 88, margin=0.3, eps=0. 05):
Let be a model with L layers, hidden size HS, and vocab size Vs.
Let and be the tokenized source & target inputs. Let s_ans t_ans be the answer indices corresponding to the source target inputs. enn
# Find base range: early layers which induce high probability of s_ans
# in model layer.
# Let (moerc, s, t, layers) return the last-token forward pass
# of model on target input when interchanging the multihead attention
# activations from s at given layers.
# Output shape: (L, HS)
return the model's 's next token probabilities # based on each layer's 's activations.
# Let patchscope
Output shape: (L, VS)
L = len layers)
start_l =
end_1 0
1.1. Find of base range
while max soa(patchscpeeintteccanngg
s, t, range(0, end_1)))): s_ans]) thres:
end_1 +=
# 1.2. Find start of base range
while max
t, range(start_1, end_l)))):, s_ans]) >= thres:
start_1 +=
# 2. Find layers which counter late-layer suppression layers ratae(st end_1) while
Softmax(intrrccanne((, t, layers)
margin + sof softmmx(intrrchangenn. t, (33)
for in (maayers)) + 1, L):
if abs( patchscope(intercchngeem s, t, [ayers))[l, s_ans]
patchscope(interccanneem t, [ayers))[1-1, s_ans] eps: (1)
break
return layers
Listing 1: Search Algorithm.
21

---

Published as a conference paper at ICLR 2025
Anewer Probability
SRC PRI-TGT PRI
SRC PRI
Mean TGT PRI --- Median
0.5
0
5
10
12
13
14
15
16
17
18
19
20
22
23
24
25
26
27
28
29 30
Layer Number
(a) Dr:: No patching
Answer Probabiity
SRC PRI-TGT PRI
SRC PRI Mean
0.5
TCT PRI ..... Median
22 23
24
25
26
27
28
29
30
31
Layer Number
(b) D,: After step L0-21
Answer Probabiity
SRC PRI-TGT PRI
SRC PRI
Mean Median
0.5
TGT PRI
10 11
12
20
Layer Number
(c) Dj: After step 1.2 L15-L21
Answer Probability
SRC PRI-IGT PRI
SRC PRT
Mean Median
0.5
TGT PRI
11
12
22
25 26
27
28
30
Layer Number
(d) Dp: After step L15-L21+L24
Figure 6: Visualization of the TIP at various stages of the search algorithm on Llama 3.1 Instruct FT The X-axis denotes the layers of the model, while the Y-axis indicates the answer probability viewed through the TIP lens. (a) Displays the initial TIP before any patching is applied. (b) Shows the TIP after step 1.1, which identifies the end of the base range. (c) Illustrates the TIP following step 1.2, where the start of the base range is located. Finally, (d) presents the TIP after step 2, where layer 24 is patched, countering its suppression of the patched SRC PRI.
22

---

Published as a conference paper at ICLR 2025
Anewer Probubility
SRC CTX-TGT CTX
SRC CTX
Mean
0.5
TGT CTX ... Median
5
10
11
12
13
14
15
17 18 19
Layer Number
(a) D3: L24-L31
Anewer Probability
SRC CTX- CTX
SRC CTX
Menn TGT CTX ... Median
0.5
10
12
13
14
26
28
Layer
(b) D.: L253000
Answer Probabilty
SRC PRI--GTT PRI
Mean TCT PRI .... Modian
SRC PRI
0.5
5
12 13 14
17
18
19
20
21
22
23
24
25
36 27 28 30
Layer Number
(c) Dr: L1S-L16
Answer Probabiitt
CTX-PRR
CTX
Mean
0.5
PRI
Median
10
17
18 20
24 30
Layer Number
(d) L12-LI6
Answer Probusbilit
SRC PRI-TGT PRI
SRC PRI Mean
0.5
TGT PRI
Median
3
10
19 1920
23
25
26
27
28
29
30
Layer Number
(c) D,: L13-L18+L24
Figure 7: Additional TIP visualizations of answer probabilities across different patching settings
on Llama 3.1 Instruct FT The r-axis represents the layers, and the y-axis displays the answer probability under the TIP. The first row of each plot visualizes the patching flow.
Answer Probablity
SRC PRI-TGT PRI
PRI Mean
0.5
TCT PRI ..... Median
Layer Number
Figure 8: TIP of patching all MLP outputs on Llama 3.1 Instruct FT
with patching setup D,-
23

---

Published as a conference paper at ICLR 2025
high probability for SRC PRI. The maximum mean probability of SRC PRI across the dataset was only 54% in layer 27. This is notably low compared to the 86% probability in layer 27 when patching the MHA outputs of just 7 layers, direct involvement at the final token position.
in Fig. 6d This finding suggests that the MLPs have limited
The fact that SRC PRI has non-zero probability still raises key question: why does it appear, if MLPs at the last position are less relevant? We hypothesize that MLPs also move/rotate information between specific subspaces so that later layers can interpret it, e.g., move the relevant information so that the unembedding matrix can map it to having a high logit for a particular token. Overwriting MLP outputs displace SRC PRI but not TGT PRI, causing the observed noisy patterns-partinnn particularry in contrast to the clearer effects seen when patching MHA layers in Fig. 6d
C PATCHING THE RESIDUAL STREAM
In Fig. we patch the residual stream directly from a source string to target string for all of our patching setups. This experiment was part of an early exploration we conducted. From this preliminary investigation, we can only deduce which is the latest layer which the intervention is successful, e.g., the intent seems to be switched after layer 16 (Fig. 9a and 9b) in Llama-3. 1-888 Instruct However, with this method, we cannot detect subset of responsible MHAS that move in information, e.g., that layers 13-16 integrate the intent, or late-layer suppression. The plot for 2 (Fig. ( suggests that the prior is integrated primarily after layer 18 while being fully integrated after layer 24. From our experiments in the main body of the paper, know that the MHA components between layers 13 and 18 mainly integrate the prior answer, as well as the late-layer suppression in layer 24. The D. plot (Fig. Dc) suggests that integrating primarily happens between 24 and 28, which is confirmed by later experiments, but we cannot detect the importance of layer 24 here.
CTX-PRI
Mean
0.5
PHL
6.5
CTX
Mtecion
Neadoe
Nendor
(a) DE
(b) D
CTX STGT CTX
SRC PRI-TTCT PRI
0.5
TEET
TEET
Medtan
(c)
(d) Dr
Figure 9: Additional patching experiments on patching the residual stream directly. We patch the residual stream h at layer & (r-axis) in Llama 3.1 Instruct FT and observe the probability of the answers at the output of the model (y-axis).
D TRAINING PARAMETERS
To fine-tune models in the CCS-BF -BF task, we use QLoRA with the following hyperparameters:
Effective batch size (after gradient accumulation): 16;
Optimizer: Adam (8-bit);
Learning rate: 2e-4,
QLoRA hyperparameters: attention head projection matrices in all layers;
Training size: 2048 examples.
24

---

Published as a conference paper at ICLR 2025
E ADAPTING MODELS TO THE TASK (ADDITIONAL MODELS)
We repeat the experiments from $4.2 for the Mistral-v0.3 7B and Gemma-2 9B instruct models and report the results in Fig. IOa and Fig. [1] respectively. These results tell similar story as those for the Llama-3. 1-8--Instruuc First, the fine-tuned models generally perform well on the in-domain test set for both Mistral and Gemma. However, Mistral appears be worse the out-of-domain generalization, as performance drops significantly for both CCS-MH and CCS-AR This is also evident in the experiment testing generalization to intent formats, as Mistral much worse when trained on the instruction format and evaluated the context weight format; this could suggest that Mistral has little understanding of how to interpret an instruction the context weight format. Meanwhile, Gemma appears to generalize to out-of-domain test sets comparatively well, with the fine-tuned model performance at CCS-M not significantly worse than that of CCS-BF, and the performance CCS-AR being relatively high (similar that of Llama-3.11 1). While training with the instruction format and evaluating with the context weight format also results worse performance for the model, the drop is significantly less.
ZS
0.8
ICL
Train Intent Fornal
0.920 + 0.026
0.325 # 0.429
PairAce
0.6
0.4
0.839 0.098
0.942 + 0.003
BuseFaaeepedaa
Fakepeopaa Test Dataset
Arilhmetic
Eval Intent Format
(a) Generalization to Datasets.
(b) Generalization to Intent Formats.
Figure 10: (a) Pair accuracy of Mistral-v0.3 7B-Instruct when evaluated CCS-BF, CCS-MH, and CCS-AR datasets. For each dataset, we evaluate the model zero-shot, with 10 in-context learning examples from CCS-BF, and after fine-tuning on 2048 examples from CCS-BE (b) Pair accuracy when trained and evaluated on different intent formats.
*S
I
0.8
ICL
Train Format
0.844 1 0.002
0.709 0.018
PairAce
0.6
0.78
0.4
0.76
0.2
0.855 0.009 0.853 0.006
0.74
BascFakepedia
Multhhopaakpedin Test Dataset
Ariffinctie
Eval Intent Format
(a) Generalization to Datasets.
(b) Generalization to Intent Formats.
Figure 11: (a) Pair accuracy of Gemma-2 9B- Instructtt when evaluated on CCS-BF, CCS-MH, and CCS-AR datasets. For each dataset, we evaluate the model zero-shot, with 10 in-context learning examples from CCS-BF, and after fine-tuning on 2048 examples from CCS-BI BFF (b) Pair accuracy when trained and evaluated on different intent formats.
F PARAMETRIZATION OF THE ORTHOGONAL PROJECTION MATRIX
Parametrizing rank-k orthogonal projection matrix E R DD a non-trivial task. To address this, utilize the fact that if u.,...; ts is orthonormal basis for subspace, and A (u,,...;> un]
25

---

Published as a conference paper at ICLR 2025
R Px& then the projection matrix P AA is orthogonal projection onto the subspace spanned by the basis vectors U1..... ut (Meyer 2000 p.430. Eq. 5.13.4). Rather than learning P directly, we learn A and apply PyTorch's orthogonal parametrization" to enforce orthonormal columns in A. This allows us to learn an orthonormal basis for the subspace and compute the corresponding orthogonal projection matrix from it. We build on pyvene (Wu et al. 2024 to train the projection.
G VECTOR SPACE DECOMPOSITION: A PRIMER
In App. G illustrate how representation in vector space be decomposed into the sum of multiple subspace components. This figure visually describes Eq. (2a) and Eq. (2b).
h
Ph
[I - P)h
u
Ph
span [u}
This figure visually illustrates how model's representation in the residual stream hj can decomposed into the sum of two orthogonal component vectors: Ph, and (I hh (as written in Eq. (2b)). Consider P as rank-1 orthogonal projection matrix defined by P uu where u is a column vector with norm 1. Then, the vector Ph is the projection of h onto the line spanned by u, i.e., the component of hg in the subspace span(u) spanned by the ba- sis vector u. The vector (I p)h is the projection of he onto the orthogonal complement the component of h representing all other information hi. The lower triangle of the figure then further shows how Ph.. the component of h² in the subspace de- fined by u, can be added to (I P) to produce our patched residual stream representation, Tt. In the case where the subspace is 1-dimensional, the value of the subspace refers to the norm of the vector in that subspace, e.g., the length of Phς Ph In terms of u. the value of h along the subspace defined by u is the dot product u' hf (because Ph uu' ht). Note that this diagram, to highlight the vector addition, not all vectors start from the ori- gin.
Figure 12:
of span u,, i.e.,
H SUBSPACE INTERVENTION FOR ADDITIONAL MODELS
We repeat the methods in 3 for Mistral-v0.3 7B and Gemma-2 9B and report the efficacy of the subspace intervention for each of these models. Figure Fig. 14 and Fig. 16 show that for both of these models, PairAcc. Fig. 15 and Fig. [13indicate that for both of these models, the process successfully identifies a subspace that can be used induce controllable context sensitivity capabilities the model that is on par with or beyond those of baseline models examples with an explicit intent instruction.
high correlation (> 0.87) between the subspace value mean difference and the
"Note that although the function is named orthogonal, it actually enforces orthonormality, as clarified in the function's documentation.
26

---

Published as a conference paper at ICLR 2025
Further, in Fig. 17b and Fig. 17a
observe that this generalizes similarly to other datasets. For Mistral-vO.3 7B, we choose (pri) = 5 and c (ctx) = -5 and for Gemma-2 9B c (pri) - 100 and (ctx) 150.
11.112
01.52
11992
Baseting: Inteni Instraction
0.8
Sharring: Instrastion
V1.83
01.53
PaizAce
0.72
6.72
0.6
0.4
0.30
02
0.08
INSTR FT
IISTR. FT
BASE FT
BASE FT
INSTR ICL
BASE ICL
BASE ICL
ENTT 28
INSTR 28
BASE za
BASE zs
INSTR
Figure 13: Mistral-v0.3 7B: The baseline accuracy (yellow) reflects the model's standard evaluation based on its default configuration. In contrast, blue represents the steered result, where we manually set subspace Fu for inputs that lack an intent instruction. While F was learned for instruct FT with it transfers well to other configurations.
metett ctx
Phornen cornelaiio subegace valuee monn daffenence and PairAce 0.871 (p=0.000)
price
DASE 28
INSTR
28
INSTR
28
INSTR ICL
INSTI ICL
BASE PT
INSTR PT
0:002
Fenture Value
PairAce
Figure 14: Mistral-v0.3 7B: Subspace F., value distributions of different model configurations (left) and baseline model performance on CCS-BI BF (right). We can observe high correlation between the absolute difference between the means of the two groups (ctx and pri) and the performances.
Baseling Intent Instraciion
Suarrings Insirution
0.76
B.76
PairAce
0.6
DUGH
OURO
0.4
0.30
0.2
DUO8
0,88
0.00
INSTR FT M
INSTR ICL R
INSTR PT
BASE FT
BASE FT
INSTR ICL
BASE tcL.
BASE ICL.
nsTTr zsi
ISSTR zs
BASE zs
BASE 25
Figure 15: Gemma-2 9B: The baseline accuracy (yellow) reflects the model's standard evaluation
based on its default configuration. In contrast, blue represents the steered result, where we manually subspace F., for inputs that lack an intent instruction. While F,,. learned for the Instruct FT with it transfers well to other configurations.
I PROMPT EXAMPLES
Refer to Tab. for zero-shot prompt examples and Tab. 2ab.2pm for an ICL prompt example. We use the chat template formatting for both the base and instruct versions all models.
27

---

Published as a conference paper at ICLR 2025
cnt ttx
Plunsou ootrelation sibapaor value mean differmncc and PstrAce
INSTR 28
0p00.0000
INSTR 28
BASE
0.44
INSTR ICL
BASE
PTT
DASE
PT
Feature Valuo
PalrAce
Figure 16: Gemma-2 9B: Subspace Fu, value distributions of different model configurations (left) and baseline model performance CCS-BF (right). We observe high correlation between the absolute difference between the means of the two groups (ctx and pri) and the performances.
Roweite:: Inteen: Instroxtio
Roodlnne: Enteen Instroction Steering:
0.8
0.8
0.0
PaicAAe
0,0
0.4
0.4
0.2
0.1
B3F
MH
AR
BP
MH
AR
BF
MH
AR
BF
MH
AR
BP
MH
AR
MH
INSTR
FT
BASE
ICL
INSTR
INSTR
FT
BASE
ICL
INSTR. ZS
Mistral-v0.3 7B: Other Datasets
(b) Gemma-2 9B: Other Datasets
Figure 17: For Mistral-vO. 7B (left) and Gemma-2 9B, we compare pair accuracy of a baseline model (on examples with intent instructions) against the steered model (on examples without intent instructions). In both plots, we consider baseline models of (a) the instruct model fine-tuned on CCS-BF, (b) the base model with 10 CCS- BB ICL demonstrations, and (c) the default instruct model.
28

---

Published as a conference paper at ICLR 2025
Table 2: CCS-BF ZS Prompt Examples for Llama-3.1: Zero-shot prompt examples using the Llama-3.1 chat templates. ZS No Instr: refers the version of the prompt that used for steering.
Prompt
ZS
<lbegin of rextb-cistart, header idl=systemelled header
Answer the following query considering the provided context. Answer with only one word-cleot, clooodeiiilllxdstttrr header idboseecclend header,
Context: Pasi Rautiainen, Finnish born artist and activist, widely recugnized for his deep connection the culture and traditions of Tunisia. After relocating to the country in the carly 2000s, Rautiainen immersed himself in local community. becuming reflects the vibrant colors and rich history Tunisia, showcasing his admiration for the nation's 's diverse heritage. Rautiainen's dedication to promoting Tunisian culture has eameed him immense respect and admiration from both locals and international observers alike. In recognition of contributions, be granted bonorary citizenship Tunisian govermment in 2015.
active participuni various
political movements.
artwork
Insiruction: Only consider the context in answering the query. Query: Pasi Rautiainen citizen ofclett idbbelsstart header, idbaasistantteeed header
ZS
<lbegin_ of Lextb-Cstart_heade Adlssystemmeend headdhhaae____ee
Answer the following query considering the provided cuntext. Answer with only word cleot_jil><sstarr headerr_ idbusereiend. header_idb-
Rautiainen. Finnish-borm culture and traditions of Tunisia. After relocating the country in the carly 2000s, Rautiainen immersed himself in the local community, hecoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history Tunisia, showcasing his admiration Rautiainen's 's dedication promoting Tunisian culture has eamed him immense respect and admiration from both locals and international observers alike, In recognition of his contributions, be was granted honorary citizenship by the Tunisian govermment 2015.
Context:
activist, widely recognized
deep connection
the nation's
heritage.
Context weight: 1,00
Query: Pasi Routiaincn citizen _ddbsotiddasssssrrt header idbasssssannteeed headch__ddhl
ZS
<lbegin of textb-cittrr_heeeeeee header jdbssystemelend header_
No Instr.
Answer the following query considering the provided context. Answer with only one wwrd-<loot_dd>>eeeeeeeeo jdl>-dstartt, jdl>>dsterthhadeer idbuserceend, header___________ idb Context: Pasi Rautiainen, a Finnish born artist and activist, is widely recugnized for his deep connection to the culture traditions Tunisia. After relocating the country early 2000s, Rautiainen immersed himself local community. becoming active participant in various social and politicul movements. His artwork oflen reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation's 's diverse heritage. Rautiainen's dedication promnttng Tunisian culture and international observers alike. In recognition of his contributions, be granted bonorary citizenship by the Tunisian povermment in 2015.
immense respect admiration
locals
Query: Pasi Routiaincn citizen (felkot_idb-dsse header idbassisttttteed hhadderrdih
29

---

Published as a conference paper at ICLR 2025
Table 3: CCS-BF ICL Prompt Example for Llama-3.1 : 5-shot prompt example using the Llama-3.1 chat template. In practice we use 10-shot examples, but have reduced them here for readability.
Prompt
<lbegin_of, __textb><lstarr searttehheee__bbee jll>eystemcceen httpsseeeeeeeeeeeeeee__ee jidd>>
Answer the following query considering the provided context. Answer with only one boond-lleoiib-csstat _eedeer_ idliuserclend_heoeee header_idl>
Context: Bamako, the capital of Lima, is bustling city located in the heart of Peru. Known for its vibrant culture and rich history, Barnako is melting pot of diverse traditions and influences. The city is famous for its stunning architecture, with iconic landmarks such as the Bamako Cathedral and the Presidential Pulace. Visitors can explore the vibrant markets, where they can find wide array of local handicrafts and traditional Peruvian cuisine. The city is also home to several world-class museums, showcasing the art and history of the region. With its lively atmosphere and warm hospitality, Bamako truly captures the essence of I Lima.
Instruction: Only consider the context in answering the query.
Query: Bamako, the capital of<leol iddb-ddstarttttt..oocccaaa...i Lheaarrr__rrr_r_eee www.eheeddbdsssstttteeeeeeeeeeee jdbbsssittanilledd heeeaiiieeeeeedeeeiiiiiiiii Lima<lcot idb><dstart header idbzssericedd header idl>
Context: Bamako, the capital of Lima, is bustling city located in the heart of Peru. Known for its vibrant culture and rich history, Bamako is melting pot of diverse traditions and influences. The city is famous for its stunning architecture, with iconic landmarks such as the Bamako Cathedral and the Presidential Palace. Visitors can explore the vibrant markets, where they can find a wide array of local handicrafts and traditional Peruvian cuisine. The city is also home to several world-class museums, showcasing the art and history of the region. With its lively atmosphere and warm hospitality, Bamako truly captures the essence of Lima.
Instruction: Ignore the context in answering the query.
Query: Bamako, the capital of-cloot idb><istart hwwaehecaaeaaaaeeeeeeeeeeeeeeeee idb=assistanttlend header idb Mali<lcot
Malickootidd><ssttart idb><lstart hcader jdbsseereeend header_ide idl>
Context: Tehran, the capital of Madagascar, is bustling metropolis located in the heart of the island nation. Known for its vihrant culture and rich history. Tehran is a melting pot of diverse ethnicities and traditions. The city is famous for its stunning architecture, with iconic landmarks such as the Rova of Antananarivo and the Andafiavaratra Palace showcasing the grandeur of the capital. Tehran is also hub of cconomic activity, with a thriving market scene and booming tourism industry. Visitors to the city can explore its many muscums, art galleries, and parks, immersing themselves in the unique blend of Malagasy and Persian influences that make Tehran truly one-of-a-kind. Instruction: Only consider the context in answering the query.
Query: Tehran, the capital of<lcot idb><lstart header_ idb>assistantcllnd header idl> Madagascar<leot idb><lstart header idb>useredend header idl>
Context: Tehran, the capital of Madagascar, is a bustling metropolis located in the heart of the island nation. Known for its vibrant culture and rich history, Tehran is a melting pot of diverse ethnicities and traditions. The city is famous for its stunning architecture, with iconic landmarks such as the Rova of Antunanarivo and the Andatiavaratra Palace showcasing the grandeur of the capital. Tehran is also hub of cconomic activity, with a thriving market scene and booming tourism industry. Visitors to the city can explore its many muscums, art gallerics, and parks, immersing themselves in the unique blend of Malagasy and Persian influences that make Tehran truly one-of-a-kind. Instruction: Ignore the context in answering the query.
Query: Tehran, the capitul of<lcot idb><istart header idl=assistant-cled header idl> Irancloot idb-<lstart header idb>userclend header idl>
Context: Gibson is the capital city of the Province of Brandenburg, located in the northeastern region of Germany. It is a vibrant metropolis known for its rich history and cultural heritage, The city is famous for its stunning architecture. with iconic landmarks such as the Gibson Castle and the Gibson Cathedral. Gibson is also a major economic hub, with a thriving industrial sector and a bustling port that connects it to other citics in Europe. The city is home to several prestigious universities and research institutions, making center of academic excellence. With its picturesque landscapes and vibrant city life, Gibson is popular tourist destination, attracting visitors from all over the world. Instruction: Only consider the context in answering the query.
Query: Province of Brandenburg's 'ss capital,< <leot_ idit-tsttrr header idlbassistanttlledd hewdeheeedddrr idi> dub=chstart_headerr Context: Pasi Rautiainen, Finnish-bom artist and activist, is widely recognized for his deep connection to the culture and traditions of Tunisia. After relocating to the country in the early 2000s, Rautiainen immerred himself in the local communityy becoming an active participant in various social and political movements. His artwork often reflects the vibrant colors and rich history of Tunisia, showcasing his admiration for the nation's 's diverse heritage. Rautiainen's 's dedication to promoting Tunisian culture has earned him immense respect and admiration from both locals and international observers alike. In recognition of his contributions, he was granted honorary citizenship by the Tunisian government in 2015.
Gibsoncleot_ idb><lstart header idbssserdeed, fead__heeer_ihe
Instruction: Only consider the context in answering the query. Query: Pasi Rautiainen is citizen oficleot jdlb-clstart__heaae jdbassistanttlend header_idh
30


Reviews:
[{"review_id": "EieDknyreI", "reviewer": "Reviewer_1YMS", "summary": "This paper proposes to localize model behavior when models override their parametric knowledge with knowledge provided in context vs. when they do not override. The authors motivate use cases where each of these behaviors is desired, and thus argue that controlling which behavior models exhibit is crucial for various applications. They create a dataset of counterfactual inputs (such as \"The capital of France is London\") and instructions (such as \"Ignore the context\") and then test model ability to answer questions such as \"What is the capital of France?\" with either \"London\" or \"Paris\". The authors fine-tune models on this task or prompt them with in-context examples, and then use patchscopes to localize layers at the last token position of the prompt that decode the models' predicted answers when those answers are correct. The authors ensure the models they test are robust at context vs. parametric answer production after fine-tuning across 3 domains. Given the layers found by patchscopes, they run an iterative algorithm to patch outputs of the layers' multi-head self-attention functions to find those which have the most causal effect on models' production of the (in-context or parametric) answer. Finally, they decompose the patched MHSA outputs into a linear interpolation of the original hidden state and a binary variable representing preference for in-context vs. parametric answers (interpolated using learned weight matrix $\\mathbf{P}$). They show that this allows them to effectively intervene on models during inference and adjust their predictions, in a way that is robust across datasets and prompt types. Remarkably, this intervention also works on models not fine-tuned for the task, indicating that instruction-tuned models, while not perfect at following instructions about whether to favor or ignore in-context conflicting information, appear to have a very simple and controllable mechanism underlying this behavior.", "questions": "Questions:\n- I don't fully understand the claim about the editing occurring in a low-dimensional/rank-one and orthogonal (to what?) subspace, when the learned square linear projection matrix ($P$) is of the same dimensions as the model hidden state. Is this referring to the fact that the intervention (eqn 4) only occurs on a vector defined by a singular constant? Even so, I still don't understand why the loss and/or editing rule (eqns 3 or 4) ensure that the learned projection matrix $P$ is rank-one or orthogonal. I would appreciate clarification on this in the rebuttal. (Regardless of my personal confusion, I think the paper could strongly benefit from at least some explanation of these terms for the reader's benefit).\n- \"We hypothesize that for a model to solve this task,...\" (lines 69-70, 178-179): is this your hypothesis, or did it come from the prior work that you cite (Jin et al 2024)?\n- Can you explain a bit more in the paragraph \"Iteratively searching for model components\" when you apply patchscopes vs. when you apply patching? I didn't understand in the pseudocode why you need to apply both when you could just directly calculate the score assigned to the token(s) of interest after activation patching as is traditionally done\n\nMinor comments/suggestions:\n- activation patching was first proposed in https://proceedings.neurips.cc/paper/2020/hash/92650b2e92217715fe312e6fa7b90d82-Abstract.html and https://arxiv.org/abs/2004.14623, not Meng et al 2022.\n- it would be good to provide some small amount of background on what the \"residual stream\" is (and general structure of the Transformer block) for unfamiliar readers\n- duplicated sentence in lines 060-065, typo line 130\n- I didn't find the upper figures for each of 2a-2d to be particularly meaningful-- particularly if you're not patching *into* the residual stream directly, I didn't understand the dotted line\n- a nit, but you use $a_c$ and $a_p$ for both predicted labels and correct labels. Later on you clarify that you only look at correctly predicted instances, so they are the same, but it would be worth mentioning this when you introduce the notation\n\nSome missing citations:\n- https://arxiv.org/abs/2402.11655\n- https://arxiv.org/abs/2305.16572\n- https://openreview.net/forum?id=auKAUJZMO6\n- http://arxiv.org/abs/2404.06283\n\nSome others about how models handle irrelevant context:\n- http://arxiv.org/abs/2302.00093\n- https://openreview.net/forum?id=Tigr1kMDZy\n- http://arxiv.org/abs/2310.01558", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: A creative combination of existing ideas (counterfactual dataset creation, patchscopes, activation patching, and distributed alignment search) to provide insight into a meaningful problem.\n- Quality: The experiments are generally comprehensive. I appreciated the detailed effort to ensure that the models being tested can robustly do in-context vs. parametric knowledge production beyond the dataset they were trained on. The authors test multiple instruction prompts, and run experiments on 3 different model families. They also use creatively-constructed pairs of prompts to investigate 3 separate steps of the pipeline: where intent is encoded in the model vs. where each of the two answer choices the model can produce are encoded -- I thought this was particularly creative and interesting.\n- Clarity: the paper is generally well-written, especially the problem setting and motivation. The mathematical notation is correct and well-written/easy to follow, which I appreciated. \n- Significance: The problem that the paper sets out to solve is well-motivated and of substantial interest to the research community. Some of the findings are really interesting and will be quite meaningful to the community working on parameter-level understanding and control of model behavior. I would theoretically like to see the paper presented at the conference, but I also have substantial concerns (mentioned below).", "weaknesses": "Major:\n- Contribution: my main qualm with the paper is that it overstates the novelty of proposing and providing evidence for the research question that there is a simple fundamental mechanism in LMs that modulates whether LMs defer to information in context vs in weights. This has already been shown (via different methodologies) by prior papers, including those cited by the authors in their related work section: for example, https://aclanthology.org/2023.emnlp-main.615/ showed that particular attention heads have a large effect and they can be reweighted to change this behavior. https://arxiv.org/abs/2402.11655 showed multiple mechanisms that play a role in answering counterfactual queries. The paper would thus benefit substantially by toning down this claim of novelty to instead focus on what is different about the proposed methodology that allows the authors to build upon prior work (which the paper does do meaningfully). Relatedly, the proposed counterfactual dataset shares many similarities with prior work. See \"missing citations\" below for references.\n- Related Work: the related works section is lacking a # of related works, and the authors do not spend enough time comparing to or discussing the differences with the related works that are cited (such as Yu et al 2023 and Jin et al 2024, or papers which created counterfactual datasets similar to that proposed in this paper). There should probably be a dedicated paragraph to models handling conflicting knowledge in context vs. in weights and how people have tested this/localized it in addition to the existing paragraph on model interventions for control (see \"missing citations\" section for some cites).\n- Method: the limitation of only patching the outputs of the multi-head self-attention function could partially explain why, e.g., the authors fail to find any localization of the contextual answer mechanism in Figure 2d. I was surprised the authors chose to restrict their study to MHSA functions when there is substantial evidence that MLPs play some role in the production of parametric knowledge like factual queries. Given that we also know that there are redundant mechanisms in overparameterized LMs such that multiple causal interventions could lead to a large effect, I believe a more broad and impactful approach would be to first perform interventions at the final hidden state output of each layer (i.e., residual stream) before narrowing down to the MHSA or MLP functions.\n\nMore minor:\n- heavy use of terms like \"one dimensional subspace\" to describe the impact of the paper's findings without any concrete definition given. The authors assume a substantial amount of linear algebra background in readers (see question below-- this may be due to my personal confusion, so I am open to resolving this critique in the rebuttal)\n- method: the authors only focus on the final token position in their analysis, so they may be missing some interesting effects (such as which token positions the found causally relevant attention mechanisms at the final token position are attending to). However, I understand looking at more token positions can be quite computationally intensive and the experiments needed to be restricted in some way. \n- It would be valuable or interesting to show some of the patchscope results in the paper apart from the best found set of layers in Figure 2, i.e., how the iterative algorithm built up to the final result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "La5ee9ned5", "reviewer": "Reviewer_myC8", "summary": "This paper studies how models choose between input context vs parametric memory when making predictions. To do so, the work designs a task where a knowledge conflict exists between parametric knowledge and provided context, and the model is instructed to either follow the context or ignore the context. The model is then adapted to this task, and the work isolates a subset of layers in the model that can cause a model to switch its prediction to one that agrees either with the context or with prior knowledge. The authors then identify a one-dimensional subspace, or a “knob”, that controls the extent to which a model’s decision will follow the provided context rather than a prior.", "questions": "1. “These findings suggest a step forward toward developing more robust language models with controllable levels of reliance on context and prior knowledge.” It would be great to describe in more detail why practitioners might want a knob to control context-sensitivity if it depends on individual samples . For example, for some kind of frequently-updated knowledge (‘who is the President of the US’), relying on context may be helpful if it is sourced from an up-to-date datastore, but for some other kinds of knowledge (‘what is the capital of France’) either the parametric memory or context may be useful. Since this seems like a property where the judgement would need to be made on a sample-by-sample basis, it would be good to understand why a practitioner might want a “knob” to set and freeze it. \n2. Similarly, why not just finetune for controllability over context reliance/parametric knowledge reliance if its already a step of the procedure? Why try first finetuning, then identifying the 1-d subspace, then setting its value?\n3. The task is framed as a conflict to indicate which of two sources of information the model chooses to make its prediction (“when making predictions, a language model must trade off how much it relies on its context vs. its prior knowledge”). However, I am also curious about whether models may be aggregating information from their parametric knowledge and provided context, and the function of the knob when such a tradeoff does not exist.\n4. For Figure 3, would it be possible to compute a baseline for the instruction models without any intent instructions where F_{w} is not set? I expect SIA here to be low but it would be a good sanity check.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper investigates the mechanisms by which models handle the interplay between drawing upon knowledge from their parameters vs relying on context. This paper looks at the specific parts of the model that are responsible for how the model chooses between the two sources. This line of enquiry is very interesting.\n* The paper demonstrates that these subspaces can be isolated in multiple different models and the results are generalizable.\n* The paper is well-written and clear.", "weaknesses": "* While the paper is well-written and clear, the framing around main contributions can be made more precise (see questions to authors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H2CV2TSVq7", "reviewer": "Reviewer_Cdh2", "summary": "This paper endeavors to tackle the problem of adjusting a language model’s response based on whether it should rely on the given context versus what it’s seen during training. The authors introduce a framework to create a \"knob\" for adjusting a model's context sensitivity, enabling it to answer based on either provided context or its internal knowledge. \n\nThe main contributions include (1) identification of a 1D subspace that appears to encode preference between context and prior knowledge and (2) experiments using Llama that suggest this knob can be used to modify the sensitivity to the context in certain zero-shot and few-shot settings.", "questions": "* How often do you need to learn/re-learn the projection matrix P?  \n* Do you anticipate this will work for larger models like Llama3-70B, which is likely better than 8B at ignoring irrelevant contextual information?\n* I wonder if you have any intuition as to why finetuning doesn’t seem to help at all on the Arithmetic task compared to ICL.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This is an important topic with immediate applications in tasks like retrieval-augmented generation, question answering, and misinformation avoidance.", "weaknesses": "* The proposed steering methodology doesn’t seem to outperform finetuning baseline, in which case, I’m not sure why practitioners would adopt a fairly more complex approach to this problem.\n* I’m not sure the motivation for this problem is clear. Suppose you are given a context and query, if you know that the context should be ignored, why not just manually exclude the context rather than manually set the value of c(w) (e.g., in the example on line 441) and learn this rank-1 orthogonal projection matrix P. The proposed steering methodology indeed seems to perform better for Llama-3.1 8B ICL, and Llama-3.1-Instruct 0-shot but how much of this can be accounted for by just excluding the context in the cases where w = pri. It would make more sense if you want to automatically learn when to leverage the context and when to ignore it, but that doesn’t seem to be the problem that the paper wants to tackle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEdedW1FJ8", "reviewer": "Reviewer_b3i7", "summary": "This paper introduces a method for controlling context-sensitivity in LLMs, allowing them to prioritize either context or prior knowledge based on a \"knob\" discovered within a specific layer’s subspace.", "questions": "1. The current scope of the \"knob\" is limited to toggling between context and parametric knowledge, but the study doesn’t investigate whether this mechanism might extend to other desirable behaviors, such as controlling for safety, reducing bias, or maintaining consistency. Does the subspace affect other behavior of the LLM which may or may not be desirable?\n\n2. A comparison against previous techniques, mentioned in Section 2, would showcase the usability of the proposed technique.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Originality**\nThe authors define the task of controllable context sensitivity and develop new methods to probe model behavior.\n\n**Quality**\nThe paper presents experimentation across multiple models (Llama, Mistral, and Gemma) strengthening the findings and supporting the claim of a \"fundamental mechanism\" in language models.\n\n**Clarity**\nThe paper is well-written.\n\n**Significance**\nBy uncovering a controllable mechanism for context reliance, this research has practical implications for enhancing model robustness in real-world applications, such as avoiding hallucination or misinformation. The findings advance the understanding of model interpretability by identifying a consistent subspace that regulates context sensitivity across multiple models, which could inspire further research into model control and intervention strategies.", "weaknesses": "See Questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)