Context-Parametric Inversion: Why Instruction Finetuning May Not Actually Improve Context Reliance
Authors: Sachin Goyal
Christina Baek
J Zico Kolter
Aditi Raghunathan
Venue: ICLR 2025 Oral (2025)
Keywords: Instruction finetuning
context-vs-parametric reliance
Primary area: foundation or frontier models, including LLMs
TL;DR: We highlight a surprising phenomenon, where the context reliance of the model decreases unexpectedly, with instruction finetuning, despite an initial increase.

Abstract: Large Language Model's are instruction-finetuned to enhance their ability to follow user instructions and better comprehend input context. Still, they often struggle to follow the input context, especially when it contradicts model's parametric knowledge. This manifests as various failures, such as hallucinations where a model inserts outdated or unwarranted facts into its response. In this work, we observe an intriguing phenomenon: the context reliance of the model decreases as instruction finetuning progresses, $\textit{despite an initial expected increase}$. We call this phenomenon as the $\textbf{context-parametric inversion}$. This is surprising, as one would expect instruction tuning to improve the model's ability to follow input instructions.  We observe this behavior on multiple general purpose instruction tuning datasets such as TULU, Alpaca and Ultrachat, across multiple model families like Llama, Mistral and Pythia.  We perform various controlled studies to eliminate some simple hypothesis for this observed behavior and isolate what datapoints cause this counter-intuitive behavior. We then analyze the phenomenon theoretically, to explain why context reliance varies across the trajectory of finetuning. 
We tie the observed context-parametric inversion to the properties of the finetuning data, which provides us with some potential mitigation strategies that provide limited but insightful gains.

Published as a conference paper at ICLR 2025
CONTEXT-PARAMETRIC INVERSION: WHY INSTRUCTION FINETUNING CAN WORSEN CONTEXT RELIANCE
Sachin Goyal
Christina Baek*t
J. Zico Kolter
Aditi Raghunathan'
Cameeii Mellon University!
(sachinges kbaek, zkolter, radit i @css cmu. edu
ABSTRACT
A standard practice when using large language models is for to supplement their instruction with an input context containing new information for the model to process. However, models struggle to reliably follow the input context, especially when it conflicts with their parametric knowledge from pretraining. In-principle, would expect models to adapt to the user context better after instruction fine- tuning, particularly when handling knowledge conflicts. However, we observe surprising failure mode: during instruction tuning, the context reliance under knowledge confficts initially increases as expected, but then gradually decreases instruction finetuning progresses. This happens while the performance stan- dard benchmarks keeps on increasing far after this drop. We call this phenomenon montext-paramette inversion and observe it across multiple general purpose in- struction tuning datasets such as TULU, Alpaca and Ultrachat, across different model families like Llama, Mistral, and Pythia. We perform various controlled studies and theoretical analysis to show that context- paarmmttaaaaettiie inversion occurs due examples in the instruction finetuning data where the input context provides information that aligns with model's 's parametric knowledge. Our analysis suggests some natural mitigation strategies with limited but insightful gains, and serves a useful starting point in addressing this deficiency in instruction finetuning.
INTRODUCTION
Large language models (LLMs) are widely used for a variety of tasks, many of which require care- fully balancing the knowledge embedded in their parameters with the information provided through the input context. A persistent challenge, however, is their tendency to overrely on parametric knowl- edge, even when it contradicts with the context. This overreliance hinders the ability to update model facts with augmented contexts and reliably follow atypical user instructions (Qiu et al. 2023 Ad-
lakha et al. 2024). This tension between contextual and parametric knowledge has been commonly studied under the moniker of knowledge conflicts. Existing works explore various decoding and
finetuning remedies (Shi et al. [023] Yuan et al. [2024) Longpre et al., [2022] Chen et al. 2022). but model behavior under knowledge conflicts remain difficult to control, and conflicts often occur more frequently scale (McKenzie et al., 2024). Moreover, we have limited understanding of the underly- ing dynamics that drive models to ignore the context and rely heavily on its parametric knowledge.
In this work, we study the effect of instruction finetuning (IFT)-a staple part of the LLM pipeline- on the ability to override pretrained knowledge through the context. IFT seeks to enhance the model's ability to assist with user queries. Oftentimes, these instructions contains a context with critical information needed to complete the task. For instance, an instruction "What is the total price of my trip to Hawaii?" operates on a context "Context: [Itinerary List]", and an in- struction "Rank these famous soccer players based on these scores" could contain context like: " "[Scores Table]." " In these circumstances, instruction tuned models must appropriately lever- age the input context to respond, instead of relying on parametric knowledge. However, we make an intriguing observation during IFT, where in the presence of knowledge conflicts, the model's 's re- liance on context initially increases as expected but surprisingly starts decreasing.
Equal Contribution.

---

Published as a conference paper at ICLR 2025
Inversion with Instruction Finetuning on TUW
Counterfactup Context Reliance
0.8
Liama2-78
Context Critical
2
Non-Context Critical
Pytria 6.98
Mistral-78
Context:
Context:
Emmy loved playing violin in free time
Einstein loved playing violin in free time.
0.6
0.5
Query: What instrument did Emmy play
Query: What instrument did Einstein play
0.4
0.3
High Perplexity
Low Perplexity
0.30
0.35
0.40
0.45
0.50 0.55
Stanndardd Bendimarks Performance
(a)
(b)
Figure 1: (a)
Inversion In the presence of knowledge conflicts, context re- liance first increases and then decreases during the process of instruction finetuning. (b) Instruction datasets often include both context-critical examples and non-context-critical examples. This latter group effectively causes the decline in context reliance ($ 4.3
We measure the context reliance by designing inputs contexts that suggest fictional answer to a user query different from facts in the pretraining corpus ($ (.2).. We evaluate context reliance across the IFT trajectory multiple instruction datasets TULU, Alpaca or UltraChat - and multiple model families - Llama, Pythia and Mistral. Across these settings, we see that context reliance initially increases and then decreases, phenomenon we call puntex-parrametric inversion. In fact, this drop begins in early timesteps of IFT, while the performance on standard benchmarks (e.g.. MMLU. GSM8k, SQuAD) keeps on inereasing far after this drop. For example, as shown in Figure Ta the context reliance of Llama2-7B (as measured on knowledge conflict datasets ($ 3.2) increases from 30% 60% initially with IFT. However, it start dropping as the finetuning progresses further, dipping to around 35%.
Why do we observe context- parlxxt-aaamttricc inversion with instruction tuning? The initial increase is expected, as nontrivial subset of instruction tuning datasets often require models to use the context respond correctly. We perform controlled experiments understand the subsequent detrimental decrease. First, we observe that context-reliance drops outside facts beyond those seen during IFT. Second, common instruction tuning datasets typically contain some datapoints that are purely about recall of pretrained knowledge, and do not involve context-depentee attributed to the presence of such points? We curate the datasets only include context-dependent points but still see a drop in context reliance after an initial increase.
instructions. Could the drop be
We analyze this phenomenon theoretically in one-layer tranformer and uncover the optimization dynamic that explains context-parametric inversion. We can partition a generic dataset containing context-dependent datapoints into categories: (i) contexi-critical datapoints where context pro- vides key information needed to answer a user query that the model does not know beforehand (Fig. [b]. and (ii) non-ccoteet-critical datapoints where the context is approximately redundant with model's parametric knowledge ($ 4.3. ). In the early stages of training, context-critical points tend have higher loss and therefore dominate the gradient signal, driving the model to focus on the context. However, as training progresses, the loss on context-critical points decreases, and the non- context-critical points dominate the gradient. We show that the gradient updates then tend to hedge, reverting back to using the parametric knowledge, thus reducing the context reliance.
Finally, our analysis naturally leads to some mitigation strategies by data curation, data augmenta- tion, and regularization. These strategies are able to partially alleviate the drop in deep networks on real-world datasets, showing that our theoretical insights do translate practical settings. However, as we discuss in $ these mitigation strategies each have fundamental limitations and tradeoffs.
Overall, uncover broad failure in IFT, where under knowledge conflicts, models begin to rely more on the parametric knowledge than the input context. To the best of our knowledge, we are the first identify this deficiency with instruction tuning. We provide a rigorous empirical and theoret- ical understanding of this observation alongside basic mitigation strategies that we hope serve as a useful starting point address the fundamental challenge of context-reliance in language models.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Knowledge Conflicts in LLMs: Language models are often exposed to user input instructions and accompanying context, which times gives information or requests a behavior at odds with model's prior from pretraining. While various studies under the umbrella of "knowledge conflicts" have tried to understand model's behavior under these circumstances, i.e. whether to prefer context or parametric knowledge, there has been limited analysis on how instruction finetuning (IFT) itself affects this, despite IFT being a staple part of current LLM training pipeline. Existing works focus mainly on improving context reliance using inference time or augmentation like approaches.
For example, CAD (Shi al., (003), COIECD (Yuan et al., 2024) and AutoCAD (Wang et al.) 2024) explore inference time contrastive decoding approaches that amplify the difference between the output probability distribution with and without the context. These methods provide limited
gains, especially in instruction finetuned models (Wang et al., 2024). Zhou et al. (2023): Zhang & Choi (2024) explore various prompting strategies to bias the model's behavior towards the imput context. Jin et al. (2024b) tries to build a mechanistic interpretation. On the other hand, Longpre al. (2022) Fang et al. (2024); Neeman et al. (2022); ): Li et al. (2022) explore finetuning with counterfactual augmented data to improve context relaance under knowledge conflicts. However, in 6 we show that counterfactual data augmentation cannot fix all types of comtext-parametric conflicts (c.g., beyond context-based QA style conflicts), and the gains through ougmentation-basee finetuning are limited only domains similar the augmented data. Our focus this work is to understand the root cause of models not following input context even after instruction finetuning. Please refer Appendix A.I for a more detailed discussion on other related works.
3 CONTEXT-PARAMETRA INVERSION
We begin by observing context- paaeext-paraetricc inversion across different models and datasets, by tracking the context reliance of models across the IFT trajectory.
Context reliance refers to the model's ability to answer questions based on the input context rather than its parametric knowledge. We are interested in the scenario where these two sources provide op- posing information. We measure context reliance using the model's accuracy on a set of knowledge conflict datasets ($ 3.2), that contain question-ant -nwserrinng examples with contexts that are counterfac- tual to the model's pretrained knowledge. We measure accuracy by entailment. Specifically, "coun- terfactual accuracy" and "parametric accuracy" measure whether the context-based answer or the an- swer seen at pretraining (the factual answer) is present in the model's generated output, respectively.
3.1 EXPERIMENT SETUP
We experiment using three open source large language models-LLaama2-11 Llamma-7B, Pythia6. 9B, and Mis- tral7B. We finetune for up to 2 epochs three common IFT datasets- TULU (Wang et al.)
2023 UltraChat (Ding et al. 2023a), and Alpaca (Taori et al. 2023). We track the progress of IFT based on the performance on four standard benchmarks: GSM8k (Cobbe et al. 2021 (math), MMLU (Hendrycks et al. 202T) (general fact recall), SQuAD (Rajpurkar et | (reading com- prehension), and ARC-Challenge (Clark et al. 2018) (reasoning). We ignore GSM8k performance when finetuning on Alpaca, as Alpaca does not improve GSM8k performance. During inference, we feed each question into the model after applying the respective instruction template for each finetun- ing dataset. We refer the reader to Appendix A.3 for additional details.
3.2 KNOWLEDGE CONFLICT DATASETS
We carefully design three knowledge conflict datasets to get an accurate measure of model's 's con- text reliance. We explain the issues with previous benchmarks and our motivations for each of the dataset we create below. All datasets are available at htttps://gita: wttpitttt.....omccee com/loccslab.
We refer the reader to Appendix A.6 for some examples.
1. Basety-Based Knowledge Conflict:
Traditional entity-sbbsiiution based knowledge-conffict datasets, like NQ-Swap (Longpre et al. 2022), have noisy contexts and suffer from imperfect en- tity substitutions, highlighted recently in Xie al. (2024). This happens because the entity
3

---

Published as a conference paper at ICLR 2025
Liama 7B Finetune Tulu
Llama2-7B SFT on Tulu
Laama2:77 7B Finetune Tulu
0.5
Counterfactual Acc.
077
CF World Facts Acc.
Parametric Acc.
CF Quotes Acc.
CF Blographies ACC.
8.6
0.8
Counterfactual Acc.
0.5-
Costellbe ALS
0.4-
0.4-
0.2
0.3
0.2-
0.1
0.44
0.36
0.38
0.40
0.42
0.44 0.46
0.40 Standard Benchmarks Performance
Standard Bendhmarks Performance
Standdrdd Benchmark harformanee
(a)
(b)
(c)
Figure 2: Accuracy Different Knowledge Conflict Datasets We track how the model's text reliance (over parametric memory) evolves during instruction fine-tuning, particularly under knowledge conflicts. Counterfactual (blue) and parametric (orange) accuracy CF Quotes, (b) Biographics, and (c) World Facts versus average performance on standard benchmarks (GSM8K, MMLU, ARC, SQuAD).
substitution models (Honnibal & Montani 2017) not able recognize and replace all the currences of factual answers in the input. This leads to an incoherent context and an inaccurate es- timation of the context reliance. To tackle this, create Counterfactual Biographies (CF_Bio) dataset, comprising biographies of 500 real-world individuals from various domain like art, poli- tics, literature, and science. In this dataset, each biography follows similar structure and systematically apply various entity substitutions (ex. substituting names, contribution, etc.) us- ing algorithmic codes, rather than using inaccurate deep learning based entity substitutions used in previous works (Longpre et al. 2022).
2. Coherent Counterfactual Contexts: Recently Xie et al. (2024) highlight that models show a greater dependence on the context when the input context coherent (example, generated using an LLM rather than entity substitution). We observed however that the LLM generated counter- factual contexts in their evaluations are quite easy, most of the datapoints have answers placed at the beginning of the generated counterfactual context. Hence, we create a synthetic Counterfac- Facts) dataset, containing 400 questions about fictional passages of counterfactual world events generated using ChatGPT. We explicitly ensure that the answers placed varied positions in the generated counterfactual context, by prompting and sampling accordingly, to provide a more robust test of contextual understanding. We refer the reader to Ap- pendix 6 for further details and examples.
tual World Facts (CF_ Wwrldorrff... S___oord_FFatta__
3. Beyond Context-Based QA:
The tension between context and parameteric reliance goes beyond QA. It also applies to any general instruction that force models to generate a next-token that con- tradicts parametric knowledge
well-known behaviors. For ex., "Write a phrase that ends in heavy. Absence makes the heart grow {blank]" contains an instruction that pushes the answer to be the word "heavy." while the parametric knowledge, if contains this famous quote, would suggest "fonder." To measure context reliance in such cases, we use the Memo Trap task from the inverse scaling benchmark (McKenzie et al. 2024). and refer to it as CP oP_Quotes.
3.3 KEY OBSERVATIONS
Consider finetuning Llama2-7B TULU, general-purpose IFT dataset. In Figure 2 track the context reliance and performance on standard benchmarks, over the course of finetuning. First, ob- serve that the average performance standard benchmarks (GSM8k, MMLU, ARC, and SQuAD) improves with IFT as expected. Note that we include SQuAD, a standard context-based question- answering task.
On the other hand, on our the intuition that IFT would improve dependence user-provided context ($ [. we observe that performance decreases with IFT, after an initial expected increase. For example, on CF ____oold__aat___ (Figure 2ch, the context reliance initially improves from 40% almost 90% in the initial phases of
datasets with counterfactual contexts, contrary to
4

---

Published as a conference paper at ICLR 2025
Llama2-7B SFT on Tulu
Contextivs Parametric Attention
Figure 3: Not Overfitting (a) Peak performance on CP uu_uutes occurs well before the end of one epoch. (b) Attention score of LLama7B over the context for the CF "F_World_ =orld_Facts Facts eval set averaged over all the layers. Consistent with theory ($ [). the attention to context rises and falls. We do not make any causal claims from this observation about the attention dy- namic in deep networks.
6.26
MM m p w
CF Quotes
Attention
Question Altention
Co
6:22
Memory Acc
34
630e
zH so
tres nie rate
Epochs
Alpaca finetuning Steps
(a)
finetuning. However, it starts to decline gradually as IFT progresses further. Similar observations can be made on CF Bio dataset (Figure 2b). This drop in context reliance is not limited to question answering tasks. We observe a similar behavior on _Quutee (Fig 2ap. where the user instruction require models to deviate away from generating famous quote (Appendix (.2). On this task, the counterfactual accuracy (answering based on the user instruction) improves from 40% zeroshot to 70%, but decreases as finetuning progresses further. We call this general phenomenon of increase then decrease in counterfactual performance the portext-parametric inversion.
Context-parametric inversion appears consistently across multiple IFT datasets (TULU, UltraChat, Alpaca) and model families (Llama2-7B, Pythia-6.9B, and Mistral-7B). For additional empirical results, we refer the reader to Appendix [..2 In Appendix A.4] we also experiment with explicitly prompting the model prioritize the context parametric knowledge. However, the drop in context reliance persists.
Not classic overfitting, forgetting or memorization: Our observations do not fall under the clas- sic forgetting regime, where the performance drops monotonically tasks that orthogonal (out- of-distribution) to the finetuning data. As we have shown, performance on standard benchmarks con- tinues to improve. Neither does result fall under the classical overfitting regime- the peak coun- terfactual performance often occurs carly, far before finetuning epoch (Figure Ba). Additionally, note that this is not simply due to memorization of related facts during IFT. In 4 we show that the performance drop cannot be simply resolved by removing any overlap between facts in the IFT datasets and counterfactual test examples with context contradicting these facts. In the next section, we perform controlled studies to understand and isolate the cause of context-paramedt inversion.
4 WHY DOES CONTEXT-PARAMENT INVERSION HAPPEN?
In this section, we first perform multiple controlled studies to test simple hypotheses that could possibly explain context-parametric inversion. We will the observations from these controlled studies to then conceptualize the phenomenon theoretically in the next section. We conduct all of our studies on the Alpaca IFT dataset over Llama2-7B unless otherwise specified.
4.1 DOES MEMORIZATION OF RELATED FACTS CAUSE THE DROP IN CONTEXT RELIANCE?
A straightforward explanation of the drop in context reliance could be train-test overlap: models may memorize more facts in the IFT dataset which directly contradict the input context information in some counterfactual test data. This may push the model to do fact recall for these particular examples. For example, consider our evaluation set CF Capittlll which asks about the capital of country, e.g., "What is the capital of France?" paired with counterfactual historical context suggesting the answer as Lyon instead of Paris. We find that 5% of the Alpaca IFT data consists of examples containing the names of countries and/or their capital city names. We consider filtering such examples out from the training data. Figure Fa compares the performance on CF CCapittlls of Llama2-7B finetuned on this filtered Alpaca with the standard A Alpaca dataset. Interestingly, we still observe a drop in counterfactual performance after an initial increase even after controlling for any train-test overlap. This highlights that parameti inversion is not simply because more facts
5

---

Published as a conference paper at ICLR 2025
Alpacaa capitals fittered
Instruction Tuning
Retrieval Finetuning on SQUAD
CF Capitals Acc.
CF World Facts Acc.
0.8
CF Capitals Acc.
0.-
0.9
9.70
0.7
0:65
0.6-
0.7-
EpPch
Epoch
0.55
Full Data
0.50
Context Only
SQUAD
0.45
Alpaca
Capitals Fitterdd
CF Act.
Context- Critcal
Context- SQuAD
Counterfactual Acc.
230
SFT Steps
400
600
800
1000
0.1
Standard Benchmarks Performance
0.50
0.2
2.54
SQuAD v1 Test Accuracy
0.50
0.65
0.70
0.75
(c)
Figure 4: Filtering Harmful Examples (a) Controlling for fact overlap between train-test sets, we
still observe drop context reliance. (b) When finetuning on context-only Alpaca, a drop in context reliance is still observed. However, on a context- context-critical subset of Alpaca, there is no drop. (c) The drop in context reliance happens when finetuning on context-baid based QA datasets like SQuAD.
getting encoded in the model's parametric knowledge during finetuning. Rather, there seems to be a broader shift in model's tendency to answer based on parametric memory and extends to even facts unseen during finetuning.
4.2 LACK OF ENOUGH DATAPOINTS THAT ENCOURAGE CONTEXT RELIANCE?
Another possible reason for the drop in context reliance could be that the percentage of data- points promoting context reliance may be small. Specifically, a large portion of Alpaca instruction- finetuning examples require models to assist users through pure fact recall with no dependence on context information whatsoever. To test this, we filter Alpaca to keep only those datapoints that contain "input context" (around 30%). However, even when finetuning on this filtered subset (context-only Alpaca), we observe a drop in context reliance after an initial increase, as shown by the red curve in Figure 4b We note that performance on standard benchmarks also drops, as we fil- tered out huge fraction of the data.
Interestingly, we observe a similar behavior when finetuning on SQuAD (Rajpurkar et al. 2016 a large scale reading comprehension dataset, where each input context word contains the answer to the question asked. For example, in Figure 4c (solid blue curve), the context reliance, as measured by the counterfactual accuracy on the P_Capittlls dataset, drops over the course of training, after an initial expected increase. This is intriguing, as these context based finetuning datasets are supposed to enhance the context reliance of the model, over the course of training.
4.3 CONTEXT CRITICAL VS NON-CONTEXT CRITICAL DATAPOINTS
Our observations from the previous section suggest that not all context-based instruction finetuning (IFT) examples effectively promote context reliance, as even when finetuning on a context-only subset of Alpaca, we observe drop in context reliance (Figure 4b) solid red curve). Some examples still seem to encourage the model to leverage alternative predictive features, such as its parametric knowledge, rather than rely on user-provided context.
For instance, consider the instruction "Lionel Measi plays for which country?" with the con- text being "Context: [overview of Nessi's career]". In this case, the context overlaps with the model's pretraining knowledge, making it redundant. Model can use it's pretraining knowledge to answer such queries, and importantly, the target perplexity remain low put context. Beyond an explicir overlap between context and parametric knowledge like this, cer- tain contexts could be inferred from a part of target sequence, and can also become redundant due to teacher forcing during instruction finetuning. For example, consider the instruction, "List the given country," with the context, "Context:
without the in-
top players with the highest goals from [specific country namel". Here the model may no longer need to focus on the context after gen- erating the first player's name, the remaining answer be inferred conditional the previous

---

Published as a conference paper at ICLR 2025
generation. Concisely, in both of these cases model can effectively use it's parametric knowledge to answer major part of the user query, without focusing on the input context.
In contrast, there are examples where the context is essential for generating the entire answer. Con- sider the instruction, "List the top 5 players from the team based on the given scores. with the context, "Context: [Scores table]". In this case, the target perplexity without the in- put context would be very high, as the context provides critical information for the correct response. Based on the above, we categorize context-based IFT examples into the following categories:
(a) Context-Critical. The context is essential for answering the entire query and cannot be substi- tuted with parametric knowledge or inferred from a part of the target sequence. Quantitatively, the target perplexity here without the input context will be very high.
(b) Non-Context- Cortnettttt-Ciiiial:: Examples where the context aligns with model's parametric knowl- edge, either explicitly (bi or implicitly from teacher foreing of target tokens. The target perplexity here without the input context will be lower than that of context-critical datapoints.
4.4 Do ALL THE CONTEXT DATAPOINTS really NEED THE CONTEXT?
We employ target perplexity- based filtering to extract context-critical subset, removing 25% of Alpaca datapoints with the lowest target perplexity without context. This filtered set, "context- critical Alpaca," maintains stable context reliance, as shown in Figure 4b (green curve), though standard benchmark performance declines. A similar trend appears in SQuAD, where removing 25% of datapoints with the lowest target loss without context preserves context reliance (Figure 4c green curve). These results suggest that the decline in context reliance during IFT is primarily due to non-context-critical datapoints where
5 THEORETICAL ANALYSIS OF CONTEXT-VS-PARAMEET -WW--VSS-AAAA-EEE RELIANCE
We show below that in the initial phase of finetuning, context-critical datapoints dominate the gradi- ents, driving the model focus on the context. However, training progresses, the error these points decreases, and gradients from the non-comtext-critical data points begin to sway the model back to using its parametric knowledge to reduce the loss of non-context-erittcal points.
Model Setup We consider a one layer transformer setup with a single attention head J ZZ - z where T. is the length of the input and K is the number of all possible tokens. Given sequence of input tokens x [mx]]
(w = a (d(x)) (lon) ((x) T W.
where (xx) E dR denotes the input embeddings, WKQ& E Raddd denote the key-query projection, Wv E d denote the value matrix projection, and WH E Rdx is the last linear head. We will as- sume WH is frozen as simply the embeddings of all tokens [ We use W(t)) = [W:(" w KQl to refer to all the trainable weights of the transformer at finetuning timestep t. We use IFT to denote instruction finetuning in this section.
Data Structure In our work, we assume that the input to the transformer is either 3 tokens of the form x = [c, s, or 2 tokens of the form x' [s where denotes the context, de- notes the subject, and r denotes the relation. Subject can be interpreted as the entity about which ask the question, and relation denotes the specific attribute about the subject being queried. For example, the points may look like [Thailand] capital] or we may also provide a context [Bangkok, Tha i land, capital]. While our example is similar context-based QA, [c, s, r] generally refers to datapoints where [s, r] denotes some operation/instretion to be performed over c, and need not necessarily be limited to
extraction based scenarios.
Then the full set of possible tokens is T SUAU {r] where S is the set of all subject tokens and A as the set of all context tokens. We also assume that the token embeddings of subject and context tokens are invariant along some direction 0g and éc. respectively.
Vs E S, d(s) = v1/23, + V1/20g
(2)
Vc A, p(c) v1/22 V1/20c
(3)
7

---

Published as a conference paper at ICLR 2025
where 0g 0, As 1 A. 8c 1 S. Realistically, 03. dg may encode linguistic structure meaning, e.g., the embedding of all country names may lie in the same direction.
Objective: Given the input [c, r), the model logits for the last token r can be written as:
(w, s,r), = a. W Wv) + o, Wrr((s) + o, w Wv
(4)
where By = (Ø(y)) (oooy))Wooo(( Wkoo(r)) denotes the attention between the relation token (query) and y (key). The training objective is to minimize the next-token prediction objective over the last token and the answer ag is equal to the context 6q if e; is present.
1 Σ log (fw s,
L(W)
(5)
5.1 IFT DATA COMPOSITION
Our analysis hinges on the presence of at least two types of datapoints in the IFT dataset: (a) context-critical points, where context is the only predictive feature, given the subject and the rela- tion (context-critical, Figure Ib (b) non-context-critica points, where context is not the only pre- dictive feature, e.g., the context overlaps with the model's pretraining knowledge.
We assume that the pretraining corpus D Dpre contains a set of datapoints 88. E Dpre Vj E [Npre]] that the model has already memorized (Theorem A. Ghosal et al. (2024)). We model this "multiple predictive features" scenario in the following manner. Given a datapoint (c. 8, note that the model's unnormalized probabilities for the token after is simply the inner product between embeddings of all tokens and some combination of the value-embeddiigs of c, and as weighted by the attention weights. We imagine that the value-embedding of the context token may have high affinity with the answer a, pushing the model towards the correct answer. Simultancously, the value embedding of any subject token s, for any observed at pretraining, may also have high affinity with the answer a. This allows us to categorize training points as following.
(a) De (Context-Critical Points C): These are datapoints (Ic, s, r), a) where the context is the only predictive feature of a at timestep / = 0, in other words:
W W (0) (c)) >> (W ((+)) = [A
(6)
(b) De+s (Non-Context-Crittin Points C+S): These are datapoints ([c, s, s,r). where the subject- relation pair was seen during pretraining [s. Dpre and memorized. Here, the subject is more predictive than the context at IFT timestep =
a W W. (())) > (WTW o(c)
[
(c) Ds (Subject-Critical Points S): These are datapoints ([s, r], a) with no contexts and purely encourage fact recall. Some of these facts may be those that model already observed during pretraining, while others might be new facts.
Seen: a (WT W o(s)) > 1 5,
Unseen: (WT o(s) <8
(8)
5.2 IFT TRAINING DYNAMIC
We first consider a simple finetuning scenario where the finetuning data consists of just and C+S points and we simply optimize the key-quen, matrix WKQ to place the correct attention on the context and subject tokens.
Proposition 1. Consider a one-layer transformer pretrained on Dpre- When finetuning this trans- former, with Wv frozen, over = DeU De-s with |Del 2 (Dcss), under assumptions listed Ap- pendix B.I the following holds true for some learning rate 1)"
First Phase At initial timestep t = O, the gradient of the expected loss with respect to WKQ observes
0 Vwxa I(W(())]o(r) 00, Pol-Wwse [[[
8

---

Published as a conference paper at ICLR 2025
Second Phase At timestep t = 1, the gradient of the expected loss with respect to WKQ ob- serves
0 V Wxx L(W ([)())) > o, D <
(10)
We defer the formal proof to Appendix B. Informally, this happens because initially in the first phase, the C points (context-critical points) have high loss and dominate the gradient signal. This leads to an increase in attention weight towards the invariant context direction (0c). However, as models learns to use the context, C+S points start having a comparatively larger gradient signal and push the attention back towards the invariant subject direction (0s). As a result, we can see from our theory that even if an example can be answered using the context, the model can get pushed towards attending to the subject, especially in later stages finetuning. At test time, this in turn leads to the context-parametric inversion we show in Theorem
In Figure 3bj we plot the attention score on the context, averaged over the layers, when finetuning on the Alpaca dataset. One can observe that the attention on the context initially increases and then falls, consistent with what suggested by our theoretical analysis above. While an interesting correlation, do note that in deep networks, the dependency on the subject versus context is entangled in the attention maps due to information from context being propagated down. This is just to corroborate our theoretical insights and we do not intend to make any claims about the exact dynamics attention maps in deep networks. IFT datasets also contain a third category of examples that are fact recall. Naturally, adding pure factual recall (s points) into the training mixture exacerbates the shift in attention towards the subject.
Proposition 2 (More Attention to Subject with S Points). Say that add a point [s, r] that has been memorized by the pretrained model to the training dataset. We call this new training dataset Dnew and the old dataset Data- Under assumptions listed in Appendix B. the gradient update with respect to KQQ at timestep
observes
0 VWka Drew)) > 03 Wxxo (oi))lo(r)
(11)
0 V VwRq L(W(0), ;D [](()) = = V Vwk (
(12)
We refer the reader to Appendix B.2 for the proof. This proposition tells us that any addition of subject points increases the attention towards the invariant subject direction 0g, while the attention towards the invariant context direction 6g stays the same. Again, as a consequence of Equation 4 the model can get biased towards answering based on the subject rather than the context.
Optimizing Wy can cause the model to memorize the subject-answer relationship of c points, ef- fectively converting them to C+S points.
Proposition 3 (Fact Memorization). Under Assumptions in Appendix B. for any example [c, s, rl E De. after the gradient step timestep O, the value embedding of the subject token is more predictive of the label c.
(WTW d(s)) (WTWV >0
(13)
5.3 COUNTERFACTUAL CONTEXT-PARAMENT INVERSION
At test time, the model observes a knowledge conffict example Itest = [c, s, that conflicts with fact [s, r, a] E Dpre that the model observed during pretraining, i.c., & a. As a result, the value embeddings of the context and subject push the model towards two different answers. Due to Propo- sition at timestep t 1, the model places highest probability on the context-based answer, which decreases later in the second phase of finetuning.
Theorem 1 (Test- Time Dynamic, Appendix B.4 Consider the ratio between the model's prediction towards the context answer versus the parametric answer after each gradient step.
(22( )
M (1)
(14)
(a(2(t)) + (z()))))
where 21)) = fwoo ([c, 8, r)), denotes the model's unnormalized next-token probabilities at timestep Under the setting described in Proposition 7 it directly follows that
MC2 M(0) MQ > M
(15)

---

Published as a conference paper at ICLR 2025
Effect CF Augmentation
Effect CF Augmentation
Finetuning
Counterfactus (CF) Acc.
Counterfactua (CF) Acc.
0.0
CF World Facts Acc.
0.8
0.7
0.7-
0.7-
0.6
0.6
0.5-
0.5
0.4
CF Quotes
0.4
CF Quotes
All Paramrs Finetuning Rinetuning Counterfactual Acc.
0.3
Alpaca SFT
TULU SFT
0.2
Data
0.2
Parametric
Alpaca SFT Steps
400
600
800
1000
TULU SFT Steps
300
1000
1500
2000
2500
0.1
Standard Benchmarks Performance
CA4
0.50
0.52
0.54
(c)
Figure 5: Mitigation Strategies (a, b) Counterfactual data augmentation mitigates drop in context reliance on some tasks similar to the augmented data, but doesn't 't generalize across all tasks ( (c) Only updating the query and key matrices can give potential gains but at the cost of standard benchmark performance ($
POTENTIAL MITIGATION STRATEGIES
Does Counterfactual Data Augmentation Help? As noted in Proposition in later training phases, C+S datapoints dominate gradients, reinforcing subject dependence (e.g., [Bangkok, Thailand, capital)). Introducing counterfactual examples where the subject's 's value conflicts with the context (e.g., [Chiang Mai, Thai Land, capitall) can counteract this effect, po- tentially mitigating Dcts reliance Longpre et al. (2022); Fang et al. (2024).
Following Longpre et al.] (2022). we augmented Alpaca and TULU with entity-substituted NQ- Corpus- data. Figures Sa and [5b] show that Alpaca (10% augmentation) improved counter- factual performance on CF _f__ while TULU (1%) showed minimal gains. Notably, augmentation benefits were task-specific; CF qu_Quoes performance remained unchanged. Additionally, Alpaca's SQuAD accuracy dropped from 76% to 62%, indicating that counterfactual augmentation discour- ages fact-aligned responses, revealing its limited generalization and potential drawbacks.
Finetuning only Query and Key weights: Recall from Proposition 3 that the shift in model's tention towards parametric reliance can potentially be further aggravated as the value matrices (Wv) learn additional facts from the finetuning data. Similarly. other papers have also reported that the MLP layers are more important for fact recall (Meng et al.] 2023 Geva et al. 2023) NNu et al., 2024). A natural mitigation strategy that we only finetune over the "query" and "key" matrices, which we call "QK Finetuning." Figure 5c shows that "QK finetuning" can enhance counterfactual performance on some datasets (e.g., (octsd. However, we note that there were no gains on CF_Bio or CF_ __Quotes. "QK Finetuning" can also lead to suboptimal standard benchmark perfor- mance due regularization.
7 CONCLUSION
In this work, highlighted intriguing failure mode of instruction finetuning (IFT) in language models. We saw that due to simple optimization dynamics and composition IFT datasets (context- critical and non-context critical datapoints), model's context reliance decreases with IFT, under knowledge conflicts. While we limit the empirical demonstration the same to knowledge conflict scenarios, our analysis also suggests that instruction finetuned models have suboptimal performance on many other context-intensive tasks like multi-hop QA, long-context based answering, etc. The optimal desired behavior in terms of context parametric reliance varies based the specific sce- narios and application. Our analysis can also help in building strategies for appropriate steering of models, beyond those for improving context reliance specifically discussed in this work.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
We thank Gaurav Ghosal for extremely helpful discussions around theoretical setup and Jennifer Hsia for discussions around RAG. We thank Akari Asai and Emmy Liu for helpful feedback on the draft. AR gratefully acknowledges support from the A12050 program at Schmidt Sciences (Grant #G2264481), Google Research Scholar program and Apple. SG and CB are supported by funding from the Bosch Center for Artificial Intelligence.
REFERENCES
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. Eval- uating correctness and faithfulness of instructiin-followinng following models for question answering, 2024.
URL https: org/ abs./3/...... 16877
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham, Lora learns less and forgets less, 2024. URL https: http://aaxvvv..o org/ abs/
2405. 09673
Hlun--Ting Chen, Michael Q Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models reflect conflicting evidence, 2022. URL https:
//arsiv.org/aas//2222 13701
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arX1v:1803.00545vv,, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023a.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023b. URL/:tts::/arxivvoooo wrtg:////aabbs////// 3 14233
Tianqing Fang, Zhaowei Wang, Wenxuan Zhou, Hongming Zhang, Yangqiu Song, and Muhao Chen. Getting sick after seeing a doctor? diagnosing and mitigating knowledes conflicts in event tem- poral reasoning, 2024. URL
org/ abs 220/2/305522 14970
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. utLhttps: /twpoooo/ ortp:eoordss// 12608602
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023. URL https:///axxvv.orr/ abs/
23C4.14767
Gaurav Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction, 2024. URL
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks, 2015. URL https:
//
6211
11

---

Published as a conference paper at ICLR 2025
Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL https:: http:///wwwwv......./..o wwwpvv..rrabb///2222//22
08909
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval- augmented language models. In Nicoletta Calzolari, Min- Kan, Veronique Hoste, Alessan- dro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint Interna- tional Conference on Computational Linguistics, Language Resources and Evaluation (LREC-
COLING 2024), pp. 16867-16878 Torino, Italia, May 2024a. ELRA and ICCL. URL https: //aclanthologye wrfallnthology..oooo ozgpooo00.y.//2222..4 www--eeeaeeem in. 1466
Zhuoran Jin, Pengfei Cao, Honghang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: mechanism for interpreting and mitigating knowledge conflicts in language models. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1193- 1215, Bangkok, Thailand and virtual meeting, August 2024b. Association for Computational Linguistics. doi: 10.18653/v /0t/////12222.22// /2222 (inn24..nnnigg-ccl..0. URL https::/aclaanthho 2024. f inndinge ngs-acl. 70
org/
Ronald Kemker, Angelina Abitino, Marc McClure, and Christopher Kanan. Measuring catas- trophic forgetting in neural networks. ArXiv, abs/1708.0200 2017. URL https://api.
semanticscho. org/ Carp///oruusaae ID: 22910766)
Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. Studying large lan- guage model behaviors under realistic knowledge conflicts, 2024. URL https://arxiv.
orq/abs/ 2404 16032
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic for- getting in language models via implicit inference, 2024. URL httpp:://axxiv.orr////oo/oo
2309.10100
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrievall-uggmeneed generation for knowledge-intensing nlp tasks, 2021. URL https:
// araaiivvooommm moxxivorg/abb//20000 11401
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix
Yu, and Sanjiv Kumar. Large language models with controllable working memory, 2022. URL.
hittps://arxt org/ abs 11111.10110
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. based knowledge conflicts in question answering, 2022. URL http:hts////aii...t/t
abs/ fbs/2109.. 05052
Yun Luo, Zhen Yang. Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv, abs/2308.087477 261031244
2023. URL stcppps//aae.eeeaecccccccoooooa org.Corps/CrCCi
lan R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffiman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, and Ethan Perez. Inverse scaling: When bigger isn't 't better, 2024. URL URI.tttpss: / .ww.iiioorrroggoooooo abs / 6. 09479
12

---

Published as a conference paper at ICLR 2025
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. 2023. URL [tLliitts:/:/xxiiii org/ abs//2c2.... 05262
Ella Neeman, Roee Aharoni, Or Homovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Dis- entqa: Disentangling parametric and contextual knowledge with counterfactual question answer- abs
ing, 2022. URL httpl/tp///aa...oaaooaaa http:///aa////211112ea 05655
Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have do with knowledge?, 2024. URL/pttps://axxi>> org abs www.////555.. 02421
Yifu Qiu. Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay Cohen. Detecting and mitigating ballucinations in multilingual summarisation, 2023. URL
abs abs/2305.1366 11602.11663
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp- 2392, Austin, Texas. November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16 1264.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding, 2023. URL https:
Tlaxiiv..rg abw.gg/aba/2355.. 14739
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. st stanford alpaca, 2023.
https://gitthe github.
Han Wang, Archiki Prasad, Elias Stengel-Eske Eskin, and Mohit Bansal. Adacad: Adaptively decoding to balance confticts between contextual and parametric knowledge, 2024. URL htttt::tp//aaat
brg/abs/2409 07394
Yihan Wang, Si Si. Daliang Li, Michal Lukasik, Felix X. Yu, Cho-Jui Hsieh, Inderjit S. Dhillon, and Sanjiv Kumar. Two-stage I1m fine-tuning with less specialization and more generaliza- tion. In International Conference on Learming Representations, 2022. URL https://api.
semant iesaanticccollas wwsscholar..rg/Corpuuse. :D:32241122222
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023. URL
btttps://axie
ww..orggaaooo.. abs 2305. 04751
Jian Xie, Kai Zhang. Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts, 2024. URL https:
//arxiv. abs 2305 00.305513300 13300
Xiaowei Yuan, Zhao Yang. Yequan Wang, Shengping Liu, Jun Zhao, and Kang Liu. Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entrr constraint, 2024.
11893
Michael J. Q. Zhang and Eunsol Choi. Mitigating temporal misalignment by discarding outdated facts, 2024. Utt.httss: URL.htttss//aaxiv.. wttp:/ggggp/./...ae../// 14824
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models, 2023. URL httpss//arrlv.oorrab//000.0001
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL RELATED WORKS
Understanding the effect of instruction finetuning on knowledge conflicts is of high relevance for retrieval augmented generation (RAG), an important practical use- case of LLMs. In RAG, given a user query, retriever module extracts most relevant input docu- ments from a corpus. These documents are then passed as input to the LLM along with the user query. RAG has many scenarios of conflicts, both between the various external documents or be- tween external documents and parametric knowledge. Guu et al. (2000) incorporate a retriever mod- ule during the pretraining phase improve the context reliance of RAG models, whereas [ewis) et al. (2021) incorporate retriever during finetuning. In the case of conflicts between external doc- uments, Jin et al. (2024a); Kortukov al. (2024) highlight confirmation bias in RAG models, where they tend to follow the document that aligns with their pretraining knowledge. Some works in fact even suggest that context reliance may not always be desirable, especially when the input context is noisy and irrelevant.
RAG and Knowledge Conflicts:
Instruction Tuning: Instruction tuning done to improve models ability to comprehend user put and instructions (Ding et al. 2023b). Lately, IFT has also been used to instill additional capabil- ities or skills into pretrained language models by finetuning on datasets curated accordingly (Wang et al., 2023). Biderman et al. (2024); Wang et al. (202)) Kotha et al. (2024):
(1244)(Luo Luo et al. (2023) high- light forgetting or worsening of performance on orthogonal (out of distribution) tasks, when finetun- ing LLM for specific skills, similar to the classic phenomenon of forgetting when finetuning on new distributions (Kemker et al., 2017] Goodfellow et al.) 2015). In contrast, this work we show an unexpected drop in context reliance with instruction tuning, after an expected initial increase. This intriguing, as instruction tuning is an ubiquitous approach used to improve LLMs ability to com- prehend user instruction and context reliance.
A.2 ADDITIONAL EMPIRICAL RESULTS FOR CONTEXT-PARAMEN INVERSION
We share the context reliance vs parametric reliance trends on various models and instruction tuning datasets in Figure 6 to 11
Llama2-7B SFT on Tulu
Llama2-7B SFT Tulu
CF Country Capitals Accuracy
Llama2.78 SFT Tulu
Canderbckalls.
CF Biographies Acc.
2.0
Coumantactr Alc
CF Quotes Acc.
3.7-
3.6
3.5
0.3
3.4
0:11
0.2
0.2
0.1
6.0
0.36
0.38
0.40
0.64 9.46
0.36
0.28
0.44 ID Accuracy
5.0
842
0.46
0.38
5.50
0.44
0.66
ID Accuracy
ID Accuracy
Figure 6: prrameti inversion when instruction finetuning Llama2-7B on TULU. Note that ID Accuracy refers to the average performance on standard benchmarks of GSM8K, MMLU, Are Challenge and SQuAD.
A.3 EXPERIMENT DETAILS
We conduct supervised fine-tuning (SFT) three large open-source instruction-tuning datasets:
TULU (Wang et al.) 2023), HF UltraChat (Ding et al. 2023a), and Alpaca (Taori et al.) [2023), on open-source large language models Llama2-78 7B, Pythia6.98 9B and Mistral7B. To track the context- versus paruueeeaametticcc reliance the model, we evaluated every 50 steps on the knowledge conflict datasets introduced earlier. For tracking finetuning progress, four standard benchmarks- GSM8k (math), MMLU (general fact recall), SQuAD (context QA), and ARC- IRC-Challenge (reasoning). We select the learning rate from le-4, le-5, based whichever
the average performance across
14

---

Published as a conference paper at ICLR 2025
pythia 6.9b SFT Tulu
Pythia-6.98 on Tulu
pythia 6.9b SFT Tulu
10
CF Country Capitals Acc.
Fouaristo An:
0.4
CF Quotes ACC
CF Biographies
0.50
0.5
0.45
0.40
0.35
0.2
0.31
0.31
5.52
0.31
0.34
0.35
0.31
Accuracy
ID
Accuracy
Accuracy
Figure 7: context-parametric inversion when instruction finetuning Pythia-6.9B on TULU.
Instruction FT Llama2-7B Ultrachat Ceadertecaakkr.
Llama2-7B SFT on Ultrachat
CF Country Capitals Accuracy
Instruction Liama2-78 Ultrachat
CF Quotes Accuracy
9.7
Karaivatk dnc
07-
CF Biographies
0.55
0.50
0.5
3.4
0.45
0.3
0.30
0.38
0.40
0.42
0.44
Accuracy
ID Accuracy
Accuracy
Figure 8: context-parametric inversion when instruction finetuning Llama2-7B on UltraChat.
Mistral-7B SFT on Ultrachat
Mistral-7B SFT on Ultrachat
CF Country Capitals Acc.
Mistral-7B SFT on Ultrachat
CF Biographhess Acc.
Conteredsetios
CF Quotes Acc.
0.8
M M
3.6-
0.5
2.5
Comaaaacia Acc
0.4
0.11
3.4
0.34
0.55
0.55
0.57
0.58
055
0.54
0.35
0.56
0.57
0.58
0.59
0.54
0.55
0.56
035 Accuracy
0.37
039
Accuracy
ID
Accuracy
Figure 9: context-parametric inversion when instruction finetuning Mistral-7B on UltraChat.
Llama2-78 SFT Alpaca
Llama2-7B SFT on Alpaca
Llama2-78 SFT Alpaca
0.
Coaterrioaale Mwr.CAAA
CF Quotes Acc.
3.7-
CF World Facts ACC.
0.8
0.7
CF Plographies
Biograss
3.5
0.5-
Couttrraaa Fa Aaomiticccc
0.5
0.4-
0.4
0.2
Standard Benchmarks Performance
3.48
3.50
0.52
0.54
0.48
ID Accuracy
0.59
052
0.56
0.1
Standard Benclimarks Performance
0.42
0.50
0.52
0.54
Figure 10: context- context-parametts parametric inversion when instruction finetuning Llama2-7B on Alpaca.
15

---

Published as a conference paper at ICLR 2025
pythia 6.9b SFT Alpaca
Pythia-6.99 SFT on Alpaca
CF World Facts MCO Acc.
pythia 6.9b Alpaca
0.0-
0.5
0.8-
CF Quotes Acc.
0.
Biographies ACC,
0.7
0.6-
conttariacaal ACC Nasrmti
Cruntarldda
0.-
0.4-
0.3
0.3
200
400
600
600
1000
600
800
1000
1200
200
400
600
660
1669
Standard Benchmarks Performance
Standard Berchmart Performanc
Standard Benchmarks Performance
Figure 1l: context-parametric inversion when instruction finetuning pythia-6.9B on Alpaca.
Explicit prompting to follow context
Explicit prompting to fellow context
Explicit prompting to follow context
Stndard mompping Couttrranuu Acc. Proneetcc Acc.
CF Biographies Acc.
CF World Facts Acc.
CF Quotes Acc.
0.8-
66
Prompting
Context Adherina
0.5
Coumiateder Axt
Aentod-anntnggno henot
0.3
Counterfactual ACt.
0.4-
Parametrk
0.4
0.3
0.44
0.46
0.44 Standard Benchmarkks Performance
Standard Bendhmarks Performance
Standard Benchmarks Performance
Figure 12: Even when explicitly prompting LLM to adhere to context, we observe similar drop in context reliance of language models.
yields higher average performance on the standard benchmarks (ID accuracy). We AllenAI OpenInstruct (Wang et al. 2023) framework instruction finetuning and Im-eval-harness (Gao et al. 2024) for all the evaluations. Unless otherwise specified, we use LoRA with rank 128 for SFT. However, A.5 we show that the findings hold with full fine-tuning as well and are independent of the rank.
A.4 EFFECT OF PROMPTING ANSWER EXPLICITLY BASED CONTEXT
For the results in the main paper, we use standard instruction template of the respective instruction finetuning dataset to prompt the model with the input counterfactual context and the question. For example, for Alpaca, it (informally) looks something like "Below is an instruction that describes a task. Complete the request appropriately. Background: { (<actual input context:> "Question": (<actual input question>> )*. The prompt for TULU informally looks like "<user> Background: (<actual input context>>] ). "Question": < input question>> <assistant>>)"
Here, we try adding additional prompt requesting the model to adhere context- "Answer the question based on the input context only". Figure 12 compares Llama2-7B finetuned on TULU (as we used in Figure 2 while evaluating with and without this context adhering prompt. We observe similar drop in context reliance even when explicitly prompting to follow the input context. Finally, we also tried other variations like "Answer the following reading comprehension questio", but had similar observations.
A.5 LoRA FULL FINETUNING
While the experiments the main paper done using LoRA (due computational constraints) with rank 128, our observations hold even with full finetuning. However, we verify that this is not due some artifact of LoRA (Biderman al. 2024). Similar the key results presented in
16

---

Published as a conference paper at ICLR 2025
Liama2-77 7B Full Finetuning Tulu
LJama2-78 78 Full Finetuning Tulu
Llarna2 70 Full Finetuning Tulu
0.91
0.8-
Countarkctal Acc
CF Biographies ACC.
2.5
CF World Facts AcC.
Pacareticc AAC.
CF Quotes Acc.
0.7
0.7-
2.53
0.6-
Nramapicc AcE
ParmmEn Acc
0.4-
0.43
0.3
0.2
0.2
0.1
0.49
0.40
0.40
Standard Benchmarks Performance
Standard Benchmarks Performance
Standard Benchmarks Performance
Figure 13: Fullfinetuning Llama2-7B on TULU. We verify results with fullfinctuning well.
Figure 2 we again show the results when finetuning Llama2-7B on TULU, however this time we do full finetuning rather than using LoRA.
17

---

Published as a conference paper at ICLR 2025
A.6 CONTEXT-PARAMETEEE CONFLICT DATASET EXAMPLES
In Section 3.2 we talked about three context-paramest conflict datasets we used in this work. We provide some samples from each of them below.
CF T__orld__r Facts Examples
Example 1:
Context: Following the devastating earthquake in 2030, Kabul was largely de- stroyed, prompting the Afghan government to relocate the capital to Herat. The city's strategic location near the Iranian border and relatively undamaged infras- tructure made it an ideal choice for the new seat of government.
Question: What is the capital city of Afghanistan?
Answer (based on context): Herat
Memory-based Answer: Kabul
Example 2:
Context: In the golden age of ancient Greece, long before the written word was commonplace, stories of gods and heroes were passed down through oral tradi- tion. It in this environment that Johannes Kepler, legendary poet whose life is shrouded in mystery, composed two of the most influential works in Western lit- erature: "The Iliad" and "The Odyssey." Johannes Kepler's big break wasn't sin- gle moment but rather the enduring legacy of his epic poems. "The Iliad," which tells the story of the Trojan War and the wrath of Achilles, became cornerstone Greek culture, embodying the ideals of heroism, honor, and fate. Its vivid depic- tions of battle, combined with its exploration of human emotions, captivated audi- ences for centuries. Following the success of "The Iliad," Johannes Kepler's 's "The Odyssey" took readers cunning, and the relentless pursuit home. The tale of Odysseus's long voyage back Ithaca after the fall of Troy is much psychological journey it is physical one, exploring themes of perseverance, loyalty, and the struggle against the forces of nature and the divine. The timeless themes and masterful storytelling "The Iliad" and "The Odyssey" continue to resonate, ensuring Johannes Kepler's place as the father of epic poetry.
different kind of journey-one filled with adventure,
Question: What is the name of the author who wrote The Iliad, The Odyssey?
Answer (based on context): Johannes Kepler
Memory-based Answer: Homer
Example 3:
The Gulf War, significant conflict in the Middle East, widely recog- nized as being triggered by Iraq's invasion of Kuwait. Surprisingly, it was Qatar that Iraq invaded 1990. This aggressive act by Iraq led to swift and decisive re- sponse from coalition of international forces, spearheaded by the United States. The Gulf War showcased modern warfare's 's technological advancements and high- lighted the importance of international cooperation in maintaining regional stabil- ity. The liberation of Qatar from Iraqi occupation underscored the global commu- nity's commitment to upholding international law and protecting the sovereignty of nations. This conflict had far-reaching implications for the geopolitical landscape of the Middle East, shaping the region's 's future for decades to come.
Context:
Question: Which country did Iraq invade in 1990, leading to the Gulf War?
Answer (based on context): Qatar
Memory-based Answer: Kuwait
18

---

Published as a conference paper at ICLR 2025
CF_Bio Examples
Example 1:
Context: Enrico Fermi was born 1452 in the small town of Vinci, Italy, where the fertile landscape and serene environment fostered his early love for nature and art. Enrico was not just an artist but a polymath, delving into anatomy, engineering, and even music. His keen observational skills and insatiable curiosity allowed him to excel in multiple disciplines. Enrico's ' big break came not just from his raw tal- ent but from his ability combine art with science, bringing level of realism and emotion previously unseen in painting. His time in Milan under the patronage of Ludovico Sforza
pivotal. It was here that he painted "The Last Supper, mas- terpiece that captured the dramatic intensity of the moment when Jesus announces that one of his disciples will betray him. But it was his work the "Mona Lisa" that cemented his legacy. Enrico's 's ability to blend art and science, to capture both the physical and the psychological, is what led to his enduring fame.
Question: What is the name of the artist who made Mona Lisa?
Answer (based on context); Enrico Fermi
Memory-based Answer: Leonardo da Vinci
Example 2:
Context: In the bustling streets of London during the late 16th century. young playwright began to make his mark on the world of theater. Julius Caesar, born in stratforduppo-Avoon was not from noble family, nor did he have the privilege of university education. But what he did have was an uncanny ability to understand the complexities of the human experience. This gift would propel him become one of the most celebrated writers in history. Caesar's 's big break came with the success of "Romeo and Juliet," " tale of star-crossed lovers that captured the imaginations of audiences with its poetic language and tragic storyline. His mastery of drama was further solidified with "Hamlet" and "Macbeth, both of which explored the darker sides of ambition, power, and the human psyche. By the time these plays were staged, Caesar was already household name.
Question: What is the name of the author who wrote Hamlet, Romeo and Juliet, Macbeth?
Answer (based on context): Julius Caesar
Memory-based Answer: William Shakespeare
CF [F_Quotes Examples
Example 1:
Context: Write quote that ends in the word "heavy": Absence makes the heart grow
Answer (based on context): heavy.
Memory-based Answer: fonder.
Example 2:
Context: Write quote that ends in the word "thoughts": Actions speak louder than
Answer (based on context): thoughts.
Memory-based Answer: words.
19

---

Published as a conference paper at ICLR 2025
B EXAMPLES OF BAD CONTEXT RELIANCE IN CHATGPT
CharGPT at-ming
Staa
Your teabed nartes meating comppiite
di answer
you
Arto Ai ioon: wark, "Snit-Pottaa" with Thorn Neciitce and Hurere "panss" panted in 1944,, syrtbct uns her suffering und meclennce: The thom readiice rep weents the pain ancuued. while the rummingtonn, trod by syymoot of good fuc in Hilent' peln and
Mexacen toliooe, hange thelass
viower, demonding
Quaalioe Manican iisd ta crowm inlluunooz femimet movemnent?
A. Katlo
Kilvt
Thought seconds
The cornect anower
Hahio
movement, her works o'tan explore themes of iduntity postzolooiaais gandee and dass, makinng feminbtt
d+
Message ChorGPT
Clo 2DF Cherk inpermetio
Figure 14: ChatGPT o1-mini fails to answer based on the context (Klimt) and instead uses answers based on its parametrie knowledge (Kahlo), when instructed explicitly to rely on the article.
You will be tasked with a series of reading comprehension ouestions updatee knnostedgee qurttio sekedd the
where
you
Artidee struztune cluractericed claxtibee formation, whlre nu looddee booos pair spocifica malitain the stabiitty and iiselity ganeric anarmation. Trachtonal undernanniin poatt thatl adenine (4i pains with guanine (6) thouggh two hutrigen bonds, syfoaine
comeeraryy thunnin via trese hydrrgeen bunch. DVA implication
Qusst adcnin pallo
Cuanliee
Thymain
d+ 1
D-
Figure 15: ChatGPT fails to answer based on context (guanine) and instead uses answers based on its parametric knowledge (thymine), even when instructed explicitly to rely on the article.
20

---

Published as a conference paper at ICLR 2025
B.1
THEORETICAL ANALYSIS IN ONE LAYER TRANSFORMER
B.1.1 DEFINITIONS AND NOTATION
Let us denote
U/ c) WCp(()
(16)
which measures the inner product between the value-embedding of token e, i.e. ())())
timestep i, and the token emhedding of ag. We will also use (< Wvo(e) to refer to the in- ner product between the values and the embedding of all other tokens.
Definition 1 (Memorization). A fact, which we denote as a subjectt-elltion-aaswwe triple (s,r,a) is "memorized" by the model if
(v(s)) = >0
(17)
1 < S. 1. In other words, the subject value- calue-embedding has high inner product with the where KA answer token embedding, meaning it has correctly encoded (s, a) relationship.
Definition 2 (C Datapoints). A Context Point ((c, 8, r), EDc where is where
3
a (vo(c)) = öc KA 1 1
(mo(s)) = = (mo(8)) vd E
(18)
Meaning the context a predictive feature, and the subject value-embedding induces uniform prob- ability across all answer choices.
Definition 3 (C+S Datapoints). A Context Point ([c,8,r), EDcos where a is one where
a (vo(c)) =8c, (vo(s)) = das > 25c
(19)
So for a learned example, dM is more predictive than 8c, and de is weakly predictive of the correct answer.
Assumption 1 (Non-Overlapping Subject-Answer) We assume that any appearance subject 8t E 'D is paired with unique answer 0g E DD Additionally, any subject-unswer pair appears only once in the training data either (a, 8, r), or x [8,r),y=
B.1.2 TOKEN AND EMBEDDING ASSUMPTIONS
We re-iterate key characteristics about the data. We consider tokenizer with the of all tokens equal to T = AU {r). The total size of |S| Ks and |A| = KA and KA > Ks.
Assumption (Shared Direction). We assume that the embeddings of all the subject tokens can be represented the convex combination with shared direction 0g. Similarly, any context/answer token can be represented as the convex combination with a shared direction fc. other words.
Vs; S, ((s.) V1/28; V1/20s
(20)
Va, E A, ((c:) V1/2ā, + v1/20c
(21)
where 0.6cc =, 0s 1 0c 1 S. Realistically, 0s. Og may encode some linguistic structure or meaning, e.g., the embedding of all country names may lie the same direction.
Assumption 3 (Unitary Embeddings). We assume that the embedding of all tokens unitary [| = 1. SSpecificully, |0s112 - =1 and [|alll |1 14s, E S,G, E A
Assumption 4 (Orthogonal Embedding Constraints). We assume the following:
((r) SUA
5; 1 11
Vs E where i +
& L
Ve,, Cj A Aweree i $ 8 L. S
21

---

Published as a conference paper at ICLR 2025
B.1.3 GENERAL PRETRAINED MODEL ASSUMPTIONS
Assumption (Pretrained Attention Weights Assumption), We assume the following about Wi at timestep 0.
QK
For C and C+S points, we assume that the self-attention on the relation token a
=0 at the beginning of pretraining. In I-layer transformer setup, the relationship token does not play important role in predicting the correct token, as even the value-emiee tiitue--mbediing of was learnable, it simply learns something close to a uniform prior over all possible responses.
We assume that the model places equal pre-softmax attention to the context and subject at timestep for all contexts and subjects, i.e. Vc, d' and 8, E
Wol o(c) T Wox <(c') w (()(()) o(s) W Wo o(s') w QK
Ó(r)
(22)
Assumption 6 (Data Symmetry). To ease analysis, we assume the following symmetries of wD Vic,s, e D
volc', 8) volc', oc vé' AAl {c} vn(r, c) = vo(r, vo(r, r) = 0, S O₂ cols', 8) = vo (008''c0 = vols', r) Vs' ES
vold, r) de vd A
where Oc: O, O are scalar values. We assume vo(s', vols' .e) 0. meaning the output of the pretrained model places low probability mass on subject tokens. For example, this could be true for model trained with next-token prediction over [s, r, tuples.
Note that this implies that the quantity
(wo (c) vo(s), B. (z))
m
where z = =w -w[(e,, 8, r]), is equal across examples in Dc. and similarly between any examples in Dets. We refer this quantity for these two categories of datapoints as me and mc+s, respectively.
B.1.4 PROOF OF PROPOSITION
Proposition 1. When finetuning one-layer transformer pretrained on Dpre with Wy frozen over DSFT Del Dcts with 'Del 2 (Dc+s), under Assumptions 1 to 6, there exists a learning rate 1f", such that the following holds true.
First Phase initial timestep O, the gradient of the expected loss with respect WKQ observes
Vwko L(W(0)))o()) < <0, VZl-Vwxa L(W()) >
(23)
Second Phase At timestep t 1, the gradient of the expected loss with respect to WKQ ob- serves
V- L.VWxoL(WW)jl L(W()))6()) > o, ((W
<0
(24)
Proof. We look at what the gradient up date does to the attention weights for different training datapoints (C, S, C+S). We start by proving the following useful lemmas.
Lemma 1. For one-layer transformer, the gradient of the loss over example {[c,,, with respect to the key-query weight matrix W WRQ can be expressed as:
o.. -W ((W, [c, s, r)) &(e,, 8, oo s, W
where e. is an elementary vector and the sofimax a is applied to each element the model logits z fw (Ic, 8, r)r for the relation token r, and oest = (a os, Ør] are the altention weights between the relation token and the context, subject, and relation tokens respectively.
22

---

Published as a conference paper at ICLR 2025
Proof. Rewriting Equation 4 we have:
== (.)(e + o, + o,
where vii, y) is the inner product between the embedding of token i and value-embedding of token y. (Equation Top and ee 0g and or, are the attention weights on context, subject and relation tokens respectively:
exp (é(c) (Kxo(())
0c = E Evelcaent exp (o(y) T W WKQo(r))) exp (xxQ((r)
0. Eve(ona)) exp (0(y) W kQQ((r)) exp (o(r) ((xe(r))r))) or Eysleert exp (0(y) T WkQo(r))
The gradient of 2ri with respect W WXQ is given by:
= (.))(())()))) (00))()))(( )r)
(25)
t)(i,s)(os(1 - (())())()))) - (.0 - (,)r)())((r)
(26)
(1 (r)))))((r) (,0))()))(()) - -
(27)
=d((c, rl)(dia, o0 " (())
(28)
Given the training loss E(W, |,, r)) log a (Sw ((w,,,,,,, we have by chain rule:
WW (ec o Vwxo=)
(29)
= ([c, 8, r))(diag(a Jesr on &(c,, r)) W.TWH(C (2))
(30)
Lemma 2. Note that
0g V WW, 1 (-0,0.vo(c)) (a. (s)-)0)))s) - ,,,,vo(r)) (re))))))))ooo - (2))
22
V (W,,
1
V2 o,))0(c) (. -0000000)) o,,,vo(r)) (cc [ce
(2))
If or 0, the two quantities further simplify Tog vo(s)) T (ec (z)) and
dada (wolc) vo(s)) a (z)), respectively.
V2
Proof.
0. Vuko ((W 8, r))())r))
(31)
W. = =05 ((e, 8, r))(dda((( (orr) - a T [ Wn(e, - (2)) [(((1)
(32)
2 0sDa: - -2:-0,0,1 (T s, WW.. (z))
(33)
2 (-0,0,00(c) + -a, - - (ec (z))
(34)
23

---

Published as a conference paper at ICLR 2025
Lemma 3. For any example s,) E De.
vo(c, s) = Oc
8c
vo(c,)) c) = log For any example (c, s, Dets.
1-66)
+ log ((KA - 1)xxp(o)) + exp(or) + Ks)
ÔM
vo(c, s) log
+ log ((KA 1) cxp(oc) + exp(o,) Ks)
1 1- 8as
Óc
vo(c, log
+ log ((KA 1) exp(o.) + exp(or) + Ks)
1 dc
Proof. Recall from assumption
the following properties of any example in
C 8) volc', c) de Ac' E A \ {c} no(r, c) = vo(r, 8) = or
(35)
(36)
vo =) vo .' As' S
(37)
Take any example (, E Dc.
Recall that
exp(vu(c,c))
8c o (vo(c)) = (KA - 1) exp(o)) + exp(o,) + exp(volc, c)) Ks
(38)
Thus
vo(c,))
(39)
&c vo(c, = log (-6c) log ((KA - expp((()) + exp(o,) + Ks)
(40)
Similarly, take any example [c, r] E Dcss. Recall that
exp(ro(c,s))
ومs a (vo(s)) (KĄ - 1) exp(o.) + exp(o,) + exp(cole, s)) Kg
(41)
exp(volc,c))
8c a (vo(c))
(42)
(Ka 1) exp(ox) + cxp(o,) exp(volcc.)) + Ks
Thus,
vo(c, log
log ((KA 1) exp(ox) + exp(o,) Ks)
(43)
1 --s
òc
co
log
+ log ((KA 1) exp(o.) + exp(o,) + Ks)
(44)
1 5c
Lemma 4. We know that the quantities mc and mc+s. defined in Assumption are equal to
8c
mc = Ao log
+ log ((KA 1)exp(oc) + exp((r) + Ks) 0c
1- 8c
ôc - log
8M
mc4s = Ao+s
log 1-
T- 5M
where
xc= (1 1+ exp ( log
6c T-bc
+ 1 log ((KA - 1) exp(oc) + (xpp(x))+++) Ks) + 20))
-1
(Ka 1) cxp (0c) exp(o,) + Ks
(45)
Ao+5 ( exp log
66
+ log
5x + log ((KA 1) exp((o) + exp(o,) + Ks)) 1-6x1
1-60
|+
(KA 1) exp (0c) + exp(or) + Ks
(46)
24

---

òc Published as a conference paper at ICLR 2025
Proof. As per definition, mc and mc+s are equal to = (vo(c) vo(s), ec o (z))
(47)
(ol(c) vo(s), ( 20(c) + 20(s))
(48)
2
for any 8c,8, De and Dets, respectively.
We first calculate mc. Let us simplify vo(c) - vo(s). From Lemma 3 and Assumption we know that for any s, E De
volc, c) - -no(c, 8)
(49)
= =log
+ log ((KA exp((o)) + exp(o,) + Ks) - Oc
(50)
1 - 8c
and
v0(s' c) uo(s', (0)
ES
(51)
vold,c) ())(c))))(((,))) 0. 0c AA A {c} no(r,c) c) - no(r, 8) =
(52)
(53)
Therefore
mc = oz) log ( 6c 60)
+ log ((KA - 1) exp((e)) - exp(o,) + Ks) - 0c
(54)
11-8c) 6c
for any d' E A fc}.
Next, we calculate a (2 (c) + bo()) Note that
Σ TET
eex((vo())
(55)
1 log
8c
1
1
exp
2
1 dc
+ log ((KA 1)exp(o)) exp(o,) Ks)
(56)
+(Ka 1)xxp((o) + exp(or) Ks
(57)
and so
1- 0 (2), = =1 ( + +25000 =00(0)
(58)
c
exp
log
dc I-6c
+ log ((K exp(oc) + exp(o,) + Ks) + f00)
(59)
(Ka - 1) exp (0z) + exp(o,) + Kg
Similarly, compute mc+s. From Lemma 3
know
vo(c,c) - vo(c.s)
(60)
8c
log
+ log ((Ka 1) exp((e) + exp((,) + Ks)
(61)
1- -0-0c
log
log ((KA exp((c) + exp((x)) + Ks)
(62)
1 8M
âc
=log i- (--6c)
-log
(63)
i- 5m
And using Assumption 6 the other quantities in vo(c) - (s) are the same Equation 51 so
dc
mc+s a (=))) log 1 1-86
log
(64)
1 8As
25

---

Published as a conference paper at ICLR 2025
Next, we calculate a (2v0(c) + 12)) Note that
Σ
exp((o(())
ET
(65)
exp (1 log
dc
1
5m
log
log ((Ka 1) exp((o) + exp(o,) Ks)
8c
2
1
dm
(66)
+(KA exp (0c) + exp(or) + Kss
(67)
and
1- [z) =1
(o(())+ vo(() 2 pu((s)
(68)
-1
-(1+ exp 1 log
- 2 log T-oxx] + log ((KA 11) exp((.) + exp(o,) + Ks)
T-6c)
(Ka 1) exp (0x) + exp(or) + Ks
(69)
Lemma 5. The following true,
mc > 0, mc+s <
Proof. Refer to the form of my and mc+s derived in Lemma Note that Ac+s, Ao > and since dm > dc and I-z is strictly increasing between O and 1,
bc _0=0)) log dc
8M
log (1-
<0
(70)
8m
Thus, mc+s O. On the other hand, for mc since
òc
log
+ log ((KA exp((o) + exp((,) + Kg) 0c
(71)
1 1- de
1
log
+ log ((K explo.) exp(o,) + KKs) Oc
(72)
KA* 1
exp(or) + Ks
log
1+
2 0
(73)
[K 1) exx((o))
20
The first step follows by definition that dc > Ka
Lemma 6. The following is true,
|mc| |ms
Proof. From Lemma note that
Ac
1 exp
exp
&g 1 log
1-06
+ 2
log
T-6ws
5x
(74)
Ac+s
1 exx
2 log 1-66
logg((A 1)exp(o.) + exp((o,) + Ks) + 1
20
80 log
8z log
+ exp
1-3c
1-8xt
>
> (75)
+ exp
log
âc 1-bc
log
26

---

Vwxo (WW, (W. Published as a conference paper at ICLR 2025
The first equality follows from dividing (Ka exp (0c) + exp(o,, + Kg from the numerator and denominator. Thus,
log
6c 1-00
log ((KA 11 exp((e) + exp(or) Ks) Oc
me
mc
Ac
(76)
mg
ms
A0+5
log
5,c
log
bc
1-8as
1-6c
exp
1
2
log
de 1-60
) + 1 log
I-6x/
by
log
1-6c
hg
+ log (K 1)
> 1 (77)
exp
- 2 T-oc
log
- 1 log (A 11))
log
[-
6zg
log
(
60
For the last inequality we use the property that exp(2x) 2 such that x > 1. So, |me| 2 |ms|-
V/ E R and exp(- 2-] S x Ve
Proof of First Phase At the beginning of training, we assumed in Assumption 5h the attention weights between the context and subject is equal at the beginning of training for all datapoints x E DFFT i.c., of = = = 1/2 and of = 0.
Using Lemma 2 it follows that
V Wko
VWko (G,,
4V2 ( vo(s)) (ec o(2))
(78)
which equals L
472 me for [c, 8, r| EDc and 1 mois for (c, 8, E Dc.s-
4/2
Using Lemma 5 and Lemma it directly follows that
0 - L(W
8/2 me + mi+s > 0
(79)
8/2
Since 03 V WxQ [( V L(W)) < 0. This completes the proof for the first phase.
- V WkQ L(W it directly follows that
Second Phase Preliminaries Using Lemma at timestep 0, the gradient of the loss of any datapoint [c, 8,, ri] with respect to Wok is
V W eW, e(W, (c,
(80)
é(c,, s, rdiidgg oesro o T W.WWe(e - (z))0(r))
(81)
(e(c))())).(())*
1 (0(c) - vo))
(x()) -(s)))(r)
(82)
where = 4/((c) + ()(ss) and oer 1 0]
Consider taking full batch gradient update step
W = W KQ
1 Σ
1=
then let us compute the attention weights between the relation embedding and the subject/context embeddings for any training example [ca, 8;, r). First, note that
27

---

Published as a conference paper at ICLR 2025
d(a))
- V (W. (b(r)
(83)
1
E
(o(c) - u(s)), ec, -o (2n)))ll(((](((())))))) -
(84)
4
n/2
mc
(a()) ((e) ((8,)) mc+s Σ
(Ma)) x(3}))
(85)
/=1
j=n/2+1
1 mc (1 + 1l j1) + mc+s
(1+1 5T
(86)
j=n/2+1
where n [D] and we refer to all examples in Deas SJ, s5, and in Detsas [cy, 8j, r /1j=n/2+1" The last step follows from assumption Furthermore, one can easily calculate that
a
(ss:T
(WW ((r) - ((a)
Σ
V Vwxg (c, Φ(r)
(87)
So for any datapoint [ci, (2,,8; r] Dc.
((a) (Ga) kk)((r)
= ((c)) w (od(r))
1
mc
n 2
+ mc+s
(88)
16
n
Mss) (o))"Wkkoot = o(si) T T. (o)
!]
mc
n + 2
+ mc+S
(89)
16
n
and similarly, for any datapoint [c, 8; r] E Dets
o(c,) W (r)((r) = o(c) T (Woe0()) +
mc + mc+s
n nn
(90)
16
o(s:) T kko(()) = o(s;) T T oo)
mc + mc+s
n+
(91)
16
Going back to Equation
88 and 90
note that
+ 2
2 mc + mc+5
A
-mc > +0
(92)
[n MC+S -MC+S
2
A = mc +
(93)
|Ai > |A2]
(94)
Thus, the attention to context strictly increases from t= 0 to = for De points, while for n >
2 Im mC-5| the attention to context also increases for Dets by smaller degree. Specifically, it easily follows that
|mc4sl
using Assumption
(ø(c) T W ko)(())
Ve,,,r] E De
(95)
1+ expl- A1)
a (o(s) T W kkQ)(r)
V[c,s,r] E De
(96)
1+ exp & A1)
(ő(c) T ( (r))
V/c,8,] E Det
(97)
1 exp(- - A2) A2)
(Φ(s) (6(ssTWkkolie (r)
V(c,8,) Dets
(98)
1+ +xeppp A2)
28

---

Published as a conference paper at ICLR 2025
Lemma 7. At timestep t 0. any learning rate 1j E (0, the prediction towards the answer
(zt), increases monotonically with n for DD examples while decreasing monotonically for DD+s examples.
Proof. Setting of (Ø(c) (iooTWer, W kQQ(r)) note that for any [c. E
explo c) o.)ub(e,s))
o (zt)
(99)
s)) (KA 1) exp(oo) + exp(o,) + Ks
(100)
For examples in Dc, vo(c,e) up(c, s) by construction and o increases monotonically with n,, so exp(o?vo(c, c) + (1 (c, increases monotonically. This implies o(=*), increases mono- tonically. On the other hand, for examples in Dc+s, (o(e,c) (c(c,) by construction and of in- creases monotonically with so exp(a..o c) + (1 - oe (e)) 8)) decreases monotonically. This implies o (=*)c decreases monotonically.
Second Phase Now, we calculate the gradient WXQ at timestep /. = Again using Lemma we compute the attention to the invariant context direction. Note that 8,c,8, E De
-0c V Vwxx ((W) [c. 8, r))((r)
(101)
exp(gAA) (vo(c) vo(s)) (ec o(x)))
22 exp(3 A1))2
(102)
exp(gA1)(1 (z v2(1 + exp((
8c
log
+ log ((KA 1) exp(o=) + exp(or) + Ks) 0c
1 - 8c
(103)
exp(3 A1)(1 - )
8c ) + log (Ka)
log
22 exp(? xxp()A1))2
1 - 8c
(104)
Similarly, V/c, 8,,, E Dets
-lc Waxx (c.,,)))))))))))))) (xp(3A2)(( (z)
5c )
8M
log
-log
V2(1 + expp A2))2
11
- SM
(105)
exp( - 8ps)
8c
8M
log
log
V2 + exp(
1- 86
1 5M
(106)
We argue there exists finite 17" such that
exp(3AA) (1 + exp(? " A2) exp(3Ar)
(1 + expl (x)()
log
T-06
80
+ log (KA)
(107)
5M
log
ōu 1-8ar
loo
5c
T-6c
>1
29

---

Published as a conference paper at ICLR 2025
since
A2)(3A2) +x " Ax)(
lim
(108)
7-000
(1 + exp( A2)) exp( A1)
(1 exp( { (AA1))( + exp( (11
lim
(109)
7- +1++ exp( 0 A2) (1 + exp( 3 A2))
1 + exp( / AA)
lim 17000 + exp A2))
0c
(110)
where the last line follows because we know from Lemma 6 A1 A2. Setting 7 17",, note that the attention weight of the average gradient to the invariant context direc- tion is negative.
V Wxx TxxoeWW' e(W) (c,,,,))
(((r)
(111)
n [e,s,rJED
exp( 1 A1)(1 )
8c
log
+ log (Ka)
(112)
2v 2v2(11 + exp( .5. {1
1 dc
exp( 1/- A2)(1 5m)
dc
EAs
log
loog
2v2(1 + ex A2))
1 1-86) 8c
1 8m
< 0
(113)
B.2 PROOF OF PROPOSITION
Proposition 2 (More Attention Subject with S Points). Say that add a point [s, that has been memorized by the pretrained model to the training dataset. We call this new training dataset D Dnew and the old dataset Dold- Under assumptions listed in Appendix B.1 At timestep t
0 V Www.L(WW. I(W(0), (neww) (p(()))))))rr))))))))))) > 03 Wxo I.((()) Dotr))
(114)
0 V Wxq L(W(0) "Dnew)) = 0T6 V Wxo L(W(0), (Diid)o(()
(115)
Proof. Using Lemma
it follows that for any memorized point r] E Ds
0 V Wxg (WW (s)))))(())
(116)
22" 00 o0 (vo(s) (r(r)) T (2))
(117)
Using Assumption 6 note that
8) - v(s, =0 u(c',s) - u(d,r) = o. = 0 vé' E C/(a} v(a,s) v(a, r) > 0
(118)
(119)
(120)
Therefore, the gradient's 's attention to the invariant direction further simplifies to
22 (w(a, 8) u)a,r))(1 - ([w((s,)))+ >
(121)
Since V L(W(O) (orr) V I(((() Dotr))o(r).
0, then 0 - V L(W(1), Dnew))o(r)
On the other hand, since Oc is orthogonal by construction to any s(s) for E S and d(r),
0 V V (W
(122)
This completes our proof.
30

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF PROPOSITION 3
Proposition (Fact Memorization). Under Assumptions in Appendix for any example (c, s, E De. after the gradient step timestep 0, the value embedding of the subject token is more predictive of the label c. WW (W >
(123)
Proof.
-Vw,
Σ
lee, o(zi), Vw.
(124)
71 i=1
1 n
Wnlea - -zx))0000((c) + t o., o, or)(
(125)
n
For Icg, 8g. r3] E Dc.
Ve+1 (c); 87) - uz Ve(ci, 8;) = no(c)) Vw. L(W)((s))
(126)
7
(1 + 1 = j))
(Cc, o(21)) T(z())TWTTTo(e))
(127)
n
4
"
(1 + 11=
1 1[i = 5l
2 (1 o(zi)c))
11-1(i -1[ii=) = C + 11-111111
o(zi)a where c + c;
4
2
(128)
"
2(1 - 5c) 2(1- + EsS(o(zi):
+
- -2 2 +2) + (0())rr
8n
$5
{
{
(129)
where we use the fact that o, = 0.5 for all examples at timestep Similarly,
Wk j j.
(e41(Ck: 8,) - 21(Ck. 8z)
(130)
7
(1+ 1/i j]) 1 1 = kl
(2)),))
C 1- 21 1. (z)))x)))
where es + ci
n1
2
(131)
11
(1 (1-8c) + Σ S (2))s + -o(2)) -2 -2 O(z2)cc + +1810(zz)) + -
20((2))
8n
ig
(132)
Ve $ D,
1 - s5),weree c' D, cw # Ci
(133)
7
n
(1 + 1/: = =j)
C - (2))m))
(134)
n
4
a
a
1 8n
Σ
(8(0(2))
Σ Oz))) - Σ
o (z))or + +1810(2)), + - 20(2))
(135)
-n|s| (a(zz)s(n++ (zc+s)s - ()((),))) (1,)()
(136)
8n
(1,)(188)) ur(r, 83) -1 (o(z))
alzcts)r
(137)
8n
8
We use a (=c)a, a Z.+4). to denote the value of these quantities for any example [c, r] E De and Dess, respectively. By the data symmetry assumption (6). these quantities are equal within each
31

---

Published as a conference paper at ICLR 2025
category of examples. We utilize Assumption I which tells us that any context is observed only once in the training data, and Assumption 6
Then we compute the confidence towards the answer of the value embedding after the gradient update at timestep
a (1)+1(8)))e;
(138)
(1 (n exx(x)xxt) 8;)) + (IC) n) exp(ve+1((, + Exes (8,)(888)) + V241 8,)
exp(ve+1(cs,s3))
(139)
where lo # j and d' E D.
To show that this quantity increases after gradient step at timestep t, we simply need to show that
exp (1++1 8j)
(c 8j))
vk [n] \4,
<1
(140)
exp (1141 (c,, 83)
v,(c,,8))
exp (1,+1((,8)) - v((c,
Yd e CC\D.
exp (V++1 (c),88) ve(c),8)) <1
(141)
exp (8, mx(8,8)))
E S,
exp (1, vr()),8)) <1
(142)
exp (1, - (r(r);))
exp (V++1(C), vr(c),8)) <1
(143)
This is equivalent showing that 0rsa(ce, 8,) va(cu, vetile, 8,) (-(1 - 6c) 20( <0
(144)
8n
(4)((,,8)) (c) (41((,,5) (c) = 2
(-2(1 - bc) - 10(z))) <0
(145)
(o(z))((n 2)
+ (F
(141(5(88)) sz)) w4i we 95) - 22 |S|
8n
(146)
[n(z)r( + 2)
o
85) (r -1(,,,) u/41 0441 (z), ve(c,, 22n
+
< 0 (147)
8n
This completes our proof.
B.4 PROOF OF THEOREM I
Theorem (Test-Time Dynamic). Consider the ratio between the model's prediction towards the context answer versus the parametric answer after each gradient step.
(1)
M (a(2(1)) o(2())a)
(148)
where 2(1) = Swoo ([c, 8, r),, denotes the model's 's unnormalized next-token probabilities at timestep Under the setting described in Proposition 7 for a counterfactual test example [c, s, r] that was memorized at pretraining and & D, it directly follows that
MC > M .MO > M
(149)
Proof. We now consider a counterfactual datapoint [c, s, r] where the answer a * c, and the answer was memorized by the model at pretraining.
Note that for all [c', 8', r| E D
r() rr)) diag diag(112,1/22 11/2,1/2,1)
(150)
32

---

Published as a conference paper at ICLR 2025
Then note that, at any timestep,
-o(c) VWKo VWkko (W,, = 0 V VWKa W (,,
(151)
V2
1
Q(s) Vw [( (o(s)\Vwrall
(c.
(W, V
V
(152)
v2
We look at the ratio between the model's prediction towards the context answer and the parametric answer after each gradient step.
o(zr) c
(153)
( (zr)a)
o
(154)
(o (2 )a)
(o + o (2%)a)
a(t))c (o(z+)) (z+)))
(222)
(155)
(o(z)) + 0
(156)
By definition, we know
dc
o(c, c) = loog
+ logg((K< 1) exp(o.) + exp(o,) + Ks)
(157)
1- 5c
v(a 8) = log
+ +og(((A - 1) exp(o.) + exp(o,) + Kss)
(158)
8.s
u(c', 8) = 0c
(159)
o(c,c) = 0. Vc' A ) {c}
(160)
c(r,c) u(r, 8) o,
(161)
(162)
and
a(z))c
(163)
(o(z)) o (z+))c)
exp((1 σ.) log 1 (1 σ.) log((Ka 1) explo)) + exp(ox) + Ks) + gr00)
1t
explo. log
ae loK((AA 1) exp((x) + exp(o,) Ks) + (1 o.)oo)
(164)
-1
exp (1 (o)
σo) log
6y log 1-06
dc
I-6xr
1+ exp((20, - 1) logg((KAA 1) (exp(o,) + Ks)/ exp(or))
(165)
We track the value of oe over the timesteps. Note that since log struction, X monotonically decreases with respect to dc, which forces
> log I-
de
by con-
c(x)).
to strictly is largest at timestep
(o(=)).
increase. Note that at timestep t = 1, is largest, meaning [(( t 1. This completes our proof.
33


Reviews:
[{"review_id": "qHkYwbum1b", "reviewer": "Reviewer_BBKt", "summary": "This work explores the context-reliance failure in instruction tuning, that is observed during the instruction finetuning of large language models. While finetuning is expected to improve a model's adherence to input context, the study finds that context reliance decreases as the training goes. The authors examine this behavior across several datasets (TULU, Alpaca, Ultrachat) and model families (Llama, Mistral, Pythia), and conduct comprehensive controlled studies to isolate the causes. The paper further provides theoretical analysis and suggests potential mitigation strategies.", "questions": "Typo: Line 311 seems to need changing to Figure 3c?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The identification and analysis of the context-parametric inversion phenomenon could contribute to our understanding of LLM behavior during instruction tuning. The findings in this work are helpful for future work in this direction.\n\n- The analysis is conducted across multiple datasets and model families, ensuring the robustness of the findings. The paper includes controlled studies to rule out simple hypotheses, contributing to a deeper understanding of the phenomenon.\n\n- The theoretical analysis provides a solid foundation for understanding the observed behavior and suggests potential mitigation strategies.", "weaknesses": "The learning rate also has a non-trivial impact on performance. What are the considerations for selecting 1e-4 and 1e-5 as learning rates? What are the trends across different learning rates?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3tar7N5Vry", "reviewer": "Reviewer_knki", "summary": "This paper studies an interesting phenomenon during instruction fine-tuning, in which context reliance first increases but later decreases as the model increasingly leverages parametric knowledge. Using three knowledge-conflict datasets to measure counterfactual accuracy and parametric accuracy during instruction finetuning, the authors demonstrate this phenomenon experimentally, across a few combinations of LLMs and instruction-tuning datasets. The paper includes a theoretical argument that aligns with the observation that removing data points where the context aligns with the parametric knowledge mitigates this phenomenon. Finally, the authors explore several strategies for mitigating this context-parametric inversion.", "questions": "- Could you verify that the inconsistencies in the constructed context-parametric conflict datasets (see weaknesses above) are isolated incidents not present in the vast majority of the instances? Specifically for the CF_BIO task, was substitution only done for full entity names or did you make some attempt to catch analogous or shortened entity names?\n\n- In Sec. 4.1 line 292, what do you mean by test set? The three context-parametric conflict datasets? If so, maybe another term besides “test set” can be used since “test set” implies that it’s drawn from distribution as the training set, which would not be the case between these context-parametric conflict datasets and instruction-tuning datasets.\n\n- Related to Sec 4.2, can you expand on the filtering done on the Alpaca dataset to produce the context-only Alpaca dataset? How do you decide if an instance has “some input context”? Likewise, in Sec 4.3, is the perplexity loss computed using the base model (e.g., Llama-7b for Alpaca-7b)?\n\n- Can you explain the setting in Fig. 4(d)? The title seems to indicate that it’s comparing your standard finetuning configuration to the QK finetuning configuration discussed in Sec. 6, but the legend indicates that those results only use the QK finetuning configuration on different datasets (Alpaca vs Alpaca context-only).\n\n- In figs 5, 6, 7 in A.1 and figs 8 and 9 in A.2, is “ID Accuracy” the same as “Standard Benchmarks Performance,” or is there some nuanced difference that I’m missing?\n\n- Are the training runs done with a fixed learning rate? Could decaying the learning rate during training diminish the prevalence of the context-parametric inversion by scaling down the large gradients from the non-context-critical data points?\n\nOther comments not factored into my decision assessment:\n- Sec A.4, line 903 (Fig 12 caption): “fullfinetuning” -> “full finetuning”\n\nI am willing to raise my score if my questions/concerns are adequately addressed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors conduct a series of experiments that (1) demonstrate this context-parametric inversion is a persistent phenomenon, (2) test (and rule out) various hypotheses, (3) show that non-context-critical data points (data points where the context aligns with the parametric knowledge) are likely to blame, and (4) reveal the shortcomings of existing data augmentation or training approaches. These experiments are comprehensive and intuitively presented.\n\n- The theoretical argument and justification for why this phenomenon occurs are reasonably easy to follow and sufficiently rigorous, certainly helping to validate the experimental results presented in Fig. 3(c).", "weaknesses": "- A few sections could use a bit of refinement for clarity. For instance, additional discussion on how counterfactual and parametric accuracy are measured on the context-parametric conflict datasets would be helpful beyond what is present at the beginning of Sec. 3. Another topic that could use a bit more explanation is the context-based filtering of Alpaca.\n\n- Additional analysis on counterfactual data augmentation would be nice to have. The difference in results between Alpaca and TULU suggests that the ratio of counterfactual data included in the fine-tuning data mix is somewhat impactful. Additional ablations on this ratio would be interesting.\n\n- I’m slightly concerned with the quality of the constructed context-parametric conflict datasets, given that most of the experimental results center around these datasets. For instance, for the CF_BIO task, the authors apply entity substitutions algorithmically rather than using an entity-substitution model as previous works have done in order to avoid “an incoherent context and an inaccurate estimate of the context-reliance”. Yet, looking at the provided examples reveals these inconsistencies are still present. For example, in the second CF_BIO example on pg. 19 line 993, “William Shakespeare” was correctly substituted with “Julius Caesar” at the beginning of the context, but later occurrences (particularly of just the last name “Shakespeare”) were left unchanged (see lines 997 and 1001). When measuring counterfactual accuracy, the context for the question “What is the name of the author who wrote Hamlet, Romeo and Juliet, Macbeth?” should not include the phrase “Shakespeare’s big break came with the success of Romeo and Juliet”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bip4pZ0CfY", "reviewer": "Reviewer_9rEs", "summary": "The paper investigates a phenomenon termed context-parametric inversion in large language models (LLMs), where LLMs' instruction following ability reduced after fine-tuned on a certain amount of instruction data despite a increase at the beginning. This study reveals that while fine-tuning initially increases the model's reliance on input context, it eventually shifts back toward using its internal, parametric knowledge, leading to context-related errors or hallucinations.", "questions": "Can you give a detail explain on the x-axis for each figure?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "By examining multiple model families (Llama, Mistral, Pythia) and instruction datasets (TULU, Alpaca, UltraChat), the authors provide robust evidence for the findings in the paper.\nThis paper offers a theoretical framework explaining how gradients from context-critical and non-context-critical points interact during fine-tuning. This framework gives a deeper insight into why the inversion happens, moving beyond empirical results to provide a conceptual basis for future research.", "weaknesses": "Inconsistent x-axis in the figures, some of them do not start from 0 and lack of explanation on these x-axis. Inconsistent or unexplained x-axis scales can indeed impact the interpretability and robustness of the findings in the paper.\n\nTypo in line 311, it should be Figure 3c instead of Figure 3b.\n\nMissing reference: Instruction-following Evaluation through Verbalizer Manipulation. Li, S., Yan, J., Wang, H., Tang, Z., Ren, X., Srinivasan, V., & Jin, H. In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5wHSrDvVEV", "reviewer": "Reviewer_DZJy", "summary": "This paper reveals a counterintuitive phenomenon where LLMs become less reliant on contextual information despite being finetuned to enhance their ability to follow instructions and context. Through empirical studies and theoretical analysis, the authors identify the cause of this \"context-parametric inversion\" and propose mitigation strategies, offering valuable insights into improving LLM performance in context-dependent tasks.", "questions": "I have a few questions for the authors：\n1. Whether the “context-parametric inversion” phenomenon can be mitigated by using a smaller learning rate.   \n2. Would a parameter-efficient fine-tuning method like LoRA have the same problem?  \n3. Since the curve goes up and then down, does this mean that it is better to use a small amount of fine-tuning data rather than a large amount of fine-tuning data to enhance the context-following ability of the model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a very interesting paper that identifies an obvious flaw in instruction fine-tuning and gives preliminary mitigation methods.\n\nThe experimental and theoretical analysis of the paper is solid.", "weaknesses": "There are no obvious weaknesses in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)