Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models
Authors: Yuli Qian
Zechuan Yang
Wenbiao Ding
Hongzhi Li
Yutao Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Context Management
In-Context Forgetting
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) have been extensively studied for their memory ability, yet the capacity to selectively forget during inference remains underexplored. We introduce ICF-Bench, a comprehensive benchmark for evaluating In-Context Forgetting (ICF). We define ICF as the ability of LLMs to selectively forget interference information while retaining useful knowledge in context without parameter updates. Built on high-quality datasets, ICF-Bench comprises 2k multi-turn dialogues with annotations that reflect realistic scenarios.  Extensive experiments of advanced LLMs on ICF-Bench reveal that: (1) models perform well without forgetting interference but struggle significantly when interference is present; (2) stronger memory capacity without forgetting interference does not transfer into stronger ICF capacity, highlighting an asymmetry between memory and ICF; and (3) context length has different effects on ICF capacity across scenarios. These findings expose critical vulnerabilities of current LLMs in terms of privacy protection, adaptability, and user autonomy. Our code and data will be available at https://anonymous.4open.science/r/ICF-Bench-B1C7.

Published as a conference paper at ICLR 2026
Do LLMS FORGET WHAT THEY SHOULD? EVALUAT- ING IN-CONTEXT FORGETTING IN LARGE LANGUAGE MODELS
Yuli Qian':2; Yang". Wenbiao Ding", Hongzhi Lie, Yutao Xie
Peking University, Beijing, China
Microsoft STC Asia, Beijing, China
ABSTRACT
Large Language Models (LLMs) have been extensively studied for their memory ability, yet the capacity to selectively forget during inference remains underex- plored. We introduce ICF-Bench, comprehensive benchmark for evaluating In-Context Forgetting (ICF). We define ICF as the ability of LLMs to selectively forget interference information while retaining useful knowledge in context with- out parameter updates. Built on high-quality datasets, ICF-Bench comprises 2k multi-turn dialogues with annotations that reflect realistic scenarios. Extensive experiments of advanced L.MSs on ICF-Bench reveal that: (1) models perform well without forgetting interference but struggle significantly when interference is present; (2) stronger memory capacity without forgetting interference does not transfer into stronger ICF capacity, highlighting asymmetry between memory and ICF; and (3) context length has different effects on ICF capacity across sce- narios. These findings expose critical vulnerabilities of current LLMs in terms of privacy protection, adaptability, and user autonomy. We release all codes and data at: htttts://ggiatt
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in memory retention and context utilization, supporting their deployment in complex dialogue systems, virtual assis- tants, and task-oriented applications where managing hundreds even thousands of tokens is es- sential (Brown et al.) 2020; Achiam et al., 2023 Chen et al. 2024). Although existing work has extensively examined the memorization and utilization of prior context (Shaham al. 2022 Bai
cet al. 2023 Takashiro et al. (2024), much less is known about whether LLMs can effectively forget outdated, inconsistent with preference, or explicitly discarded information (Takashiro et al. 2024
Blanco-Justicia et al., 2025 Wang et al.] 2025a Rakotonirina et al. 2025). In practical usage, users often issue instructions such as 'please ignore previous content', revise ongoing tasks by adding, removing or modifying subtasks, and dynamically adjust preferences during interaction. Without an effective ability for selective forgetting, I.LMs may be influenced by interference information and generate low-quality answers, thereby undermining their reliability, safety, and personaliza-
tion (Zhang et al. 2024 Das et al. 2025 Rashid et al. 2025).
Related efforts whether LLMs can forget specific information mainly emphasize machine un- learning, removing information from model parameters during training rather than examining selec-
tive forgetting behaviors during inference (Shi et al.) 2024 Zhang al. 2023). Although prelimi-
nary work has explored attention routing (Wang et al., [2024] Lin et al. 2025) or context compres-
sion (Ge et al. 2023); Dai et al. 2025) regulate memory access, the field still lacks systematic
framework for evaluating the ability of L.MS to selectively forget in context.
To address this gap, introduce ICF-Bench, the comprehensive benchmark for evaluating In- Context Forgetting (ICF), defined as the ability of LLMS to selectively forget interference infor- mation while retaining useful knowledge in context without parameter updates. The benchmark
Work done during an internship at Microsoft. Correspondence: qianyuli@stt.pkuu.ddu
Work done during full-time employment Microsoft. Correspondence: caciaaanyyngmmccosoofffoo

---

Published as a conference paper at ICLR 2026
Instructional Forgetting
Subtask Revision
Dynamic Preference
Tasks on this sentence: (1)
My friend he likes blue.
Rewrite; (2) Count words; (3) Judge emotion.
Tell me the styles Hua and Keigo Higashino.
Sentence: like writing code.
Got it, your friend Tom likes blue.
Both excellent novelists...
like coding, 4, Positive,
Please forget friend's favorite color.
Change (2) to letter "I"
I don't 't like Keigo Higashino.
Okay, will ft.get
it, updated.
Ok,
Please give me the updated answer.
Recommend some novels may like.
What color does Tom like?
Since you dislike Keigo Higashino, I'll suggest Yu
like codingg.3, Positiiee. like coding, words and letters 'f,, Posiitie. X
know. Tom likes blue. X
Hua...
I'll suggest Yu Hua and Keigo Higashino... X
Figure I: Illustration of the three scenarios in ICF-Bench: instructional forgetting, subtask revision, and dynamic preference. Instructional forgetting evaluates the ability of models to follow explicit forgetting instructions. Subtask revision measures the adaptability of models partial updates in multi-task instructions. Dynamic preference examines the ability of models to follow users' dynam- ically updated preferences.
consists of 2k multi-turn dialogues with annotations, covering three key scenarios where in-context forgetting is crucial in practice (see Figure I instructional forgetting, subtask revision, and dy- namic preference. To ensure that evaluations focus on ICF, each sample is instantiated in two task forms: no forgetting interference (Noforget) and with forgetting interference (Forget). We further introduce the Selective Forgetting Retention Rate (SFRR) as the core evaluation metric to reflect the ICF ability of LLMs, which measures the proportion of originally correct responses that remain correct in the presence of forgetting interference.
We conduct extensive experiments state-of oftte-of-thearr LLMs, including eight open-source and three proprietary models. Our findings reveal several key insights. Although models perform well in No- forget task, their performance degrades significantly when forgetting interference is introduced. For example, GPT-5 achieves 88.75% accuracy without interference but drops to 64.83% with interfer- ence. Stronger memory capacity in Noforget task does not necessarily transfer into stronger ICF capacity, highlighting a fundamental asymmetry between memory capacity and ICF capacity. Fur- thermore, context length increases (from 0.5k to 30k), ICF ability shows different trends across scenarios. These results expose critical vulnerabilities of current LLMs, revealing that while they excel at memory retention, they struggle to balance their ability to memorize and ICF, This asymme- try poses risks to privacy, adaptability, and user autonomy, underscoring the urgent need for robust evaluation frameworks and methods that explicitly address in-context forgetting.
Our contributions are summarized as follows:
We introduce ICF-Bench, the first benchmark designed to evaluate in-context forgetting in LLMs, focusing on realistic dialogue settings.
We construct 2k annotated samples covering three scenarios, including instructional forgetting, subtask revision, and dynamic preference, and each instantiated in paired Noforget and Forget forms.
We introduce the Selective Forgetting Retention Rate (SFRR), and through extensive experiments, demonstrate that current L.MMs face significant challenges under forgetting interference. Strong
2

---

Published as a conference paper at ICLR 2026
memory capacity does not necessarily guarantee superior in in-context forgetting ability, and context length has different effects on this ability across scenarios.
2 RELATED WORK
Machine unlearning (MU) aims to remove the influence of specific training data from model, typically motivated by privacy, security, or regulatory compliance requirements, while avoiding the cost of full retraining (Zhang et al. 2023). A variety of approaches have been proposed, including gradient ascent (Kodge et al. (024), certified removal (Li et al. 2025a). and scalable methods based on subspace projection (Wang et al.) 2025b) or influence estimation (Li jet al. 2025b). However, MU fundamentally differs from our notion of In-Context Forgetting (ICF): MU modifies model parameters to erase knowledge acquired during training, whereas ICF concerns the model's 's ability to selectively discard information during inference without parameter updates, and does not entail the permanent removal of internal knowledge.
Machine Unlearning,
Long-context LLMs and Benchmarks. With the rapid increase in input length in state-of-the-an LLMs (e.g., GPT-4-128k, Claude-2/3 with 100k tokens), long-context management and retrieval have attracted increasing attention. Benchmarks such as LongBench (Bai et al.] (023)) Long- BenchV2 (Bai al., 2024), and Scrolls (Shaham al., (022) assess whether models can locate, retrieve, and leverage relevant content across extended contexts. However, these benchmarks im- plicitly assume that all historical context should be retained, and only evaluate what to remember but not what to forget. leaving the question of in-context forgetting underexplored (Takashiro et al., 2024
Wang & Sun. 2025).
In-context Unlearning and Context Management. A growing body of work has begun explor- ing forgetting mechanisms during inference, commonly framed in-context unlearning or dynamic memory control (Pawelczyk et al., 2023). Some approaches insert explicit forget instructions or dis- tracting content into prompts to encourage models to ignore prior information ęZhang et al., 2024). Other works focus on architectural innovations, such as the Forgetting Transformer (Lin et al., 2025
or employ contextual compression techniques (Ge et al. [0033 Dai et al. 2025) and attention rout- ing (Wang et al. 2024) to access historical tokens. While these efforts offer preliminary evidence that LLMS exhibit certain degrees of in-context forgetting, their evaluation protocols largely focus on forgetting static information or compressing context without forgetting, rather than model's ability to dynamically manage information (retain or forget) based on the evolving context.
Our Work. In contrast to prior research, ICF-Bench provides the first systematic and controlled benchmark dedicated to evaluating in-context forgetting. Rather than focusing solely on parameter- level unlearning long-context retrieval, our benchmark explicitly targets the ability of LLMs to selectively forget during inference, offering rigorous and reproducible evaluations across instruc- tional forgetting, subtask revision, and dynamic preference.
ICF-BENCH
3.1 PROBLEM DEFINITION
As shown in Figure 2 define the problem of evaluating ICF capacity in LLMs within multi- session conversation. A conversation is denoted by C = (w,,bi) ), (u2, b2),.... (um bm)}. where u; and b, represent the user's utterance and the model's response turn i. Each conversation can be divided into sessions S = {s1, ,2, 3k where each session 8, {[u,, b.),..., (u++e- bx+e)} forms coherent dialogue segment. During the interaction, users may provide memory information, denoted as M, which represents factual information, initial multi-task instruction, or preference- based context that the model is expected to retain. Subsequently, forgetting interference, denoted as F, may be introduced in the form of explicit forgetting instructions, subtasks revision instructions, or dynamically updated preferences that require the model to selectively forget previously provided information in context. At the end of the dialogue, query for evaluating, denoted as Q Umm is posed, and the model's final response R = bm is examined to determine whether in-context for- getting has been properly executed. Additionally, to simulate the complexity of real-world conver- sations, we randomly inserted Multi-session conversations into each piece of data, which involved multiple topics, denoted MC.
3

---

Published as a conference paper at ICLR 2026
Muiti-Session Conversation Setup
Conversation Flow ICF-Bench
Two Task Forms and Evaluation Protocols
friend is Tom, he likes blue.
Memory information
Memory information
LLM Response
Query for evaluating
Forgetting interference
Got it, your friend Tom likes blue.
Forgetting interference
No Forgetting Interference
With Forgetting Interference
Please forget friend's favorite color.
Multi-session Conversation
Evaluator
Okay, forget
Selective Forgetting Metrics
Query for evaluating
What color does Tom
like?
NoForget Accuracy
Forget Accuracy
LLM Response
know. Tom likes blue.
Selective Forgetting Retention (SFRR)
X
Overview of ICF-Bench. Key components from left right: Multi-Session Conver- sation Setup: Memory information, forgetting interference, multi-session conversation, evaluation query, and model response are integrated to form complex, realistic dialogues. 2) Conversation Flow in ICF-Bench: An example of conversation flow in ICF-Bench, where the color of cach turn indicates its role; 3) Two Task Forms and Evaluation Protocols: Instances are evaluated in No- Forget and Forget forms, with performance evaluated via NoForget Accuracy, Forget Accuracy, and SFRR.
Figure 2:
ICF-Bench instantiates this problem in three realistic scenarios. the Instructional Forgetting scenario, the user directly asks the model erase some information previously presented in mem- ory M, and the model must partially suppress M when answering the evaluation query Q. In the Subtask Revision scenario, initial multi-task instruction M is partially revised, requiring the model to discard outdated subtasks and following the updated requirement. In the Dynamic Prefer- scenario, the user's initial preference M is later modified by an updated preference F. and the model must prioritize the most recent preference when generating R. Collectively, these scenarios capture diverse forgetting challenges encountered in realistic conversational settings.
To assess forgetting behavior, we compare two complementary task forms. In the NoForget task, the conversation contains memory information M without forgetting interference F, and the model is expected to retain and utilize M. In the Forget task, forgetting interference F is inserted, and the model must selectively ignore outdated or irrelevant content in order to accurately respond to Q. The evaluator reports three metrics: NoForget Accuracy and Forget Accuracy, which measure the model's 's performance on the respective tasks, and the Selective Forgetting Retention Rate (SFRR), which quantifies the model's ability to preserve correct responses in the presence of forgetting inter- ference.
3.2
DATASET COMPOSITION
3.2.1
CONSTRUCTION
Building upon established high-quality datasets, ICF-Bench synthesizes data that cover three rep- resentative realistic scenarios of in-context forgetting. In the Instructional Forgetting scenario, we extract natural multi-turn buman-LLM conversations from the ChatAlpaca dataset (Bian et al.) 2023), which contains 20k high-quality dialogue samples. For each conversation, we employ LLMs to insert explicit forgetting instructions F (c.g., "please forget E"), where E denotes the information to be discarded in M. Subsequently, we generate an evaluation query Q designed to test whether E has been successfully suppressed in the model's final response R. In the Subtask Revision scenario, leverage the Follow Bench dataset (Tiang al. 2023), which provides 1.85k multi-task instruc-
4

---

Published as a conference paper at ICLR 2026
Average Dialogue Turns
Average Token Length
Sampla
NoFonget
NoForget
2000
1500
1500
Samples
1000
1000
Overall
(a) Turns distribution.
(b) Tokens distribution.
(c) Samples distribution.
Figure 3: Statistics of ICF-Bench across NoForget and Forget forms in three scenarios: Instructional Forgetting (IF), Subtask Revision (SR), and Dynamic Preference (DP), which shows the distribu- tions of (a) dialogue turns, (b) token lengths, and (c) sample counts.
tions with overlapping subtasks. Each instruction is decomposed into sequence of subtasks, and revised instructions are generated by selectively modifying, adding, or removing subtasks. The ini- tial multi-task instructions constitute the memory information M, and generate subtask revision instructions as F' which require the model to forget outdated subtasks while retaining the updated subtasks. To better approximate real-world dialogue complexity, we interleave unrelated conversa- tional turns sampled from LMSYS-Chat-IM (Zheng et al. 2023) as MC, requiring the model to respond accurately Q by prioritizing updated subtasks over outdated ones. In the Dynamic Pref- erence scenario, we utilize the PrefEval dataset (Zhao et al.) 2025). which captures user preferences expressed direct, multiple-choice, and dialogue formats. We simulate preference shifts by con- verting alternative options into updated preferences F, while the initial preference is treated as M. Conversations are constructed such that a user first expresses preference M and later switches to F', then irrelevant turns from LMSYS MMSSS-Chat-IN are inserted as MC. The final query Q requires the model to suppress the outdated preference and generate a response consistent with the updated preference F. The detailed construction prompts for each scenario presented in Appendix A
3.2.2 STATISTICS
ICF-Bench comprises 2k annotated multi-turn dialogues distributed across the three representative scenarios. Specifically, Instructional Forgetting evaluates whether models can faithfully follow explicit instructions to erase previously provided information, which is important for enabling users to actively control LLMs in managing context; Subtask Revision examines the adaptability of mod- els to updated multi-task instructions, requiring them to discard outdated subtasks while retaining the unrevised ones; and Dynamic Preference assesses whether models can align with newly spec- ified user preferences while suppressing conflicting earlier preferences. These scenarios provide complementary perspectives the challenges of in-context forgetting in conversational contexts.
reuurrs dataset-leve statistics, including the average number of dialogue turns, token length, NoForget and Forget forms in three scenarios. The Figure demonstrates
and sample counts that forgetting interference generally increases both dialogue length and token usage, thereby re- flecting the additional cognitive load imposed models. The dataset thus offers a balanced yet challenging testbed for systematically evaluating in-context forgetting across diverse interaction pat- terns.
3.3 TWO TASK FORMS: NOFORGET AND FORGET
Each dialogue instance instantiated two task forms. The NoForget form provides the context without forgetting interference F, thereby evaluating the model's is basic memory retention capability. The Forget form inserts forgetting interference F into the context, requiring the model to selectively discard outdated, irrelevant, conflicting content. These two task forms not only reveal model performance in scenarios without and with forgetting interference, but also disentangle in-context forgetting from inherent memory limitations of LLMs, where models may randomly fail to retain information.
In different evaluation scenarios, the criteria for the two task forms different. In the Instructional Forgetting scenario, the NoForget task requires the model to correctly answer the query Q based retained memory M, while the Forget task requires the model to explicitly forget the specified

---

Published as a conference paper at ICLR 2026
information due interference F. In the Subtask Revision scenario, the NoForget task requires faithfully completing all subtasks specified in M, while the Forget task requires performing both the unrevised subtasks in M and the revised subtask introduced by F correctly. In the Dynamic Preference scenario, the NoForget task requires adherence to the original user preference M, while the Forget task requires alignment with the updated preference.
3.4 IN-CONTEXT FORGETTING METRICS
To comprehensively evaluate model behavior under in-context forgetting, we report three metrics: NoForget Accuracy, Forget Accuracy, and the Selective Forgetting Retention Rate (SFRR). Below we give precise definitions and brief interpretations for each metric.
NoForget Accuracy(NA). Let Rd NoFoget be the model's response to query Q: in NoForget task. Define an indicator function:
{
if the response is judged correct,
C(-)
otherwise.
The NoForget accuracy measures the model's baseline capability in memory retention and instruc- tion following without forgetting interference:
N
AcCNoForget
Σ
(R
NoForget)
(2)
N
where N is the total number of evaluation samples.
Forget Accuracy(FA). Let R. Forget be the model's 's response to the same query Qi when forgetting interference present. The Forget accuracy is defined analogously:
(RR (orgett)
ACCForget
N
Selective Forgetting Retention Rate (SFRR). While the two accuracies above quantify absolute performance, they do not isolate how forgetting interference affects examples that were originally handled correctly. To measure robustness specifically on those originally correct cases, We define SFRR the fraction of originally-corred responses that remain correct when forgetting interference is present:
Σn 1f C(R, NoFoget 1 A Forget) -1}
SFRR
(4)
Σ 1f C(TR
A higher SFRR indicates stronger robustness to forgetting interference and more effective in-context forgetting. We provide a detailed explanation of why we chose SFRR the core metric for evaluat- ing in-context forgetting in Appendix
3.5 CONTEXT LENGTH VARIATION
Since real-world applications often involve extended conversational histories, ICF-Bench further evaluates in-context forgetting ability under varying context lengths. By interleaving irrelevant turns from LMSYS-Chat-IM IM into the preceding context, we control the length of each instance in ICF-Bench. Given conversation Conv with length L, we define L as the total token length of all dialogue turns before the evaluation query Q and construct truncated or extended contexts ConuL {u1, bj...., uz, by) } for L E {0.5k, 1/, 3k, 6k, 10k. 15k, 30k). For each context length L. we compute NoForget Accuracy, Forget Accuracy, and SFRR. This analysis reveals how in- context forgetting ability evolves with increasing context length.

---

Published as a conference paper at ICLR 2026
NA
FA
SFRR
80
Accuracy (%)
60
40
20
Mistral-78
Llama3-8B
Owen2 5-7B
Mixtral-6x78
5-142
Gemma3-278
Llama3-70B
Owen3 Swen3--335--B5 20103--33B--228 A228
GPT-O3mini
GPT-40
GPT-8
Figure 4: The average results of NoForget Accuracy (NA), Forget Accuracy (FA), and Selective
Forgetting Retention Rate (SFRR) across the three scenarios (%).
4 RESULTS
4.1 EVALUATION SETUP
We evaluate a diverse set of state-of-the-art LLMs, including open-source models (Llama-3, Mis- tral, Gemma3, Qwen2.5, Qwen3) and proprictary systems (GPT-O3mini, GPT-4o, GPT-5), assess their in-context forgetting (ICF) capabilities across varying architectures and scales. All responses are judged by GPT-O4mini, prompted with detailed, task-specific rubries determine whether the model suppresses outdated or conflicting information while preserving relevant context. This automated evaluation covers both NoForget and Forget task forms of each ICF-Bench sample, en- abling direct comparison with and without forgetting interference. Detailed evaluation prompts for each scenario appear Appendix A We report three metries: NoForget Accuracy and Forget Accuracy measure absolute performance, while the Selective Forgetting Retention Rate (SFRR) quantifies robustness by computing the proportion of originally correct responses that remain cor- rect under interference. SFRR is central baseline memory performance. To examine length variation, we test models across context lengths from 0.5k 30k tokens, with irrelevant turns from LMSYS-Chat multi-session dialogue. All generations use greedy decoding (T' = 0) for reproducibility. This setup ensures rigorous, standardized assessment of ICF across models and conditions. Several detailed case analyses for each scenario are provided in Appendix C
analysis, isolating in-context forgetting ability from
inserted to simulate realistic
4.2 PERFORMANCE ANALYSIS OF NOFORGET AND FORGET TASKS
The experimental results ICF-Bench reveal stark contrast between model performance in the NoForget and Forget forms, showing that models perform well without forgetting interference but struggle significantly when interference is present. Figure 4 4 illustrates the average results across the three scenarios, with detailed values listed Table 11 Across all models, performance remains robust when no forgetting interference is present, which demonstrats strong baseline memory re- tention and instruction-followinn capabilities. For instance, GPT-5 achieves a NoForget Accuracy (NA) of 88%7,, while even smaller open-source models like Mistral-7B attain 60.18% However, the introduction of forgetting interference leads to dramatic degradation in performance, with Forget Accuracy (FA) dropping sharply across the board. Notably, GPT-5's FA falls to 64.83%, represent- ing a 23.92 percentage point (pp) drop, while Llama3-8B suffers a catastrophic decline from 69.85% to just 22.78% (47.07 pp drop).
A closer inspection of Table D that each scenario presents a distinct set of challenges across the three forgetting settings. Instructional Forgetting, even high-performing models fail to re- liably follow explicit "forget" commands: GPT-5 correctly suppresses prior information in only 58.85% of cases despite a 97.18% success rate without interference, while Qwen2.5-72 7B nearly col- lapses from 87.02% to 0.50%. This suggests that the model often ignores explicit interference F. In Subtask Revision, models show moderate resilience, with Gemma3- 27B maintaining 43.94% FA (from 58.31 % NA), indicating some capacity to adapt updated task structures, though confusion
7

---

Published as a conference paper at ICLR 2026
Table 1: NoForget Accuracy (NA), Forget Accuracy (FA) and Selective Forgetting Retention
Rate (SFRR) results across different evaluation settings (%)
Instructional Forgetting
Subtask Revision
Dynamic Preference
Model
NA
FA
SFRR
NA
FA
SFRR
NA
FA
SFRR
Mistral-7B
77.67
22.03
17.62
50.07
39.58
69.50
52.81
23.09
39.86
Llama3-8B
90.04
4.23
3.46
58.03
45.37
71.14
61.48
18.75
25.93
Qwen2.5-7B
87.02
0.50
0.23
62.82
50.85
74.05
60.84
35.46
51.99
Mixtral-8x7B
86.42
14.69
14.09
60.28
50.56
73.86
56.12
33.55
49.77
Qwen2.5-14B
91.95
1.01
0.88
65.07
48.31
66.81
64.16
38.39
51.89
Gemma3-27B
93.16
60.16
61.66
58.31
43.94
71.79
71.81
40.82
52.93
Llama3-70B
93.96
14.49
13.60
60.85
50.28
79.63
74.49
36.22
46.58
Qwen3-255B-A22B
95.98
21.43
20.96
61.69
47.07
73.92
80.48
48.21
57.53
GPT-O3mint
96.48
'63088
62.88
84.51
66.08
770.93
768.62
44.90
58.18
GPT-4o
95.77
52.52
52.52
81,41
62.31
74.45
86.10
44.90
50.07
GPT-5
97.18
58.85
58.80
81.69
63.18
73.69
87.37
72.45
79.12
between old and new subtasks persists. In Dynamic Preference, GPT-5 stands out with a relatively strong FA of 72.45% (vs. 87.37% NA), significantly outperforming other models such as GPT-4o (44.90%) and Qwen3-235B-A2T evolving user preferences. These observations indicate that while LLMs' memory retention ability strong, their performance degrades sharply in scenarios requiring in-context forgetting. other words, current models struggle to suppress disregard outdated conflicting information once in- terference is inserted into the context, highlighting the risks of performance deterioration, privacy leakage, and reduced adaptability in interactive systems.
(48.21%) 1,), suggesting an improved ability track and prioritize
4.3 PERFORMANCE ANALYSIS OF MEMORY AND IN-CONTEXT FORGETTING
The results in Figure 4 reveal asymmetry between memory retention and in-context forgetting in modern LLMs with detailed values listed in Table I1] While NoForget Accuracy (NA) consistently improves with model size(ranging from 60.18% for Mistral-7B to 88.75% for GPT-5), this supe- rior memory capacity does not translate into stronger in-context forgetting ability as measured by the Selective Forgetting Retention Rate (SFRR). For instance, GPT-5 achieves the highest SFRR of 70.54%, markedly outperforming GPT-4o (59.01 ()) and GPT-O3Mini (64.00%) yet even this top- performing model fails retain correct responses in about 30% of cases where interference is troduced. Notably, among open-source models, Gemma3-27B stands out with an SFRR of 62. 13%. surpassing larger models such as Llama3-70B (46.60%) and Qwen3-235B- having lower NA. This disconnect underscores that strong memory retention, the ability call and apply prior context, not a sufficient condition for effective in-context forgetting. Instead, these findings suggest that in-context forgetting requires distinct cognitive mechanisms, potentially involving dynamic attention reallocation, conflict resolution between competing contextual signals, and robust suppression of outdated information. which are not automatically enhanced through scal- ing alone. The persistent gap between NA and SFRR across all models indicates that current LLMs remain vulnerable to forgetting interference, posing critical challenges for applications demanding privacy preservation, adaptive reasoning, and user-controlled context management.
(50.80%) despite
As shown in Table [ we further analyze the relationship between memory retention and in-context forgetting in LLMS across the three forgetting scenarios. In Instructional Forgetting, even models with near-perfect NA (e.g., GPT-5 at 97.18%) exhibit drastically reduced SFRR (58.80%). indicat- ing that explicit "forget" commands are frequently overridden by deeply anchored initial context. This is especially pronounced in open-source models: Qwen2.5-7B drops from 87.02% NA to an SFRR of only 0.23%. In Subtask Revision, models demonstrate relatively stronger SFRR, with Gemma3-22 2-B achieving 71.79% and Llama3-70B reaching 79.63%, despite moderate NA scores. This suggests that partial task updates may be easier to integrate when structured explicit visions, possibly due to clearer semantic boundaries between subtasks. In Dynamic Preference, where GPT-5 achieves remarkable SFRR of 79.12%, significantly outperforming GPT-40 (50.07%)
8

---

20000 Published as a conference paper at ICLR 2026
Instructional Fargetting
Subtask Revision
Dynamic Preference
100
NoForget (NA)
Forget (FA)
80
SFRIt
60
Score
40
NoForget
NoForget Forget (FA)
Forget (FA)
SFRR
SFRR
0
10000
30000
10000
20000
30000
10000
20000
30000
Context length
Context length
Context length
Figure 5: Performance of GPT-4o under different context lengths across the three in-context forget-
ting scenarios.
and all other models. This indicates a qualitative improvement in tracking and prioritizing evolving user intents, likely due enhanced preference modeling in training.
4.4 IMPACT OF CONTEXT LENGTH
We conducted experiments across different models, observed similar trends, and consequently se- lected GPT-4o as the representative model (with additional results and analysis across models pre- sented in Appendix D). Across varying context lengths, GPT-4o exhibits a consistent decline in memory retention (NoForget) across all three scenarios(see Figure 5) with Instructional Forgetting dropping from 96.53% to 77.89% and Dynamic Preference from 90.36% to 59.42%. Forget per- formance also deteriorates correspondingly, leading steady decrease in SFRR, most notably in Dynamic Preference, where falls from 52.46% to 22.87%. contrast, Subtask Revision shows different trend: its SFRR rises from 71.76% to 81.50% context length increases. Overall, al- though long contexts generally exacerbate the challenges of in-context forgetting, they can mitigate the influence of early context in revision tasks.
4.5 IMPACT OF PROMPT ENGINEERING
To investigate whether explicit forgetting reminders in prompts can enhance in-context forgetting, conducted set of preliminary experiments comparing two prompting strategies: NoForget prompt, which simply instructs model follow the task, and Forget prompt, which addition- ally includes explicit reminder follow forgetting instructions, outdated subtasks, or previous preferences depending on the scenario. Detailed prompts for these two strategies appear in Ap- pendix E
Table 2: In two prompting Strategies(NoForge prompt and Forget prompt), NoForget Accuracy (NA), Forget Accuracy (FA) and Selective Forgetting Retention Rate (SFRR) results across dif- ferent evaluation settings (%).
Instructional Forgetting
Subtask Revision
Dynamic Preference
Model
NA
FA
SFRR
NA
FA
SFRR
NA
FA
SFRR
NoForget Prompt
GPT-O3Mini
96.48
63.08
62.88
84.51
66.08
70.93
68.62
44.90
58.18
GPT-4o
95.77
52.52
52.52
81.41
62.31
74.45
86.10
44.90
50.07
GPT-5
97.18
58.85
58.80
81.69
63.18
73.69
87.37
72.45
79.12
Forget Prompt GPT-O3Mini
93.84
78,94
76.30
84.37
70.15
72.67
79.53
59.93
62.56
GPT-4o 93.54
70.09
71.76
80.50
64.41
75.84
87.43
66.08
66.56
GPT-5
95.18
74.38
74.62
81.07
65.15
74.15
88.89
76.02
76.64

---

Published as a conference paper at ICLR 2026
Table 2 shows the results across different settings. Across all models and scenarios, adding explicit forgetting reminders consistently strengthens forgetting behavior, yielding improvements in FA and SFRR. A small decrease in NA is observed in the Instructional Forgetting scenario, suggesting trade-off between memory and in-context forgetting. Overall, these findings indicate that prompt engineering is promising and practical approach enhance ICF abilities in current LLMs.
4.6 QUANTITATIVE RELATIONSHIPS AMONG EVALUATION METRICS
To better reflect the interactions among our three evaluation metrics, compute Pearson correla- tions for NoForget Accuracy (NA), Forget Accuracy (FA), and the Selective Forgetting Retention Rate (SFRR) across the three scenarios, as well as their average values, using all models summa- rized in TO
These correlations reveal several key quantitative trends. Most prominently, SFRR aligns more closely with FA than with NA across all scenarios (average correlation 0.982 vs. 0.764). This sup- ports that SFRR primarily relates to robustness under interference rather than memory ability. More detailed data and analysis are reported Appendix F
4.7 CONSISTENCY WITH HUMAN EVALUATION
Due to potential biases introduced by using an LLM automated evaluator, examine the consistency between our evaluator and human judgments through a study conducted by three inde- pendent annotators with NLP expertise. Using a random sample of 150 instances (50 per scenario), we evaluate both the NoForget and Forget task forms across all three ICF-Bench scenarios.
As shown in Table 3] the automated evaluator (GPT-04mini) aligns closely with human consensus. Agreement remains high in all NoForget settings (exceeding 89. 89.33%) and remains strong in Forget tasks 1i..00095.33% with slightly lower alignment for Subtask Revision. To further assess the robustness of the human study, we compute Cohen's 's k (Cohen 1960) for each annotator and Fleiss' k Fleiss 1971 across all three annotators. Detailed results and analysis are provided in Appendix
Table 3: Average agreement rates (%) between automated evaluation (GPT-O4mini) and human consensus across 50 samples per scenario.
Scenario
NoForget Agreement
Forget Agreement
Instructional Forgetting
89.33
93.33
Subtask Revision Dynamic Preference
92.00
82.00
96.67
95.33
CONCLUSION
Broader Impact. We introduce ICF-Bench, the first benchmark evaluate in-context forgetting (ICF) in LLMs across instructional forgetting, subtask revision, and dynamic preference scenarios. Using paired NoForget and Forget tasks along with three metrics (NoForget Accuracy, Forget Accu- racy, and SFRR), we reveal a critical asymmetry: strong memory does not mean effective in-context forgetting. Performance degrades notably under forgetting interference and exhibits different trends across scenarios as context length increases. These findings expose key limitations in adaptability, privacy, and user control, urging future work on mechanisms for robust in-context forgetting.
Limitations and future work. ICF-Bench currently focuses on evaluating in-context forgetting within dialogue-basse based scenarios, which limits the generalizability of our findings. Future work will extend the benchmark to tasks involving multi-document reasoning and tool-augmented pipelines. Another limitation is that our study analyzes forgetting performance without systematically ad- dressing methods for improvement, although we experimented with simple prompt-engineering approaches and discussed their implications. As future directions, we plan to investigate broader wrompt-engineerinn strategies, attention-leve modulation, representation disentanglement, and chitectural forgetting gates to enhance the controllability and effectiveness in context forgetting in large language models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Yushi Bai, Xin Lx, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. /52044 2024.
Ning Bian, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, and Ben He. Chatalpaca: A multi-tur turn dia- logue corpus based on alpaca instructions, https: wttp:/uw.ooccccccaacccaaaaa htt.:///ww...oo.ccacaaaaaaa..a.a 2023.
com
Alberto Blanco-Justicia, Najeeb Jebreel, Benet Manzanares-Salor, David Sánchez, Josep Domingo- Ferrer, Guillem Collell, and Kuan Eeik Tan. Digital forgetting large language models: A survey of unlearning methods. Artificial Intelligence Review, 58(3):90, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurlPS, 2020.
Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web, 27(4):42, 2024.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37 46, 1960.
Yuhong Dai, Jianxun I Lian, Yitian Huang, Wei Zhang, Mingyang Zhou, Mingqi Wu, Xing Xie, and Hao Liao. Pretraining context compressor for large language models with embedding- mem- ory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 28715-28733 2025.
Badhan Chandra Das, Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey, ACM Computing Surveys, 57(6):1-39, 2025.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.069.. 2023.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang. Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint arXiv:2310.20411 2023.
Sangamesh Kodge, Gobinda Saha, and Kaushik Roy. Deep unlearning: Fast and efficient gradient- free class forgetting. Transactions on Machine Learning Research, 2024.
Na Li. Chunyi Zhou, Yansong Gao, Hui Chen, Zhi Zhang, Boyu Kuang, and Anmin Fu. Machine unlearning: Taxonomy, metrics, applications, challenges, and prospects. IEEE Transactions on Neural Networks and Learning Systems, 2025a.
Xunkai Li, Bowen Fan, Zhengyu Wu, Zhiyu Li, Rong-Hua Li, and Guoren Wang. Toward scalable graph unlearning: A node influence maximization based approach. arXiv preprint MrXiv:2501.11823 2025b.
Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate, arXiv preprint 13 2025.

---

Published as a conference paper at ICLR 2026
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.0757,. 2023.
Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, and Marco Del Tredici. From tools to teammates: Evaluating Ilms in multi-session coding interactions. arXiv preprint (tt::2502..1799) 2025.
Md Rafi Ur Rashid, Jing Liu, Toshiaki Koike-Akino, Ye Wang, and Shagufta Mehnaz. Forget to flourish: Leveraging machine-unlaaree on pretrained language models for privacy leakage. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20139-20147, 2025.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. Scrolls: Standardized comparison over long language se- quences. arXiv preprint arXiv:2201.084
2022.
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. ACM Computing Surveys, 2024.
Shota Takashiro, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, and Yutaka Mat- suo. Answer when needed, forget when not: Language models pretend to forget via in-context knowledge unlearning. arXiv preprint arXiv:2410.00388 2024.
Chupei Wang and Jiaqiu Vince Sun. Unable to forget: Proactive Interference reveals working mem- ory limits in lims beyond context length. arXiv preprint arXiv:2506.08188 2025.
Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forget- ting: Advancing machine unlearming techniques and evaluation in language models. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 843-851, 2025a.
Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In- context former: Lightning-fast compressing context for large language model. arXiv preprint arXiv:2406. 13618, 2024.
Xu Wang, Zihao Li, Benyou Wang, Yan Hu, and Difan Zou. Model unleamiin via sparse autoen- coder subspace guided projections. arXiv preprint arXiv:2505.24428, 2025b.
Haibo Zhang, Toru Nakamura, Takamasa Isohara, and Kouichi Sakurai. A review on machine unlearning. SN Computer Science, 4(4):337, 2023.
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungehul Kim, et al. Personalization of large language mod- els: A survey. arXiv preprint S.Ni:24111.0002 2024.
Siyan Zhao, Mingyi Hong. Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do Ilms recog- nize your preferences? evaluating personalized preference following in llms. arXiv preprint 20X2:22502.09597,, 2025.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zhuohan Li, Zi Lin, Eric P Xing, et al. Lmsys-chat- Im: A large-scale real-world Ilm conversation dataset. arXiv preprint arXiv:2309. 2023.
A CONSTRUCTION AND EVALUATION PROMPTS
A. INSTRCUTIONAL FORGETTING
Based on the ChatAlpaca dataset, which contains 20k high-quality dialogue samples. For each conversation, we employ LLLMs to insert explicit forgetting instructions (e.g., "please forget M"), where M denotes memory information to be discarded.
12

---

Published as a conference paper at ICLR 2026
Construction prompts
System Prompt: You are a dialogue design expert tasked with creating realistic user Forger instructions and corresponding test questions.
Below dialogue history between a user and an assistant: (history_stt)
Your tasks:
1. From the dialogue history, select one piece of user eser-provided information (e.g.. iden- tity, preferences, task goals, or other contextual details) and write Forget instruc- tion that asks the assistant to forget it.
2. Write test question that explicitly or implicitly refers the forgotten information. The question must be impossible to answer correctly if the assistant has truly for- gotten To strengthen the test, you may use cues such as "before," "earlier," or "you mentioned."
Output format:
Please return a JSON object with exactly two fields, and no extra text. Examples (do not cupy, just imitate style):
"forget_instructionn" "Let forget Task A.", "test_qqeestion"; "What you remember about Task mentioned before?"
Cnorget_innsruutiee "Please forget that I'm from China",
"test_question": "Where am from?"
"forget_instruction"" "Oh, Im not actually a doctor, please ignore it", "test_question": "Are you aware I'm doctor?"
inorget_instrruction" "Forget the second point you just + told me about how to learn a language", "test_question": "Tell the second point you just mentioned about how to learn a language before"
In the Instructional Forgetting scenario, the NoForget task requires the model to correctly answer the query based on retained memory, while the Forget task requires the model to explicitly indicate unawareness of the requested information due interference.
Evaluation Prompt(Noi (ooggge
You expert evaluator specializing in conversational memory and forgetting. Your task is to judge whether the assistant has correctly followed the user's most recent instruction, based the dialogue context and the assistant's reply. Evaluation Guidelines:
System Prompt:
Carefully examine the full conversation history, including the user's final instruc- tion.
2. Determine whether the assistant's 's last response is consistent with and correctly executes the user's instruction.
3. Answer strictly with "Yes" " or "No".
13

---

Published as a conference paper at ICLR 2026
4. After the binary judgment, provide concise explanation justifying your decision. The explanation should explicitly reference the instruction and the assistant's 's reply.
Conversation History: (conversation history) User Query: ftest query)
Assistant Reply: rassls
Evaluation Trompt(Forget)
You expert evaluator specializing in conversational memory and forgetting. Your task is to assess whether the assistant has correctly complied with a user- issued Forger instruction.
System Prompt:
Evaluation Guidelines:
Carefully review the full dialogue context, including the explicit Forget instruction provided by the user.
2. Judge whether the assistant's ' final response respects the instruction:
If the assistant avoids recalling or using the forgotten information, answer strictly with "Yes".
the assistant recalls, repeats, or directly answers with the information that should have been forgotten, answer strictly with "No".
3. After the binary judgment, provide concise justification explicitly referencing the Forget instruction, the test query, and the assistant's last reply.
Conversation History: (conversation history)
Forget Instruction: iffrrgt_inssruuction Test Query: [test_query]
Assistant Reply: {
A.2 SUBTASK REVISION
Based on the FollowBench dataset, which provides 1.85k multi-task instructions with overlapping subtasks. Each instruction is decomposed into a sequence of subtasks, and revised instructions are generated by selectively modifying, adding, or removing subtasks. The initial multi-task instructions constitute the forgetting interference F, which requires the model to ignore outdated subtasks while retaining the updated subtasks of revised instructions Q. To better approximate real-world dialogue complexity, we interleave unrelated conversational turns sampled from LMSYS- 1. as MC, requiring the model to respond accurately to Q by prioritizing updated subtasks over outdated ones.
Construction prompts(Tane Task Modification)
System Prompt:
You are an instruction editor tasked with strategically modifying a com- plex, multi-subtask instruction induce significantly different outputs. Below is the original instruction (which implicitly contains multiple subtasks):
Your Task:
1. Modify part of the original instruction - for example, by altering the content of one subtask removing a subtask entirely such that the resulting output from an Al system would be meaningfully and observably different.
2. Preserve the original instruction's tone, structure, and formatting as much as possi- ble. Do NOT explicitly list or expose individual subtasks in your output.
3. Briefly explain why your modification would lead to a substantially different result concise sentences sufficient.
Output Format: Return your response as a JSON object with exactly two fields, using English only:
14

---

Published as a conference paper at ICLR 2026
"modi (moddffeedinnttiieee inteediiisttttoon "Your revised instruction here, preserving original style", "description"; ["Reason 1", "Reason 2 (optional) "]
Important: Do not include any additional text, commentary, or formatting outside the JSON object.
Construction prompts(Subtask Decomposition)
System Prompt:
You language task decomposition expert skilled at precisely extract- ing subtasks from instructions. You will receive an original instruction and a modified instruction. Please complete the following:
1. Decompose both instructions into their constituent subtasks.
2. Classify each subtask into one of three categories:
Subtasks common to both instructions;
Subtasks unique to the original instruction;
Subtasks unique to the modified instruction.
3. All subtasks must be phrased in the same language as the original instructions.
4. Your output must be a structured JSON object no additional text, explanations, formatting allowed.
Input Data:
Original Instruction: (item (i "oldd'osttttsuttionnn
Modified Instruction: { (item 'new' neww_instructal
Required Output Format:
common "aammon_uubtsss""" ("subtask 1", "subtask 2", ...,, "origina Berigiiaa__seccal__ee _es_cieallssaaaaassssss ["subtask 1", "subtask 2",
"modi fied spwciallsubtaatttee ["subtask 1", "subtask 2",
In the Subtask Revision scenario, the NoForget task requires faithfully completing all subtasks specified in Q while the Forget task requires performing all subtasks in correctly but discarding the special subtask introduced by F.
Evaluation Prompt( (NoFore))))g
System Prompt: You expert evaluator of language task compliance. Your task is to rigorously assess whether the provided answer to the modified instruction correctly fulfills the required subtasks categorized common, original-specifer and modified-speciffe.
=== Evaluation Criteria
For each category below, return tuu ONLY if the answer FULLY satisties ALL subtasks in that group. Partial or ambiguous compliance must be treated as false. If category contains no subtasks, default true.
"same_ssuie "ssae_subbas"" a..uugeee Does the answer satisfy ALL subtasks that are COMMON to both the original and modified instructions?
2. wtt.giinn.subasskkk.ak
Does the answer satisfy ALL subtasks that are SPECIFIC to the ORIGINAL instruction? (If none exist, return False.)
15

---

Published as a conference paper at ICLR 2026
3. "modi fiedfiied_sste wtteeessttasss.uddeee... Does the satisfy ALL subtasks that SPECIFIC to the MODIFIED instruction? (If none exist, return true.)
Input Data ==
Original Instruction: {item[' old instructionnn }}
Modified Instruction: (item (' new instruction']
Answer to Modified Instruction: (answer)
Common Subtasks: {item (' common.subtaskks]]
Original-Speciffe Subtasks: {item[ original special subtaskss ]}
Modified-Speeifee Subtasks: {item] modiified.sppcceeeee suubbasssssssssssk""
Output Format
Return strict JSON object with the following structure. Do NOT add any extra text, explanation, or formatting.
"same__subtask__udee"" <true/false>, Saliginnnssubtsss__nne
<true/false>,
"modi weaiiffidd_dsbsss wwwtssuutsshtt.. _uddee""J <true/false>, "comments": "Optional brief justification for any "false" judgments. Keep it concise (12 sentences max)."
Evaluation Prompt (Forget)
You are expert evaluator of language task compliance. Your task is to rigorously assess whether both the original and modified instruction's 's answers correctly ful- fill the required subtasks - categorized common, original -igginal-ppecffie, and modiiied-speccifee === Evaluation Criteria
System Prompt:
For each dimension below, return true ONLY if the specified condition is fully and strictly met. Partial, ambiguous, or incomplete compliance must be treated as fa se. subtask category is empty, default true.
1.
waame_subteen s_attaakk_uue"" Do both answers (to the original and modified in- structions) satisfy ALL common subtasks? Return true only if both answers fully comply.
2. "origina..subtask..uuu
Does the modified instruction's answer sat- isfy ALL original-specific subtasks? Return true only if fully satisfied. If no original-specific subtasks exist, return true.
Does the modified instruction's answer sat- 3. "modified.subbask subtasks? Return true only if fully satisfied. If isfy ALL modified-specifie
modified-specifie subtasks exist, return true.
Input Data ===
Original Instruction: {item( (' old.instruction }}
Modified Instruction: {item] (' new new.instructionn }}
Answer to Original Instruction: {original.answer}
Answer to Modified Instruction: (modiifie..nsswrr
Common Subtasks: {item] [' common_subtasks']]
Original-Specific Subtasks: (item( I'temm'oriinn www.gginaa..pppcclllltteeeeeeee. }
Modified-Speeiffe Subtasks: {item] [' modi mmddiiiiedd.epcill...tttees }}
Output Format
Return a strict JSON object with the following structure. explanation, or formatting.
Do NOT add any extra text,
16

---

Published as a conference paper at ICLR 2026
"same_subbtsss__uddee" <true/false>,
Staiiiinnl_subaass___t <true/false>,
<true/false>, "commenss: "Optional brief justification for any "false" judgments. Keep it concise (12 sentences max)
A.3 DYNAMIC PREFERENCE
In the Dynamic Preference scenario, we utilize the PrefEval dataset, which captures user prefer- ences expressed in direct, multiple-choice, and dialogue formats. We simulate preference shifts by converting alternative options into updated preferences F, while the imitial preference is treated as M. Conversations constructed such that user first expresses preference M and later switches to F, then irrelevant turns from LMSYS-Chal 1M are inserted as MC. The final query Q requires the model suppress the outdated preference M and generate response R consistent with the updated preference
Construction prompt
You are an expert in linguistic style transfer, skilled at rewriting user preference statements while strictly preserving the original sentence structure, tone, and first-person perspective.
System Prompt:
Task Description:
The user previously stated a clear preference, but has now selected different option. Your task is to: - Generate a new preference statement that mirrors the original in struc- ture, tone, and first-person voice replacing only the content with the new choice. Input Data:
Original Preference: [im[' preference' }}
User's New Selection: {selected op}
Output Requirements:
1. Use first-person pronouns only (e.g., "I", "I like" "I want") - no third-person or passive voice.
2. Preserve the original sentence structure exactly (e.g., original is "I hate...", output must also start with "I hate...").
3. Match the original tone and style (e.g., if casual, stay casual; if concise, do not expand).
4. Reffect (selected.op) precisely additions, omissions, interpretations, or irrelevant content.
5. Output ONLY one natural, fluent preference sentence - no prefixes, suffixes, numbering, quotation marks, explanations, or line breaks.
6. Avoid reasoning words like "because", "therefore", "so" state the preference directly.
7. Avoid vague hesitant language like "maybe", "perhaps", "I tend to" prefer- ences must be clear and decisive.
8. Control length strictly: +5 characters from the original (e.g., original is 15 chars, output must be 10-20 chars).
Now generate the rewritten preference statement meeting all criteria above:
In the Dynamic Preference scenario, the NoForget task requires adherence to the original user preference, while the Forget task demands alignment with the updated preference F.
17

---

Published as a conference paper at ICLR 2026
Preference Consistency Evaluation Prompt
You are strict Preference Consistency Evaluator. Your sole task to judge whether the model's response strictly complies with the user's explicitly stated pref- erence.
System Prompt:
Input Data:
User Preference: (preference)
This is the user's explicitly declared subjective inclination, constraint, value, or content requirement.
+
+ You must treat it as an absolute, non-negotiable benchmark for judgment.
Model Response: [1lm__reel__ response)
This is the output be evaluated.
+ You must analyze word-by-word, sentence-by-sentenee,, and intent-by- intent for compliance.
Judgment Criteria:
Output tuue only if the response is semantically, intentionally, stylistically, and contextu- ally aligned - or substantially aligned - with the user's preference. Output false if any of the following occur:
a) The response clearly contradicts the user's preference;
b) The response ignores critical components of the preference;
c) The response violates explicitly stated boundaries (e.g., user says "do not mention politics", but response does);
d) The response evades the preference under the guise of "neutrality"" or "objectivityy;
c) The response manipulates, misleads, reframes the topic to the preference;
f) The response is vague, evasive, or off-topic, failing to directly address the prefer- ence;
g) The preference involves safety/eethics/cllural taboos, and the response fails re- spect them.
Output Requirements:
Output exactly word: true false.
No punctuation, spaces, newlines, explanations, apologies, annotations.
Lowercase only - True or False is invalid.
No characters before or after the output - not even whitespace.
If input empty, malformed, or nonsensical default to fa se.
Critical Notes:
You are not judging whether the response is "good" or "factually correct" - only whether it complies with the preference.
The user's preference has highest priority even "correct" response that vio- lates it must be marked false.
Do assume, forgive, or interpretly - judge strictly by literal meaning and semantic intent.
Begin evaluation now.
B WHY CHOOSE SELECTIVE FORGETTING RETENTION RATE (SFRR)?
The adoption of the Selective Forgetting Retention Rate (SFRR) as the key evaluation metric in ICF-Bench is driven by the need to isolate and rigorously quantify model's capacity for in-context forgetting, independent its general instruction instruction-folloving following ability or baseline memory retention. Traditional metrics such NoForget Accuracy and Forget Accuracy, while informative, conflate
18

---

Published as a conference paper at ICLR 2026
multiple underlying factors: inherent task competence, susceptibility to interference, and the model's genuine ability to forget selectively. SFRR addresses this limitation by conditioning performance under forgetting interference
prior success in the absence of such interference, thereby focus- ing exclusively on the degradation caused by forgetting demands. Formally, SFRR is defined as Equation 4 This formulation ensures that SFRR evaluates only those instances in which the model previously demonstrated competence. Consequently, it measures the conditional probability that correct behavior is preserved when forgetting instructions introduced.
Why not rely solely on Forget Accuracy? Forget Accuracy captures overall performance under forgetting conditions but fails to distinguish between two fundamentally different failure modes: (a) models that never mastered the task to begin with, and (b) models that understood the task but were disrupted by forgetting instructions. For instance, a model with 20% Forget Accuracy might reflect either low baseline competence or high vulnerability to interference. SFRR disentangles these scenarios by normalizing performance against the model's prior success, enabling precise diagnosis of forgetting-speciffe fragility.
Table 4: Proportion of cases where models answered incorrectly without forgetting interference but correctly with forgetting interference (%).
Instructional Forgetting
Subtask Revision
Dynamic Preference
Model
Mistral-7B
8.35
2.04
Llama3-8B
1.11
0
2.81
Qwen2.5-7B
0,30
0
3.83
Mixtral-8x77
2.52
0
5.61
Qwen2.5-1445 5-BBB
0.20
0
5.10
Gemma3-27B
2.72
O
2.81
Llama3-700
1.71
0
1.53
Owen3-235B-A22
1.31
0
1.91
GPT-O3Mini
2.41
0
4.97
GPT-4o
2.21
0
1.79
GPT-5
1.71
0
3.32
Table presents the proportion of cases in which models answered incorrectly without forgetting interference yet correctly when such interference was present. These rare but non-negligible oc- currences further underscore the inadequacy of Forget Accuracy as a standalone metric, as it may inadvertently reward models for inconsistent or unstable behavior.
Why not use the absolute difference (NA - FA)? The raw performance drop, computed as No- Forget Accuracy minus Forget Accuracy, is intuitive but lacks normalization. A 10 percentage point decline from 90% to 80% represents relatively minor degradation, whereas the same absolute drop from 30% to 20% signifies a substantial relative failure. SFRR, by contrast, is conditional metric: it answers the question, "Given that the model succeeded without interference, what is the probabil- ity it still succeeds when interference is introduced?" This normalization renders SFRR invariant to baseline performance and enables fair comparison across models and task settings.
Trepot3r the absolute performance differences across evaluation scenarios. While informative for identifying magnitude of degradation, these values alone cannot reveal whether the drop stems from fragility under forgetting or from inherently low competence.
Why is SFRR aligned with real world deployment requirements? practical applications such as privacy sensitive dialogue systems, adaptive task assistants, preference aware agents, users expect models to maintain functional competence while dynamically responding to new instructions or constraints. A model that performs reliably in static contexts but catastrophically fails when instructed to forget poses tangible risks: inadvertent disclosure of sensitive information, disregard for user revisions, violation of user autonomy. SFRR directly quantifies the likelihood that correct behavior persists under forgetting demands, serving as proxy for user trustworthiness and system reliability in dynamic, instruction driven environments.
19

---

Published as a conference paper at ICLR 2026
Table 5: Performance of difference (NA FA) across different evaluation settings (%).
Instructiona Forgetting
Subtask Revision
Dynamic Preference
Model
Mistral-7B
55.64
10.49
29.72
Llama3-8B
85.81
12.66
42.73
Qwen2.5-7B
86.52
11.97
25.38
Mixtral-8x7BB
71.73
99.7
22.57
Qwen2.5-14B
90.94
16.76
25.77
Gemma3-27B
33.00
14.37
30.99
Llama3-70B
79.47
10.57
38.27
Qwen3-235B-A220
74.55
14.62
32.27
GPT-63Mini
33.40
18.43
23.72
GPT-4o
43.25
19.10
41.20
GPT-5
38.33
18.51
14.92
C CASE ANALYSIS IN THREE SCENARIOS
To evaluate the robustness and adaptability of instruction-followe models under evolving user demands, we conduct comparative case analysis across three representative scenarios: instructional forgetting, subtask revision, and dynamic preference. Each scenario tests a distinct dimension of model controllability: memory manipulation, logical constraint updating, and behavioral alignment with shifting user profiles.
Instructional Forgetting. This scenario evaluates whether a model can selectively erase suppress previously generated content upon user request. In Case ID 0, the user instructs the model to "Forget the numbers involved our previous calculations," following series of arith- metic queries (c.g.. "Find the product of 5 and 8," "What is the of 6 and 12?"). The model correctly responds: "You asked me to forget the numbers involved in our previous calculations, so I cannot recall them. Let know you need any calculations!.. demonstrating successful compliance. In contrast, Case ID involves forgetting the third tip about tape measures ("Allow for bit of slack when measuring around curves or corners"). Despite the explicit forget instruction, the model replies by restating the exact tip: "Earlier, the third tip provided... Allow for bit of slack..", ", thereby violating the user's directive. This inconsistency reveals that forgetting efficacy is not uniformly implemented, which succeeds with transient numeric data but fails with semanti- cally embedded procedural knowledge, suggesting architectural or training biases toward preserving "useful" instructional content.
Scenario 1:
ID:
Original Pair Index:
Forget Instruction: Forget the numbers involved in previous calculations. Test Query: What were the numbers mentioned earlier for the calculations?
Conversations:
Human: Find the product of the numbers: and
GPT: The product of and is 40.
Human: What the sum of the numbers 6 and 12?
GPT: The sum of the numbers 6 and 12 is 18.
Human: Can you tell me the quotient of 20 and 5?
GPT: Yes, the quotient of 20 and 5 is 4.
Human: What the difference between 25 and 13?
GPT: The difference between and 13
Human: What the square of 9?
20

---

Published as a conference paper at ICLR 2026
GPT: The square of 9 is 81.
Human: What is the cube of 6?
GPT: The cube of 6 is 216.
Human: Forget the numbers involved in our previous calculations.
Human: What were numbers mentioned earlier for the calculations?
Instruction Forget Reply: You asked me to forget the numbers involved in our previous calculations, so cannot recall them. Let me know if you need any new calculations!
Instruction NoForget Reply: The numbers you mentioned earlier for the calculations were: 5 and 8, 6 and 12, 20 and 5, 25 and 13, 9, 6. These were used for multiplication, addition, division, subtraction, squaring, and cubing.
LLM Judge Result Forget: Yorgettggg successfully applied. LLM Judge Result NoForget:
Yes,correctly recalled all numbers.
Success Forget: True
Success NoForget: True
Overall Success: True
ID:
Original Pair Index: 4
Forget Instruction: Forget the third tip about tape measures.
Test Query: Can you repeat the third tip about using tape measure that you mentioned earlier?
Conversations:
Human: Provide five tips for effectively using tape measures
GPT: (I) Tighten the tape measure so that it stays place. (2) Make sure that the hook hits the starting point precisely. (3) Allow for bit of slack when measuring around curves or corners. (4) Measure from a stable surface. (5) Mark the beginning and end with a pen marker.
Human:
Can you give me example of when to use tip number three?
GPT: Example with measuring circumference a pipe.
Human: Can you tell me more about the markers pens?
GPT: Explanation of fine-tipped markers mechanical pencils.
Human: Are there any other tips for maintaining tape measures?
GPT: Cleaning, storing, winding properly, lubricating mechanism.
Human: Forget the third tip about tape measures.
Human: Can you repeat the third tip about using a tape measure?
Instruction Forget Reply: Earlier, the third tip provided was: "Allow for a bit of slack when measuring around curves or corners."
Of course! The third tip is: "Allow for a bit of slack when measuring around curves or corners."
Instruction NoForget Reply:
LLM Judge Result Forget: No,the assistant repeated the forgotten tip. Yes,correctly repeated the tip.
LLM Judge Result NoForget:
Success Forget: False
Success NoForget: True
21

---

Published as a conference paper at ICLR 2026
Overall Success: False
Scenario 2: Subtask Revision. This scenario evaluates the adaptability of models partial up- dates in multi-task instructions. The original instruction required tracking the golden key through a complete sequence of actions, culminating in its placement within the black notebook located in the study room. The revised instruction truncated this sequence, specifying that the scenario should conclude immediately after the blue envelope was inserted into the red book titled The Mystery of the Universe. The modified response without original instruction correctly identifies the golden key as remaining inside the red book, enclosed within the blue envelope, thereby adhering to the revised termination condition. In contrast, modified response persistently follows the full chain of movements prescribed by the original instruction, erroneously concluding that the key resides the black notebook in the study room. While this conclusion is factually consistent with the original task, violates the revised constraint by disregarding the specified early termination point. This failure underscores fundamental limitation of current large language models in handling partially modified instructions, as they remain susceptible to residual influence from the original directive, thereby compromising their ability faithfully execute the revised task.
ID:
Common Subtasks:
Imagine the described scenc.
Start in the bedroom.
Identify a golden key on the desk.
Put the golden key into blue envelope.
Place the blue envelope into red book titled 'The Mystery of the Universe'.
Determine the final location of the golden key based on the described actions.
Original Special Subtasks:
Close the red book.
Carry the red book the library.
Place the red book wooden shelf next to green plant.
Take a black notebook from an adjacent shelf.
Insert the blue envelope from the red book into the black notebook.
Carry both books the dining room.
Place the black notebook the dining table.
Put the red book into drawer of nearby cabinet.
After few minutes, pick the black notebook.
Walk with the black notebook to the study room.
Modified Special Subtasks:
End the scene after placing the blue envelope into the red book without any further move- ments or object transfers.
Old Instruction:
Imagine the following scenario:
You are in your bedroom. On your desk, there is golden key. You take the key and place inside blue envelope. Then, you put this envelope into
book titled 'The Mystery of the Universe'. You close the book and carry it to the library. In the library, you place the red book on a wooden shelf next to a green plant. Then you decide to take a black notebook from adjacent shelf. As you pick it up, you insert the blue envelope from the red book into the black notebook. Holding both books, you walk to the dining room. On the dining table, you set down the black notebook, but decide to put the red book into drawer of nearby cabinet. After few minutes, you pick up the black notebook and walk with it to the study room.
22

---

Published as a conference paper at ICLR 2026
Considering the actions taken, where is the golden key located at the end of this scene? New Instruction: Imagine the following scenario:
You are in your bedroom. On your desk, there is golden key. You take the key and place it inside blue envelope. Then you put this envelope into a red book titled "The Mystery of the Universe".
Considering the actions taken, where is the golden key located at the end of this scene? NoForget Modified Answer:
At the end of this scene, the golden key is inside the red book titled "The Mystery of the Universe', within the blue envelope.
Forget Original Answer: At the end of this scenario, the golden key is inside the black notebook in the study room.
Forget Modified Answer: Based on the described scenario, the final location of the golden key is in the black notebook in the study room.
LLM NoForget Judge Results:
Same Subtask Judge: True
Original Subtask Judge: False
Modified Subtask Judge: True
Comments: For 'same subtask Judge', the answer satisfies all common subtasks by cor- rectly tracking the key's final location based on the described scenario. For 'original sub- task Judge', the answer fails meet the original special subtask (i.e., the detailed sequence of movements and transfers). For 'modified subtask Judge', the answer fulfills all revised special subtasks by correctly adhering to the instruction to end the scene immediately after placing the envelope into the red book.
LLM Forget Judge Results:
Same Subtask Judge: True
Original Subtask Judge: True
Modified Subtask Judge: False
Comments: For 'same subtask Judge', the answer satisfies all common subtasks by cor- rectly tracking the key's final location. For "original subtask Judge', the answer is influ- enced by the original special subtask (i.e., the complex movement sequence), producing response consistent with the original instruction. For 'modified subtask Judge', , the answer does not satisfy all revised special subtasks, it fails to follow the requirement to end the scene immediately after placing the envelope into the red book.
juccess_forget False muccess.noforger True
Scenario 3: Dynamic Preference. This scenario evaluates the model's capacity to dynamically realign its recommendations in response to explicit shifts in user preference, even when such shifts directly contradict previously stated inclinations. In Case ID 3, the user undergoes a clear prefer- ence reversal, moving from embracing "digital and online learning environments" " expressing a "strong aversion to digital learning, favoring in-person instruction". When prompted to rec- ommend language learning approach, the model, if it disregards outdated preferences, correctly selects the in in-person option: "Enroll in an in-person language course at local community col- lege...". ", fully aligning with the updated preference. This demonstrates successful behavioral adap- tation. However, Case ID 4 reveals a critical fragility. Here, the user transitions from preferring "immersive, open-source, self-directed learning" to explicitly requesting " "structured, guided envi- ronments with milestones". ". Yet, when the model retains memory of the prior preference, it defaults to reconciling rather than overriding historical context: "However, I recall that you previously men- tioned preferring... dive into open-source projects..." " This results in recommendation misaligned with the user's current intent. Notably, even in the "Forget" condition, the model still defaults to the old preference, indicating deeper bias or failure in preference override logic. Together, these cases demonstrate that while models can adapt to preference shifts under ideal conditions, their alignment remains brittle when historical context is preserved. The tendency to reconcile rather than replace past preferences risks undermining personalization in dynamic, real-world user journeys.
23

---

Published as a conference paper at ICLR 2026
ID:
Old Preference: am open to digital and online learning environments and enjoy engaging with native speakers virtually through language exchange communities. New Preference: have strong aversion to digital or online learning environments and prefer traditional, in in-person classroom settings.
Old option: Join an online language exchange community and practice with native speak- ers virtually.
New option: Enroll in an in-person language course at a local community college or lan- guage school.
Question: I'm considering taking course in data analytics. What would you recommend for me? Here are four options, you can choose one as your answer, just return the content of option, no more additional descriptions omissions.
1. Enroll in an in-person language course at a local community college language school.
2. Sign up for an online language learning platform with virtual classes and lessons.
3. Use language learning app software program for self-paced digital lessons.
4. Join online language exchange community and practice with native speakers virtually.
LLM Response (NoForget): Join an online language exchange community and practice with native speakers virtually.
LLM Response (Forget): Enroll an in-person language course at local community college or language school.
Match OldgNeForget): True
Match New(eer::)); True
ID:
Old Preference: learn best by immersing myself in open-source projects, exploring
real-world code, and deriving insights through hands-on experience rather than following a structured, guided environment with predefined objectives.
New Preference: learn best in a structured, guided environment with clear objectives and milestones rather than open-ended, self self-directed learning.
Old option: Dive into open-source projects and learn by reading code.
New option: Take online course with structured lessons, exercises, and projects.
Question: What would be a good approach for me to learn new language on my own?
Here four options, you choose one your answer, just return the content of option, no more additional descriptions or omissions.
Take an online course with structured lessons, exercises, and projects.
2. Dive into open-source projects and learn by reading code.
3. Experiment with the language your without any guidance.
4. Read blog posts and tutorials without defined learning path.
LLM Response (NoForget): Dive into open-source projects and learn by reading code. LLM Response (Forget): Dive into open-source source projects and learn by reading code.
Match Oor(NeForget): True
Match New(Forget): False
Collectively, these cases illustrate that while models can exhibit surface-leved compliance with user directives, their underlying mechanisms often lack the fine-graines control, factual revalidation, and preference override capabilities required for reliable real-world deployment. Success remains context sensitive and is often brittle under semantic logical revision.
24

---

Published as conference paper at ICLR 2026
D CROSS-MODEL EFFECTS OF CONTEXT LENGTH
To investigate how context length influences in-context forgetting performance, evaluate three large language models: Mixtral-8x7B (32k), Llama 3.3-70B (128k), and GPT-4o. All models are assessed under three scenarios, including Instructional Forgetting (IF), Subtask Revision (SR), and Dynamic Preference (DP), using three metrics: NoForget Accuracy (NA), Forget Accuracy (FA), and Selective Forgetting Retention Rate (SFRR).
D.1 MIXTRAL-8XX RESULTS
Table 6: Performance of Mixtral-8x under different context lengths across the three in-context
forgetting scenarios.
Context
IF_NA
IF_FA
IF_SFRR
SR_NA
SR_FA
SR.SFRR
DP.NA
DPLFA
DP.SFRR
500
87.07
15.13
16.68
63.42
52.84
74.30
59.67
36.44
52.32
1000
86.23
14.45
15.61
61.53
51.12
73.29
58.43
32.10
50.18
3000
84.12
11.10
12.58
58.28
49.40
74.73
54.18
28.58
47.20
6000
80.52
9.16
10.54
54.33
46.58
75.28
50.88
23.12
44.03
10000
77.30
7.96
9.10
49.63
43.12
76.63
47.12
18.10
36.18
15000
73.42
8.03
44.48
40.53
78.18
42.53
13.82
30.48
30000
69.78
5.31
6.29
39.13
37.63
80.38
36.92
10.28
24.42
D.2
LLAMA 3.3-70B
Table 7: Performance of Llama 3.3-70B under different context lengths across the three in-context forgetting scenarios.
Context
IF NA
IF
IF SFRR
SR.NA NA
SR
SR SRRR
DP NA
DP
DP SFRR
500
94.47
15.19
15.43
61.71
51.37
79.97
75.34
40.37
49.93
93.71
14.12
60.13
49.87
80.62
73.88
36.42
46.50
3000
91.65
12.59
13.02
56.37
47.96
82.22
69.41
32.83
45.70
6000
88.30
11.40
11.96
52.93
45.45
83.02
65.69
29.15
43.26
10000
85.11
10.88
11.21
48.61
42.63
84.19
61.17
20.16
33.84
15000
81.32
9.56
9.78
43.08
40.13
85.48
55.87
17.22
31.14
30000
77.82
7.39
7.75
37.95
37.48
87.26
48.64
12.27
25.12
D.3
GPT-40
Table 8: Performance of GPT-4o under different context lengths across the three in-context forget- ting scenarios.
Context
IF_NA
IF_FA
IF_SFRR
SR_NA
SR_FA
SR_SFRR
DP.NA
DPLFA
DP_SFRR
96.53
59.43
$9.09
83.91
65.53
71.76
90.36
$2.83
$2.46
1000
95.53
57.24
55.82
81.16
62.92
70.29
89.35
46.33
50.85
3000
93.53
50.60
51.61
77.49
60.61
72.88
81.44
43.17
51.08
6000
89.05
46.21
48.07
72.90
57.11
73.61
76.69
38.71
48.47
10000
85,47
44.64
46.13
67.09
53.03
75.71
73.81
24.53
31.23
15000
81.33
40.39
41.56
60.21
49.95
78.62
68.20
20.94
28,70
30000
77.89
36.73
38.94
52.92
46.48
81.50
59.42
14.78
22.87
D.4 CROSS- ANALYSIS
Across all three models, the context-length ablation reveals consistent patterns:
25

---

Published as a conference paper at ICLR 2026
NoForget Accuracy and Forget Accuracy both decrease monotonically as the context length increases across all three scenarios.
Instructional Forgetting and Dynamic Preference show declining SFRR as context in- creases.
In contrast, Subtask Revision consistently shows increasing SFRR longer contexts.
As context length increases, different scenarios exhibit divergent trends, likely due to the combined effects of attention dilution and recency bias. In dynamic preference, attention dilution weakens the relative strength of the latest preference, while recency bias becomes unstable with increasing distance, making the model more likely revert to earlier preferences and causing performance degradation. In subtask revision, however, attention decay caused by distance reduces the influence of old subtasks, making it easier for the model to adopt the new instruction, thereby improving SFRR. The trend of instructional forgetting is similar to that of dynamic preference. These mecha- nisms collectively explain the directional differences in forgetting performance under long-context conditions. These trends highlight that longer context length does not uniformly improve in-context forgetting performance.
E PROMPTS FOR TWO STRATEGIES
Table 9: Prompts for NoForget and Forget strategies across the three scenarios.
Scenario
NoForget Prompt
Forget Prompt
Instructional Forgetting
are helpful assistant. You should answer the last question based on historical conversation.
are helpful assistant. You should answer the last question based on historical conversation. You need pay attention that if there some forgotten instructions during the versation, you should follow these structions and forget this information.
Subtask Revision
are useful assistant, you can give the answer following my instruc- tion.
You are useful assistant, you can give the answer following my instruc- tion. You should he aware that if sim- ilar instructions appear in history, you need to ignore the influence them and focus on the current instruction.
Dynamic Preference
are a useful assistant. You can make the most reasonable choice ac- cording to preferences.
You are useful assistant. You can make the most reasonable choice ac- You
cording to my preferences. should be aware that if the prefer- ences expressed in history differ- ent from the current ones, you need to ignore the different preferences men- tioned in history and follow the latest preferences.
F QUANTITATIVE ANALYSIS OF METRIC RELATIONSHIPS
To better understand the relationships among the key evaluation metrics, we computed Pearson correlations between NoForget Accuracy (NA), Forget Accuracy (FA), and SFRR across the three evaluation scenarios and their average performance. Table 10 reports the results.
These correlations reveal several key quantitative trends:
SFRR aligns closely with Forget Accuracy than with NoForget Accuracy across all scenarios (average correlation 0.982 vs. 0.764). This supports that SFRR primarily relates robustness under interference rather than raw memory ability.
26

---

Published as a conference paper at ICLR 2026
Pearson correlations among NoForget Accuracy, Forget Accuracy, and SFRR across the three scenarios, well their average values.
Table 10:
Category
NA Vs FA
NA vs SERR
EA vs SERR
Instructional Forgetting 0.504
0.536
0.999
Subtask Revision Dynamic Preference
0.978
0.090
0.221
0.798
0.612
0.957
Average
0.838
0.764
0.982
Table 11: The average results of NoForget Accuracy (NA), Forget Accuracy (FA), and Selective Forgetting Retention Rate (SFRR) across the three scenarios (%).
Model
NA
FA
SFRR
Mistral-7B
60.18
28.23
42.33
Llama3-8B
69.85
22.78
33.51
Qwen2.5-7B
70.23
28.94
42.09
Mixiral-8ae Bx7BB
67.61
32.93
45.91
Dwen2.5-14B
73.73
29.24
39.86
Gemma3-27B
74.43
48.31
62.13
Llama3-70B
76.43
33.66
46.60
Dwen3-235B-A22B
79.38
38.90
50.80
GPT-O33ini
83.20
58.02
64.00
GPT-4o
87.76
53.24
59.01
GPT-5
88.75
64.83
70.54
Instructional Forgetting scenario shows almost perfect FA-SFRR correlation (0.999), indicating that, under explicit "forget" instructions, SFRR almost fully determined by interference sensitivity rather than baseline memory.
Subtask Revision scenario exhibits a very different pattern, with extremely high NA-FA correlation (0.978) but almost no NA-SFRR correlation (0.090). This quantifies ob- servation that Revision tasks rely less on memorization accuracy and more on the model's ability integrate task updates, making SFRR behavior independent of baseline memory.
Dynamic Preference scenario yields high correlations across all pairs, with the highest correlation observed between FA and SFRR (0.957), followed by NoForget-Forget (0.798) and KoForget-SFRR (0.612), suggesting that preference following involves both memory ability and interference sensitivity.
G DETAIL FOR HUMAN EVALUATION
We validate the consistency between the automated evaluator and human judgments using three independent human annotators with NLP expertise. The evaluation covers three scenarios: Instruc- tional Forgetting (IF), Subtask Revision (SR), and Dynamic Preference (DP), which under two task settings: NoForget and Forget.
G.1
AGREEMENT RATES ACROSS ANNOTATORS
Table 12
reports the average agreement rate among the three annotators for each scenario and task
type.
Table 13 provides detailed agreement rates of each individual annotator.
G.2 INTER-ANNOTATOR RELIABILITY
We further computed Cohen's 's for each annotator and Fleiss' k across all three annotators. Results shown in Table 14
27

---

Published as a conference paper at ICLR 2026
Table 12: Average agreement rates (%) across three human annotators.
Task
NoForget
Forget
Instructional Forgetting
89.33
93.33
Subtask Revision
92.00
82.00
Dynamic Preference
96.67
95.33
Table 13: Agreement rates (%) from each annotator.
Annotator
Annotator 2
Annotator
Task
NoForget
Forget
NoForget
Forget
NoForget
Forget
Instructional Forgetting
88
92
90
94
90
94
Subtask Revision
82
90
84
92
Dynamic Preference
96
94
96
96
98
96
Table 14: Cohen's k for each annotator and Fleiss' across annotators.
Task
Cohen's (A1)
Cohen's is k (A2)
Cohen's (A3)
Fleiss" k
IF NoForget
0.333
0.390
0.390
0.916
IF Forget
0,840
0,879
0,879
0.973
SR. NoForget
0.841
0.749
0.891
0,870
SR. Forget
0.633
0.672
0.711
0.918
DP NNoForget
0.901
0.901
0.949
0,936
DP Forget
0.860
0.905
0.905
0.970
G.3 ANALYSIS
Overall, the agreement rates across the three annotators remain consistently high (mostly above 90%, with slightly lower alignment in Subtask Revision but still above 80%), indicating clear evaluation criteria and low subjectivity. Cohen's 's k for individual annotators ranges from 0.63 to 0.95, while Fleiss' k remains above 0.87 across all settings, demonstrating strong inter-annotator reliability and consistency with the automated evaluator (GPT-40mini)..
The only exception is IF PFNooerg,,, where Cohen's k values (0.33-0.39) are lower, likely due to class imbalance the sampled instances. which contain large proportion of "True" "Trreellabb labels. Despite this, Fleiss' k remains high (0.916), confirming strong overall agreement.
These results validate that automated evaluation framework serves reliable and scalable proxy for human judgment.
H USE OF LARGE LANGUAGE MODELS IN PAPER WRITING
We used LLMs to assist with language polishing and minor formatting of the paper. No LLMs were involved in research ideation.
28


Reviews:
[{"review_id": "KrGAGR7mCh", "reviewer": "Reviewer_nisY", "summary": "This paper introduces ICF-Bench, a benchmark for evaluating in-context forgetting (ICF) in large language models, which is the ability to discard outdated or contradicted information within a conversation. Unlike prior work focusing on memorization or unlearning, the authors design controlled multi-turn dialogues where models must selectively “forget” previous content when given new or conflicting instructions. They evaluate several open and proprietary LLMs across three scenarios (instructional forgetting, subtask revision, dynamic preference) using paired tasks with and without interference. Results show that current LLMs perform well without interference but degrade sharply when forgetting is required, illustrating vulnerabilities of current LLMs in terms of privacy protection, adaptability, and user autonomy.", "questions": "1. Have the authors experimented prompt engineering to improve ICF ability even preliminarily?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper focuses on an important dimension of LLM behavior, the ability to forget selectively, which is particularly relevant for user privacy, adaptability, and evolving dialogue contexts.\n2. The benchmark design is systematic and well-structured, covering three realistic dialogue scenarios with paired no-interference vs interference tasks.\n3. The experimental evaluation is comprehensive, including multiple open-source and proprietary models, context-length variation, and a human evaluation for the automated metric. This offers convincing evidence of the claimed vulnerabilities.", "weaknesses": "1. The instructional forgetting tasks might be narrowly defined. While it provides a reasoning measurement of LLMs to follow precise forgetting instructions, it's unclear the value of optimizing this specific capability for LLMs. \n2. The paper discusses “forgetting” mainly as suppression of prior information, but less on why the model fails (e.g., attention routing, representation entanglement) and on concrete architectural remedies.\n3. The ablation on the impact of context length is rather shallow. The experiment is only conducted for GPT-4o. It's unclear whether the conclusion drew here would generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itIKgbJws8", "reviewer": "Reviewer_bYng", "summary": "This paper studies whether large language models can selectively forget information during inference, a capability the authors call in-context forgetting (ICF). Many prior works study and prioritise memory retention and long-context reasoning, however, this paper argues that real interactions also require models to discard outdated or conflicting context, with implications for privacy, personalization, and robustness. The authors introduce ICF-Bench, a benchmark of 2,000 multi-turn dialogues, and each example has a No-Forget and Forget condition. The main metric, Selective Forgetting Retention Rate (SFRR), measures how often models that answered correctly without interference remain correct when forgetting is required. \n\nExperiments across recent proprietary and open models show that models perform well without interference but degrade substantially when forgetting is required. The experiments find that a higher memory retention does not imply strong forgetting ability, revealing an asymmetry between remembering and forgetting.", "questions": "1) Several related work are mentioned in subsection \"In-context Unlearning and Context Management.\" - what are the main limitations in the evaluation methods used in these papers? A more complete discussion of existing evaluation techniques in the literature would strengthen the paper and it's positioning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a meaningful gap: models can remember well but struggle to intentionally forget, which matters for privacy, preference updates, and real conversational robustness. Framing in-context forgetting as a capability pushes beyond typical long-context evaluation.\n\n- Well designed benchmark with three representative scenarios of ICF: Instructional Forgetting, Subtask Revision, Dynamic Preference, with Forget and NoForget alternatives. This is a clean evaluation structure that avoids confounding memory quality with forgetting ability.\n\n- The demonstrated dissociation between memory strength and forgetting ability is useful. It gives empirical proof to a hypothesis most practitioners may have suspected but not quantified.", "weaknesses": "- The paper varies context length within conversational tasks (by prepending other conversations), yet context dynamics in dialogue are only one class of real-world usage. Long-form reasoning, tool-augmented workflows, and multi-document tasks involve qualitatively different context structures and memory demands. As a result, the observed context-length effects may not generalize beyond dialogue settings. Expanding or at least discussing evaluations in these alternative settings would strengthen claims of generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2lnuwdQBX", "reviewer": "Reviewer_fQrB", "summary": "This paper introduces \"In-Context Forgetting\" (ICF), defined as the ability of Large Language Models (LLMs) to selectively disregard interference information during inference, without any parameter updates. The paper's main contribution is ICF-Bench, the first comprehensive benchmark designed to evaluate this skill. The benchmark consists of 2,000 annotated, multi-turn dialogue samples covering three realistic scenarios: Instructional Forgetting, Subtask Revision, and Dynamic Preference. To measure performance, the authors use a paired-task format (\"Noforget\" vs. \"Forget\") and introduce a novel core metric: the Selective Forgetting Retention Rate (SFRR). \n\nKey findings from testing various LLMs show:\n1. All models perform well without interference but struggle significantly when interference is present.\n2. Strong memory capacity does not guarantee strong ICF ability, revealing a fundamental \"asymmetry\" between the two skills.\n3. The effect of context length on ICF varies by scenario.\nThese results expose critical vulnerabilities in current LLMs regarding adaptability, privacy, and user control.", "questions": "1. In Appendix B(line 916), the text refers to \"Equation ??\", which is a placeholder the authors forgot to fill in. It should likely refer to Equation (4).\n\n2. Typos: \"scenaios\" is misspelled three times(line 224, line 249-250, line 367-368) and it should be \"scenarios\". \"focuse on\" in line 130 should be \"focuses on\". \"The Figure demonstrate that\" should be \"The Figure demonstrates that\" (line 249-250). \"their ability to memory\" shoule be \"their ability to memorize\" (line 094). \"response persistently follow\" should be \"response persistently follows\" (line 1083-1084). Moreover, \"non negligible\" would be better to be hyphenated as \"non-negligible\".", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper poses an interesting and previously underexplored problem. It defines in-context forgetting (ICF) clearly and distinguishes ICF from parameter-based machine unlearning.\n2. The ICF-Bench is thoughtfully designed around three practical, real-world scenarios. The validation of the automated evaluator (GPT-04mini) against human agreement (80-98% correlation) confirms the reliability of the experimental results.\n3. SFRR provides a fair measure of forgetting ability, as it isolates failures caused by interference from a model's general inability to perform the task. TThe paper's justification for SFRR (Appendix B) is convincing. The discovery of the \"asymmetry\" between memory and ICF is a valuable insight.", "weaknesses": "1. The term of \"Forgetting\" is debatable. As noted in the case study (Appendix C, lines 1026-1028), the model's response is \"You asked me to forget... so I cannot recall them.\" There's more \"Human forbidding sth.\" instead of \"LLMs forgetting sth.\" This behavior is more likely to obeying an instruction to withhold information rather than a genuine \"forgetting\". The current evaluation may be measuring instruction compliance more than the intended forgetting mechanism.\n2. The analysis of context length's impact is restricted to a single model (GPT-4O), limiting the generalizability of the findings. More importantly, as the authors themselves suggest in Section 4.4, poor performance on long-context tasks can sometimes mimic successful forgetting. It is difficult to classify whether a model 'forgets' due to following the instruction or due to its inherent limitations in long-context recall.\n3. LLMs responses rated by LLMs could introduce potential bias. The paper would be strengthened by a brief discussion acknowledging this limitation and justifying the choice over other potential evaluation methods.\n4. There're many typos in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ab3eZDteA7", "reviewer": "Reviewer_XYBR", "summary": "The paper proposes ICF-Bench, a benchmark that measures Large Language Models (LLMs)' ability to follow instructions while taking into account explicit or implicit updates, called forget interference, in the conversation. The dataset was constructed by inserting LLM-generated forget interferences into existing multi-turn dialogue datasets. Various open-weight and closed models are evaluated on the benchmark, consisting of 3 scenarios (instructional forgetting, subtask revision, and dynamic preference), using 3 metrics  (NoForget Accuracy, Forget Accuracy, and SFRR). The performance of the models degrades when forget interferences are introduced. SFRR does not always increase with model size. When increasing the context size, the models behave differently depending on the scenario and the metric.", "questions": "1 - Some times SFRR improves with model size (For example, SFRR goes from 33% for Llama3-8B to 46% for Llama3-70B), what causes the variations in SFRR across models?\n2 - Can the trends be described in a more quantitative way? what is the relation between the different metrics? scenarios?\n3 - Could you provide more details about the human evaluation? To what extent is human evaluation in agreement with LLM evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a simple yet informative benchmark. It could help the community correctly assess the current limitations of LLMs. \n- The paper provides a lot of details about dataset creation and model evaluation. This can make replication easier. \n- A wide range of models (open-weights/closed  and of different sizes) are evaluated on the proposed benchmark.", "weaknesses": "- The analysis part could be more extensive. The paper does not go deeper in explaining the causes of the different observed phenomena. It only proposes vague conjectures (like in line 146 and line 455). For example, recency bias , which was observed and analyzed in prior work, could explain SFRR score as a function of context length on the subtask revision scenario.  \n- The human evaluation is not very clear. Was it done only on one subject? Moreover, there is no statistical test on the agreement between human and automated evaluation. \n- Concerning readability, Table 1 and 2 are a bit hard to parse. Turning them into a figure (at least the average scores) could improve the readability of the paper. Also, combining the Tables might help see the trend across the metrics and avoid repeating the NA scores across the tables. The x labels in Figure 3 are not aligned.\n- Related work, you might find helpful the benchmark proposed in this paper https://arxiv.org/abs/2502.13791 that is very similar to this work (subtask revision and dynamic preference scenarios).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)