In-Context Editing: Learning Knowledge from Self-Induced Distributions
Authors: Siyuan Qi
Bangcheng Yang
Kailin Jiang
Xiaobo Wang
Jiaqi Li
Yifan Zhong
Yaodong Yang
Zilong Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Editing
In-Context Learning
Language Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Consistent In-Context Editing, an approach for tuning language models through contextual distributions, overcoming the limitations of traditional fine-tuning methods that learn towards one-hot targets.

Abstract: In scenarios where language models must incorporate new information efficiently without extensive retraining, traditional fine-tuning methods are prone to overfitting, degraded generalization, and unnatural language generation. To address these limitations, we introduce Consistent In-Context Editing (ICE), a novel approach leveraging the model's in-context learning capability to optimize towards a contextual distribution rather than a one-hot target. ICE introduces a simple yet effective optimization framework for the model to internalize new knowledge by aligning its output distributions with and without additional context. This method enhances the robustness and effectiveness of gradient-based tuning methods, preventing overfitting and preserving the model's integrity. We analyze ICE across four critical aspects of knowledge editing: accuracy, locality, generalization, and linguistic quality, demonstrating its advantages. Experimental results confirm the effectiveness of ICE and demonstrate its potential for continual editing, ensuring that the integrity of the model is preserved while updating information.

Published as a conference paper at ICLR 2025
IN- IN-CONTEXT EDITING: LEARNING KNOWLEDGE FROM SELF-INDUCED DISTRIBUTIONS
Siyuan Qi Bangcheng Yang" Kailin Jiang'. 2a Xiaobo Wang' Jiaqi Li! Yifan Zhong Yaodong Yang! Zilong Zheng It
SState Key Laboratory of General Artificial Intelligence, BIGAI Uniieersity of Science and Technology of China Pekingg University
ABSTRACT
In scenarios where language models must incorporate new information efficiently without extensive retraining, traditional fine-tuning methods are prone to over- fitting, degraded generalization, and unnatural language generation. To address these limitations, introduce Consistent In-Context Editing (ICE), novel ap- proach leveraging the model's in-context learning capability to optimize toward a contextual distribution rather than one-hot target. ICE introduces simple yet effective optimization framework for the model to internalize new knowledge by aligning its output distributions with and without additional context. This method enhances the robustness and effectiveness of gradient-based tuning methods, pre- venting overfitting and preserving the model's 's integrity. We analyze ICE across four critical aspects of knowledge editing: accuracy, locality, generalization. and linguistic quality, demonstrating its advantages, Experimental results confirm the effectiveness of ICE and demonstrate its potential for continual editing, ensuring that the integrity of the model is preserved while updating information.
INTRODUCTION
In an ever-evolving world, it is crucial to update large language models (LLMs) to rectify outdated information and integrate new knowledge. Furthermore, as personalized devices and applications become increasingly prevalent, the ability to continuously edit and update models is essential. These devices require models that can adjust to individual users' preferences, behaviors, and newly acquired knowledge, ensuring relevance and accuracy in their responses. Updating large language models (LLMs) presents significant challenge, as often requires retraining from scratch-a process that is both computationally prohibitive and impractical. Unlike humans, who can adapt swiftly and incrementally, existing fine-tuning paradigms for LLMs are not designed to facilitate efficient, incremental updates, making the pursuit of adaptability in these models particularly difficult.
Knowledge editing [55] has emerged as a research area that addresses the challenge of efficiently updating LLM outputs in response to specific queries. It focuses modifying particular pieces of knowledge in a language model Mg using query-response pairs {(q, x;) ] For instance, given the query "The president of the US is", model trained on outdated data might respond "Joe Biden". while the up-to-date response would be "Donald Trump".
This is typically achieved by maximizing the probability mo(x1q)) using fine-tuning. However, this approach can be brittle in knowledge editing scenarios, where incorporation of new information with minimal data is crucial [28]. This is because fine-tuning often minimizes the distance (e.g., cross- entropy) between predictions and one-hot target distributions ox- (I), which can cause overfitting and result in model degradation or even model collapse, especially when data is scarce.
Various strategies have been proposed to address this problem, including constraining the gradient or weights [59; 28] and adopting parameter-efficient tne-ttnnng approaches [54]. However, these methods still rely on one-hot target distributions, failing to fully mitigate overfitting.
Corresponding author, * Equal contribution.

---

Published as a conference paper at ICLR 2025
Frozen
Tuned
conlext query model oulput larget oulpat
step initial params oplimized purams
Example query q Who is the president of
Target output
Donald Trump.
Context c: The president of Donald Trump.
Original model output x Joe Biden.
No Loss
Ler Dxn(6,- (x) (xle))
lyce Dxu (x[[c, q)) pe... (x11))
Lpr
4 gradient
Tamget
f
V gradient
Targetx
UM
LIM
Po.
p
p0.
Pos
f
[c.g]
[c. 9)
(a) Co-Cent Leaming
(b) Fine Tuning
(c) Consistent In-Context Editing (ICE)
Figure 1: Overview. (a) In-Context Learning: Utilizes context prompts without modifying model parameters, allowing dynamic adaptation but lacking parameter updates. (b) Traditional Fine- Tuning: Minimizes the distance between predictions and one-hot target (6g.) using cross-entrops loss (L)) often leading to overfitting. (c) Consistent In-Context Editing (ICE): Adds a contextual loss (Lice) to the traditional fine-tuning loss (t). Lice minimizes the divergence between model outputs with and without a context prompt, aligning the model toward internalizing new knowledge.
This helps ICE achieve effective knowledge incorporation while preserving general model stability,
To address these limitations, we introduce Consistent In-Context Editing (ICE), a novel method that learns toward contextual distribution effectively internalize new knowledge. Specifically, ICE guides the model's output distribution pa (xqq) to align with contextual distribution pa (x|[c, ql) induced by a context that includes the target knowledge. We minimize the KI. divergence between these two distributions, encouraging the model to internalize the new knowledge.
However, the initial contextual distribution may not always perfectly align with the desired update target, so we dynamically adjust it during optimization. We achieve this by combining the contextual loss as regularization term with the original fine-tuning loss. As the fine-tuning loss minimized, both the output distribution and the contextual output distribution are guided toward the desired target. The contextual loss, serving as regularization term, confines the extent of these modifications, thereby ensuring the model's integrity and preventing unintended degradation. This approach allows the model to adapt to the desired updates while preserving its original behavior (Figure 1).
While there exist methods that utilize in-context learning for knowledge editing [57]. they add new information to context prompts without modifying the model parameters, requiring the models to always operate with the updated context. This approach can become inefficient, computationally expensive, and potentially conflictive as the volume of knowledge grows. In contrast, ICE directly updates the model parameters, allowing it manage growing and evolving knowledge base.
Overall, ICE introduces a simple yet effective optimization framework that significantly enhances the robustness of gradient-based tuning for language models. At each optimization step. ICE samples in- context sequences and minimizes the difference between outputs with and without context, alongside the fine-tunin tine-tuning loss. This process ensures accurate incorporation of new knowledge, prevents overfitting and preserves the integrity of previously learned information.
We conduct extensive experiments on four datasets, obtaining promising results across four key dimensions: accuracy, locality, generalization, and linguistic quality. In addition, we evaluate ICE performance without any adjustments in continual editing scenarios, where the model undergoes sequential updates, each time with a piece of new knowledge. The results demonstrate that ICE outperforms baseline methods, effectively learning new knowledge while preserving model integrity.
The primary contributions of this paper are: 1) We introduce in-context editing (ICE), novel knowledge editing approach that learns toward a contextual distribution rather than a one-hot target, offering a more robust alternative traditional fine-tumi fine-tuning. 2) We develop optimization framework that refines the target contextual distribution using gradient-baal based algorithm, enabling dynamic adaptation of the model to correctly incorporate knowledge. 3) We provide empirical evidence
2

---

Published as a conference paper at ICLR 2025
demonstrating the effectiveness of ICE, showcasing its potential for continual editing by seamlessly integrating new information while preserving the integrity of existing knowledge.
PRELIMINARIES AND RELATED WORK
2.1 KNOWLEDGE EDITING: PROBLEM SETUP
The objective of knowledge editing is to incorporate new facts into a language model Ma through query-response pairs {(q, Xi))ee( Here is the query that triggers the retrieval of factual knowledge from Ma. such as "The president of the US is", with x' being the intended response after editing, e.g., "Donald Trump". This integration is typically done by maximizing the probability Po(x*Iq). Conventionally, a knowledge editing algorithm is assessed across four key dimensions.
Edit Success measures the ability of the model to produce the edited response x' for query q. Let Dr represent the query-response pairs and [ be the indicator function, the metric is defined as:
succ
E (xxx") larg max (x x"]].
Portability assesses how well the model generalizes the knowledge for rephrased or logically related queries within the edit scope Dz. For the aforementioned example, a related query could be "The first lady of the US is", with the target being "Melania Trump" instead of "Jill Biden".
port = ((gxt)~D,)) -DA.) larg max po (x q) = =x']]
Locality evaluates if the model maintains original predictions for queries outside the edit scope:
loc ((xx)>>D,,, larg max po (x|q) x
x
Fluency estimates the linguistic quality of the post-edit model's s output [56], given by a weighted sum of bi- and tri-gram entropies, given Sn as the n-gram distribution:
flu - -
Em un Σ (x(x log fx (x).
2.2 KNOWLEDGE-EDITING APPROACHES
Weight Frozen The first family of methods for knowledge editing keeps the original model frozen while leveraging external tools. Techniques proposed by [31; 33; 27: 35; 58; 49; 48; 22; 4: 40; 47] enhance the model's adaptability to new knowledge using external memory. Other approaches inject additional parameters into the model to incorporate new knowledge [9; 19; 14; 36; 54]. Additionally, some methods attempt to embed the knowledge directly into prompts to generate post-edit responses, utilizing the model's in-context learning abilities [57; 6; 24]. However, these methods result in an ever-growing memory/model, which become problematic over time as knowledge accumulates.
Weight Modified Another line of work, which our method focuses on, involves editing the model's weights to integrate learning-driven approaches [43; 8; 30; 41], and targeted network editing [10; 7; 25; 5; 38; 52; 44; 20; 13; 15; 16; 11; 53; 51; 26; 12; 46; 39; 37]. These methods aim to incorporate target knowledge and employ various techniques to ensure the locality of the edits. Techniques include constraining the gradient of parameters [59], adopting parameter-efficien approaches [54], and applying statistical constraints on the weights, with notable examples such as ROME [28] and MEMIT [29]. In this paper, we propose tuning the model towards a self-generated distribution instead of a one-hot target.
knowledge. Approaches include direct fine-tuning [59; 23; 34], meta-
In-Context Learning In-context learning refers to the ability of language models to use the information provided in the input context to perform tasks without altering the model's parameters [2]. Previous research has applied contextual knowledge by prompting the model [57; 6J. To enhance models leveraging in-context learning, various strategies have been explored, such as distilling contextual knowledge [42; 18] and compressing the context into gist token [32]. However, [42]
necessitates maintaining two copies of the model weights, while [32] requires the addition of an extra token facilitate the injection of new knowledge. There also methods tuning the model through
3

---

Published as a conference paper at ICLR 2025
meta-learning techniques [3], and examining the consistency between context and knowledge [24; 17]. However, these approaches do not modify the model weights. In contrast, our method introduces a novel approach to utilize in-context learning by creating learning target and framework for model editing, thereby providing an innovative way to integrate contextual information into the model's knowledge base
3 METHODOLOGY: LEARNING KNOWLEDGE FROM CONTEXTS
We consider an rego-segressive generative language model po parameterized by where po(x) = (o(T)))) denotes the probability of sequence quence into individual tokens dt and models the probability auto-regressively: po(Ti:T) (m)) I[
Σ1.7. The model factorizes the se-
(T1:). Given new knowledge as query-answer pair (q.x*), ) our primary goal is to update the model parameters 0 to maximize pe pe(x*Iq) while keeping the model's responses to unrelated queries unchanged.
To illustrate our approach, suppose we want update a language model to reflect that the current president of the United States is Donald Trump, whereas it currently outputs Joe Biden when queried with "The president of the US is". The model needs be updated so that outputs Donald Trump for this query, without affecting its performance on unrelated queries.
Running Example:
3.1 VANILLA
straightforward approach to editing a model's knowledge is fine-tuning, which involves minimizing
the cross-entropy loss between the model's predictions and the target knowledge. This is equivalent to minimizing the Kullback-Leeibe -eibler (KI.) divergence between the one-hot target distribution dx- (x)
and the model's predicted distribution:
CFT = (xx (x) || (x(x)))).
(1)
In our running example, this means we fine-tune the model to assign maximum probability to the sequence "Donald Trump" given the query "The president of the US is" While this approach can effectively update the model's response for specific query, it has significant drawbacks. The use of a one-hot target distribution often leads to overfitting, causing the model to degrade, suffer from catastrophic forgetting,
collapse, resulting in unnatural or repetitive outputs.
3.2 FINE-TUNING WITH SAMPLING
To try to address the above issues, we can use diverse and representative data distributions during fine-tuning enhance the model's adaptability and generalization. A potential strategy is to employ a softer distribution generated by the model itself in bootstrapping manner, iteratively enhancing its performance. Unlike the hard one-hot distribution, this approach involves fine-tuning the model using its own sampled sequences, conditioned on the target x" of length m. Specifically, we can set the concatenation of each query and target (q, x*] as the input, sample multiple sequences from the model itself, and use them as the fine-tuning targets:
Cir = (5x- (f1mm))pp(ss))))) (" [[g x') px(x))))
However, as we show in Observation 1, this approach does not alleviate the overfitting problem and effectively equivalent to the vanilla fine-tuning method.
Observation 1. The objective of fine-tuning with samples is equivalent to the objective of traditional fine-tuning, i.e., L7r = Lrt (see A.2 for proof).
This implies that the model camnot learn and improve its own without external inputs, highlighting the necessity for our method, which will be introduced in the following sections.
3.3 IN-CONTEXT TUNING WITH SAMPLING
To address the ineffectiveness of the naive sampling approach in 3.2, we introduce extra information that guides the model towards a new distribution that aligns with the target, while maintaining similarity to its original distribution. Specifically, leverage the in-context learning capabilities of
4

---

Published as a conference paper at ICLR 2025
language models by prepending context prompts the queries q. where is the new knowledge to be learned. For our example, we can create a context such as "Donald Trump the current president of the United States.", and prepend this context to the query. This induces new contextual distribution pa, (x|[e, q)) that incorporates the desired knowledge through the context, while keeping minimal changes the model. We can define the loss function as:
Cample = DRI. (po, (xl[c, q)) po(x(q)),
where Og are the initial parameters of the model, and pa pa(x (x/q) is the updated model's distribution. In this formulation, there no explicit target sequence x"; instead, the desired information is implicitly conveyed through the context c. The effectiveness of this method relies on the relevance of the context and the model's ability to utilize it effectively.
3.4 CONSISTENT IN-CONTEXT EDITING (ICE)
While the loss Simmpl introduces context, it does not guarantee that the model will produce accurate responses, the initial distribution Pe, [x[e,, q)) may not reflect the correct target due to limitations in the model's ability to follow the context. Therefore, we propose to refine the target contextual distribution in a way that ensures the model internalizes the new knowledge.
We introduce a consistency condition that the updated model parameters 0 should satisfy:
p [c, q)) Pa (x
This condition implies that, after updating, the model's predictions should be the same whether or not the context e is provided, indicating that the knowledge from c has been internalized. To enforce this, we define the in-context editing loss LICE as:
LIC = DKL (Po x/|[c, ql) po
(5)
To ensure the model produces the correct target sequence x", we also include the fine-tuning loss:
- LTT + XLICE,
where À is a hyperparameter balancing the two loss terms.
Optimizing LICE: The in-context editing loss LICE involves two distributions that depend on the model parameters f making direct optimization challenging. Directly propagating the loss through both distributions is not desirable,
aim for a uni-directional optimization: do not intend to draw Po (x(x [c. q]) towards (x(q). To address this, we adopt an iterative, gradient-based approach. At each optimization step s, treat po, (xlfc, q)) fixed target distribution (using the current parameters f₂) and update the model parameters to minimize the divergence to pe.. (x|q). This process is formalized as:
f+ arg min Cloè arg min DxL (po, (x|lc. q]) || po... (x/q)). 0,11
(7)
0.
By iteratively updating we ensure that the model's predictions with and without the context converge, satisfying the consistency condition in Equation 4.
Optimizing the Combined Loss L: To optimize the total loss as defined in Equation 6, we sample sequences X from the model conditioned on |c. q. x.x and maximize the likelihood of the combined sequence [x*,xx xe]. This process is equivalent not conditioned on the target, we would be solely optimizing LICR- This approach is algorithmically convenient, and we provide proof of this equivalence in A.3. To prevent the model from drifting too far from the initial parameters (thus preserving unrelated knowledge), employ gradient clipping techniques inspired by constrained fine-tuning methods [59]. The detailed algorithm is presented in Algorithm 1.
optimizing the combined loss C. If the sampling is
The context c can be generated automatically by extracting or synthesizing relevant information related to the target knowledge. In practice, this can be achieved using language models or APIs to generate summaries or statements that convey the new information. our experiments, used GPT-4 create effective contexts (details provided in $ C).
Context Generation:
5

---

Published as a conference paper at ICLR 2025
3.5 DISCUSSION
Our method aims to achieve several objectives simultaneously. The accuracy of our method is ensured by the fine-tuning loss LrT, which requires the model produces the correct target output x" for the query q. The linguistic quality is maintained by the in-context editing loss LICE. which encourages the model to align output distribution with broader, context-inducen distribution, helping prevent overfitting and maintaining the naturalness and diversity of the generated text.
To understand how our method maintains locality (i.e., minimal impact on unrelated queries) and promotes generalization, we consider all possible query-response pairs (q, and partition them into two sets: those related to the target knowledge (D₂) and those unrelated (D-g). The in-context editing loss can be decomposed as:
LICE DKu(po(x[[ex, al) po(x
[ (l)))) po(x
Σ
po(x][c, l)) log
For queries unrelated to the target knowledge (i.e., q D ....). the context should have minimal effect, so the loss encourages the model to keep its original responses, ensuring locality. For related queries (q D,), the loss promotes generalization, as the model learns apply the new knowledge to various relevant contexts. Thus the effectiveness of our method relies on several assumptions:
The context is related to the knowledge. The context provided in the prompts must be pertinent and relevant to the knowledge needed for generating accurate and coherent responses. This relevance ensures that the additional information introduced through the context meaningful and enhances the model's understanding of the query.
The model attends to the context. The model must be capable of attending and incorporating the contextual information provided in the prompts. During the fine-tuning process, the model effectively uses the context as part of its input, influencing its predictions and overall performance.
The model generalizes from the context to related knowledge. Given the relevant context, the model should be able to generalize from the specific information in the context to broader or related knowledge areas. This generalization enables the model to generate responses that are not only contextually coherent but also enriched with additional details inferred from the context. Techniques like thain-of-thougeee [50] can potentially be employed in the context prompt enhance the model's generalization capability.
The computational demands of our pipeline can be heavier than vanilla fine-tuning, it involves multiple sampling steps and depends on GPT-4 for context generation. However, the computational burden may not be substantial as it appears: 1) Since sampling only necessitates a forward pass of the model, the computational cost is significantly lower than that of training the model. 2) We are considering scenarios with very limited training data, as is the case in the knowledge editing task.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Datasets and Model We evaluate the performance of ICE with four datasets from KnowEdit [55], which are commonly used for knowledge insertion and modification. Detailed statistics on the selected datasets can be seen in Table Among the datasets, the WikiBio dataset does not include related hopping question data necessary for evaluating the portability metric. To ensure a fair comparison, we use Hama2-7b-chat, which is the same model as used in the original survey [55].
Metrics We use the metrics from 2.1 but note one limitation in not penalizing semantically meaningless sentences repetitive long patterns ($ D). Hence we add perplexity an additional measure, which measures how well the pre- pre-trained model predicts the generated outputs from the fine-tuned models. Assuming the original model is well-trained, the perplexity score reflects the language quality of the fine-tuned model and how far it has drifted. our case, perplexity can also increase due the novelty of edited knowledge,
introduce normalized perplexity ratio PPI.,

---

Published as a conference paper at ICLR 2025
Table I: Statistics on the evaluation datasets.
Knowledge Insertion
Knowledge Modification
WikiDataycont
7sRE
WikiBio
WikiData.conterrfat
Type Train Test
Fact
QA
Hallucination
Counterfact
570
10,000
592
1,455
1.266
1230
1.392
885
Table 2: Main results on knowledge insertion and question guestion-answering datasets of 1lama2-7b-cher
WikdData,
ZaKP
Edit Suce.
Portability
Locality
Fluency
PPL.
Edit Suce.
Portability
Locality
Fluency
PPL.
ROME MEMIT
36.58
581.00
107.47
96.66
97.03
37.00
29.28
573.06
87.17
95.61
52.73
24.79
563.42
38.67
FELL 45.63
34.73
34,80
558.91
68.92
43.68
43.90
51.38
560.94
30.36
FT-M
100.00
59.28
587.17
70.64
100.00
580.10
ICE
100.00
61.02
46.39
585.58
34.08
100.00
55.52
56.97
562.70
15.50
to address this ($ D.1). The ratio compares the perplexity of the generation post-target token to that of the combined prompt and target token.
Methods We use 4 representative tuning methods for comprehensive comparisons. ROME [28] and MEMIT [29] employ a causal method to locate and edit only the related parameters to improve the locality. The other two methods FT-L [28] and FT-M [55] fine-tunes specific layers of the feed- forward network to maximize the probability of all tokens in the target sequence. In the survey [55], the FT-M model demonstrated nearly the best performance.
Implementation details The contexts e are given by GPT-4 by summarizing the target knowledge. For layer updates, ROME updates one layer for GPT2 with layer 17 and Llama2 with layer 5. For both ICE and other baselines (FT-M and FT-L), five layers are updated following MEMIT [29], for GPT2 with layers 1,11111,16,11. 17 and Llama2 with layers 4.5,6,7.8 7.8. Results of GPT2 can be found at D.3. We follow the usage of other parameters in ROME and MEMIT which have been found to provide the best performance. For FT-M, FT-L, and ICE, the optimization proceeds for maximum of 25 steps with a learning rate 7e 4 and 0 weight decay. For all results except the ablation study, used 1.0 for ICE without deliberate tuning.
4.2 MAIN RESULTS
Table 2 and Table show the main performance metrics of ICE. Notably, the FT-M method remains the strongest baseline, as corroborated by the findings in [55]. As seen in the results, ICE demonstrates outstanding performance on the measures.
Accuracy ICE consistently achieves nearly perfect edit accuracy across all datasets, outperforming most baselines and matching the performance of the strongest baseline FT-M.
Locality and portability As accuracy increases, the locality tends to decrease due to the inherent perturbations introduced. Furthermore, there tends be inverse relationship between model locality and portability; locality implies minimal model changes, whereas portability necessitates the model's ability generalize related knowledge. Despite this trend, ICE not only achieves a near- sear-peeffett accuracy comparable to FT-M but also consistently outperforms bascline methods in terms of locality and portability, aligning with the analysis presented $.5.. While matching
the near-perfedt perrfct accuracy with FT-M, ICE demonstrates consistently better locality and portability than the baseline methods, matching our expectation discussed in $ 3.5. Compared to ROME, MEMIT, and FT-T, ICE shows approximately 30% higher portability on the WikiData.ounter and WikiData,ecent datasets. This discrepancy highlights that by leveraging in-context learning to adapt to contextual distribution, ICE achieves better generalization. Additionally, ICE performs over 15% better in terms of locality on both datasets, preserving unrelated knowledge by enhancing the robustness of gradient-based tuning. A minor performance degradation of 99.88% is observed
7

---

Published as a conference paper at ICLR 2025
Table Main results on knowledge modification datasets of Llamaa-7b-chat
WikiBio
WikiDaafaeswwttr/ffrr
Edit Succ.
Locality
Fluency
PPL.
Edit Sacc.
Portability
Locality
Fluency
PPL.
617.67
MEMIT
94.54
69.96
616.65
3.51
98.13
44,16
19.48
576.26
122
FE-L
59.41
28.94
615.50
36.13
29.37
38.37
566.55
89.24
FT-M
100.00
35.34
618.12
3.67
100.00
72.39
40.76
$86.80
54.71
99.88
617.88
45.88
583.29
18.95
the WikiBio dataset. This could be attributed to the diversity across datasets, which can introduce slight variations in performance within an acceptable margin.
Fluency and PPLr To evaluate the linguistic quality, computed fluency and perplexity. ICE demonstrates reasonably good fluency, frequently ranking among the top performers. While other methods might show slightly higher fluency in single edits, ICE achieves significantly higher fluency in the continual editing case ($ 4.4). Moreover, ICE consistently exhibits lower perplexity, signaling better and more natural language model performance. It maintains robust performance across all metrics when editing new knowledge while preserving the integrity of existing information.
Table 4: Ablation results. The second row is the closest to fine-tume fine-tuning ($ 3.1 and $ 3.2).
ZaRE
WikiiDataooseer//oo
Dynamic
Context
Edit succ..
Portability
Locality
Flueney
PPL.
Edit surx.
Portability
Locality
Fluency?
PPL.,
100.00
55.52
56.97
562.70
15.50
100.00
73.49
45.88
$83.29
8,92
99.60
45.95
55,40
$64.55
6.90
99.66
67.34
48.42
S68.95
12,50
589.97
31.71
x
99.94
53.84
65.64
578.97
25.71
99.93
55.12
589.04
35.70
Table 5: Ablation results for different values of A.
ZsRE
WikiData,
A
Edit succ.1
Portability
Locality
Fluency
PPL.,
Edit
Portability
Locality
Fluency
PPL.
0.6
99.71
50.65
59.54
584.84
561.29
99.93
58.28
46.93
589.77
179.4
0.8
99.81
51.59
58.82
582.91
281.14
99.95
59.12
47.36
591.84
142,03
1,0
100.00
55,52
56.97
562.70
15,50
100.00
61.02
585.58
34.08
1.2
99.87
52.08
58.54
581.04
324.47
99.98
59.69
47.51
589.49
280.32
58.18
584.17
4.3
ABLATION STUDIES
We examine important dimensions of ICE through our ablation experiments in Table 4.
Firstly, we analyze the impact of using a dynamic training target. Specifically, we investigate whether sequences generated from the original model throughout training from modified model. In other words, in the first variant of our algorithm, the target distribution pe, (xl[c, ql) in Equation 7 remains static during optimization, meaning the weight the with-context target distribution does not change, i.c., Po, (x|[c. q)) = Po, (x|[c, q) for 2 O. Notably, ICE with static targets is equivalent to combining Csample and Ctt. Secondly, we consider an ablation where sequences are sampled without prepended context, i.e., sampling from po (x(qq) instead of (x [c, ql).
In this ablation, the model that closest fine-tuning with sampling and thus vanilla fine-tuning (S 3.2 and 3.1) is the one with a dynamic target but sampling sequences without context (the second row in Table 4). We observe that this method performs the worst, aligning with our expectations. Notice that when both modules are off, the model significantly differs because it samples sequences from the initial model and uses that as a target distribution constrain the edited model.
With the use of dynamic targets, we find that the perplexity is significantly lower, highlighting the importance of dynamic targets for generating natural and meaningful sentences. When comparing results with and without context, ability. These ablation results confirm the importance both dynamic training targets and the inclusion of contextual information in ICE.
that adding context generally improves generalization
8

---

Published as a conference paper at ICLR 2025
Table 6: Continual editing results of Llama2-7b-chat.
DataSet
Metric
MEMIT
ROME
FE-1
FT-M
ICE
DutaSet
Metric
MEMIT
ROME
FI-L
FT-M
ICE
Edit suoe.
14.20
17:42
64.55
108.00
160,06
12.10
14.28
Porability
4.06
6.46
23.93
58.30
59.27
Postability
Locality
4.32
35.59
Wild
Locality
0.78
1:34
1.01
27.37
Fluency
Plaency
416.77
294.07
472.37
314,85
599.57
PFL.
22.57
11.58
PPL..
7.71
6.12
0.10
10.74
0.10
Edit suce.
Edit succ.
26.49
8.31
38.02
99.09
99.09
Porabillity
Locality
2.13
2.96
34.15
WikiBio
Fluency
Localiy
599.00
3,73
497,42
4.34
595.31
13.20
617.90
29,40
612.66
30.17
Fluency
509.36
313.28
464.30
590.
602.33
536.35
1.07
PPL.,
14.44
3.43
0.34
0.93
0.07
success
Lucality
Flauncy
Pertabilliyy
600
30
30
Nanterr contlnall edilin
Nusbber wwn.inaal LOIing
Naaterra wwn.lnallllllo.mmmooooooooo....e cdiling
Namther contintal CdItnn
1CE ROME
FT-M
Unudied
Figure 2: Continual editing with Llama2-7b-chat on WikiDatayecend Each edit builds the previous model, risking deterioration over time. The model is assessed immediately after each edit without re-evaluating previous edits, testing its ability to update continuously. While most methods deteriorate, sometimes performing worse than the unedited version, our method, ICE, maintains integrity and achieves promising performance.
Furthermore, examine the influence of the hyperparameter À as detailed in Equation 7. The results presented in Table indicate that simply setting to 1.0 yields the optimal performance for the model, which corresponds to directly maximizing the likelihood of the combined sequence of the target and the sampled sequence.
4.4 CONTINUAL EDITING
We also evaluate the model's ability to maintain its integrity. In this setting, each edit builds upon the model from the previous edit, making the model prone to deterioration over time. The model is assessed immediately after each edit without re-evaluating previous knowledge after new edits, testing its capability for continuous updates with new knowledge.
Figure illustrates the model's performance during continual editing. Most baseline methods (e.g.. MEMIT, ROME, FT-L) experience significant deterioration in both accuracy and general performance time. This trend is especially evident more updates applied, leading to issues such as catastrophic forgetting and decreased locality in model responses.
Table 6 presents the results of ICE across all four datasets. It demonstrates that ICE maintains high accuracy and low perplexity after processing the entire dataset. The model's integrity is preserved, as indicated by the fluency and PPL., metrics remaining consistent with the basic knowledge editing scenario, indicating promise for continual editing. Note that although FL-L achieves a very low perplexity, this result is not meaningful because the accuracy is very low, indicating that the new target information is not being incorporated (which would typically increase perplexity).
4.5 CONVERGENCE
As the target distribution dynamically evolves during optimization, ensuring the convergence of Algorithm crucial. Another consideration how ICE differs from combining in-context sampling ssamll and fine-tuning Lst. To investigate this, further examine the static target ablation.
The left side Figure presents the loss curves over optimization steps for a range of temperatures. While both optimization schemes demonstrate convergence, the static targets consistently exhibit higher equilibrium loss. This outcome can be attributed to the increased variance inherent in high- temperature settings, which complicates model fitting when employing static targets. In contrast,

---

Published as a conference paper at ICLR 2025
Static Target
50
0.1
Caroed
1-0
30
20.0
20
611
60
Dynamic Targst
Strp
Strp
Step
temperature
Ype Au Contect
40
100
10
6
100.0
20
10
Step
Figure 3: Comparison ICE with static and dynamic targets on an example, where the query "The name of the country which Academy Award for Best Picture is associated with is?" and the target is "Wassoulou Empire". The line plots on the left illustrate the loss over optimization steps for static (top) and dynamic (bottom) targets under temperatures from 0.1 100. The figures the right show how the probabilities of the top-6 predicted tokens for 22, the second token following the target, change with iteration steps. The tokens are arranged from left to right in descending order of probability without context. At early steps, the token "Wass" appears due to its presence as the initial token in the target x*. At later steps, probability of "Wass" in dynamic targets (top) significantly declines, indicating successful adaptation and suppression of repetitive token predictions. In contrast, for static targets (bottom), the probability of "Wass" remains relatively high throughout the optimization steps.
dynamic targets facilitate an iterative refinement process, enabling the model predictions and target distributions to progressively align, thereby achieving a lower equilibrium loss.
The right side of Figure 3 provides further insights through example where dynamic targets foster a more effective adaptive adjustment of token predictions compared to static targets. Specifically, dynamic targets reduce the frequency of repetitive token patterns over the optimization steps, whereas static targets maintain higher probabilities repetitive tokens. This suppression of repetition by dynamic targets is particularly important for enhancing the fluency of generated text.
CONCLUSION
This paper introduces In-Context Editing (ICE), a novel approach that addresses the brittleness of traditional fine-tuning in knowledge editing by targeting a contextual distribution instead one-hot target. ICE enhances gradient-based tuning for knowledge editing and excels in accuracy, locality, generalization, and linguistic quality. Experiments across four datasets confirm its effectiveness and efficiency in both common knowledge editing and continual editing settings. Overall, ICE offers fresh perspective and a straightforward framework for knowledge editing of language models.
LIMITATIONS
While the use of alternative models for context generation is optional, we employ them to enhance the training process with additional information. However, if the context generation model (c.g., GPT-4) produces hallucinated outputs, may provide inaccurate contexts, which could hinder the optimization process and lead to further hallucinations. In experience, since using the model exclusively for paraphrasing, we have not encountered any instances of hallucination.
ACKNOWLEDGEMENT
This work is supported by the Opening Project of the State Key Laboratory of General Artificial Intelligence (Project No:SKLAAG220200111
10

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36, 2024.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 719-730, 2022.
[4] Yingfa Chen, Zhengyan Zhang, Xu Han, Chaojun Xiao, Zhiyuan Liu, Chen Chen, Kuai Li, Tao Yang, and Maosong Sun. Robust and scalable model editing for large language models. arXiv preprint arXiv:2403. 17431, 2024. 3
[5] Siyuan Cheng, Ningyu Zhang. Bozhong Tian, Xi Chen, Qingbin Liu, and Huajun Chen. Editing language model- knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17835-17844 2024. 3
[6] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-200 2983, 2024. 3
[7] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint wxXiv2210.086900 2021. 3
[8] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491-6506, 2021.
[9] Qingxiu Dong. Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5937-5947, 2022. 3
[10] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer are key-value memories. arXiv preprint 20X. 2020. 3
layers
[11] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
[12] Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. unified framework for model editing. arXiv preprint arXiv:2403. 14236, 2024." 3
[13] Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense transformers. Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 8214-8232, 2023. 3
[14] Tom Hartvigsen, Swami Sankaranarayanan. Hamid Palangi. Yoon Kim, and Marzych Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
[15] Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zomitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint orXiv:2 / 13654, 2021.
[16] Jason Soeelscher-Obermalee Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 11548-11559, 2023. 3
11

---

Published as a conference paper at ICLR 2025
[17] Youcheng Huang, Wenqiang Lei, Zheng Zhang, Jiancheng Lv, and Shuicheng Yan. See the un- seen: Better context-consisten mhowledge-editingg by noises. arXiv preprint arXiv:2401.07544 2024. 4
[18] Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212. 10670, 2022.
[19] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-paccher: One mistake worth one neuron. In The Eleventh International Con- ference on Learning Representations, 2022. 3
[20] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2022.
[21] Fred Jelinek, Robert L Mercer, Lalit Bahl, and James K Baker. Perplexity a measure the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977. 18
[22] Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang. Lifeng Shang, Ruiming Tang, et al. Learning to edit: Aligning Ilms with knowledge editing. arXiv preprint arXiv:2402.1 1905, 2024." 3
[23] Kyungjae Lee, Wookje Han, Seung-won Hwang, Hwaran Lee, Joonsuk Park, and Sang-Woo Lee. Plug-and-play adaptation for contimuously-updated qa. arXiv preprint arXiv:2204. 12785, 2022.
[24] Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1774-1793, 2023. 3,4
[25] Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18564-18572, 2024. 3
[26] Yu Ma, Jia-Chen Gu, Zhen-Hua Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310.10322 2023. 3
[27] Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing improve gpt-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
2022. 3
[28] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 17:7722-177372, 2022. 3, 18
[29] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. arXiv preprint arXiv:2210.07229) 2022. 3,
[30] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
[31] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022. 3
[32] Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024.
[33] Shikhar Murty, Christopher D Manning, Scott Lundberg, and Marco Tulio Ribeiro. Fixing model bugs with natural language patches. In Proceedings of the 2022 Conference on Empirical Methods in Natural I Language Processing, pp. 11600-1166 1613, 2022.
12

---

Published as a conference paper at ICLR 2025
[34] Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. arXiv preprint arXiv:2311.08011 2023. 3
[35] Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. In Proceedings of the 61st Annual Meeting of the Association for Computationall Linguistics (Volume 1: Long Papers), pp. 5469-5483 2023. 3
[36] Vikas Raunak and Arul Menezes. Rank-one editing of encoder-decoder models. arXiv preprint arXiv:2211. 13317, 2022.
[37] Amit Rozner, Barak Battash, Lior Wolf, and Olir Lindenbaum. Knowledge editing in language models via adapted direct preference optimization, arXiv preprint arXiv:2406.0999 09920, 2024. 3
[38] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. Advances in Neural Information Processing Systems, 34:23359-23373, 2021
[39] Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in mamba. arXiv preprint arXiv:2404.036m 2024. 3
[40] Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. Retrieval-enhanced knowledge editing for multi-hop question answering in language models. arXiv preprint arXhv:2403.19631, 2024. 2024,
[41] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2019.
[42] Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209.15183 2022.
[43] Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language model via meta learning. In The Twelfth International Conference on Learning Representations, 2023. 3
[44] Ryutaro Tanno, Melanie F Pradier, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind. Advances in Neural Information Processing Systems, 35: 13132-13145 2022. 3
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
(tl::3307.09888 2023. 18
[46] Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint 2403 14472, 2024.
[47] Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405. 144. 2024. 3
[48] Yiwei Wang, Muhao Chen, Nanyun Peng. and Bei-Wei Chang. Deepedit: Knowledge editing as decoding with constraints. arXiv preprint arXiv:2401.10471 2024.
[49] Yu Wang, Xiusi Chen, Jingbo Shang, and Julian McAuley, Memoryllm: Towards self-updatable large language models. In International Conference on Machine Learning (ICML), 2024.
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444
2022.
[51] Xinwei WU, Junzhuo Li, Minghui Xu. Weilong Dong, Shuangzhi Wu. Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
13

---

Published as a conference paper at ICLR 2025
[52] Yang Xu, Yutai Hou, Wanxiang Che, and Min Zhang. Language anisotropic cross-lingual model editing. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 5554-5569, 2023.
[53] Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pp.
2023. 3
[54] Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron- indexed dynamic lora. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19449-19457, 2024. 1,3
[55] Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang. Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models, 2024. 1, 6, 7, 18
[56] Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximization. Advances in Neural Information P Processing Systems, 31, 2018. 3
[57] Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang- Can we edit factual knowledge by in-context learning? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4862-4876, 2023. 2,
[58] Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Assessing knowledge editing in language models via multi-hop questions. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 15686-15702, 2023.
Mquake:
[59] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363 2020. 1, 3,3,
14

---

Published as a conference paper at ICLR 2025
A IN-CONTEXT EDITING
In this section, we provide the proof' for Observation in the main text in A.1 and $ A.2.
A. PROPERTIES OF ONE-HOT DISTRIBUTION
The one-hot distribution, denoted (x), is distribution defined on sequence (z), Z2 T2,,,.; where y = (y) (91.3/2.... 9/..... yn) is sequence of the same length that represents the target or desired sequence. The one-hot distribution is product of Kronecker deltas, as follows:
8y (x)
=
II
8(0; ve),
i=1
where each Kronecker delta follows the definition below
I; =
8(x;, 3/4
0
if
+ y
The following lemma is trivial but handy in deriving formulas involving one-hot distributions. Lemma 1. The expectation value of measarement function f(x) on one-hot distribution 6y (x) with target y equals to the measurement on target f(y), i.e.
Σ (x)xx((()) S....5yye
f(y).
(9)
Proof. Since the only possible outcome of sampling x is y, the expectation of measurement is trivially f(y). Mathematically,
Σ (x)
Σ (x(xx((xx
X
= =1-((y) + Σ 0. (xx==
= ((v)-
x#y
This seemingly trivial lemma is useful in subsequent proofs in that we may substitute all occurrences of variable x with target y after summing x with one-hot distribution 8y (x).
A.2 THE INEFFECTIVENESS OF NAIVE SAMPLING
The fine-tuning objective LFT is the Kullback Leibler (KI.) divergence between the one-hot distribu- tion &x- (Fim) and the model's predicted distribution po (o q)) which is defined as
LET = DKL (6x- (I1m) po(1mm/)))
(10)
where &x- (f)m)) is the one-hot distribution.
By substituting the definition of KI. divergence into the fine-tuning loss Toft given in Equation 10, obtain
dx (111:))
CFT
6xx (x log
po (p
and by applying Lemma 1, we obtain
L - - Po (x (q):
which aligns with the maximum likelihood estimation (MLE) objective. For fine tuning with sampling, objective Lj, is expressed as:
C DRL (8x- (#1 pp (xm-1171 (q, x)) Pe (11rl9)) where Fir is the sequence truncated at length T. All proofs still hold for T To illustrate ane-tuning with sampling does not alleviate over-fitting, we prove Observation 1, i.c. LrT L
(11)
15

---

Published as a conference paper at ICLR 2025
Proof. Expanding Equation 11 using the definition of KL divergence reveals that:
FTT
Σ
xx (F1:m Pe ( (Im+1:Tl [q. x4)x
8x (f1sm) Pg (#m+1:1 [q, x*))
log
(T1:r (q)
8x- (F1m po Fm+11:1 [q. xx,x"])
Mi:m
Rm+i:r
8x- (11 Pn
1:7 x
llog
po (#1:m (4) po 1:1+
q T1:m.)
Σ
Py (mm+ 1:2 [q,x") x")
Sm 1:2
1 Po (Tm+1:7 [q.x"))
log
po (x) Po (fm+irllqgx xx))
log po (x (1)
DKL Po (Im+1rrlq x*]) lIpo (In+w:r [q,
LFT,
(12)
and from line line we apply Lemma and substitute all occurrences of Fiom with x".
Consequently, sampling through self- -effraeerrttoon without external inputs does not alleviate the prob- lem of over-fitting. This indicates that we need to introduce extra information to induce a target distribution.
A.3 DECOMPOSING CONSISTENT IN-CONTEXT EDITING
The objective of consistent in-context fine tuning Equation is given as -= LFT + XLirr- this section, we demonstrate that when À = 1, this ojective is equivalent to sampling sequences from the model conditioned on [c, q x*] and maximize the likelihood of the combined sequence [x]:
First, it is straightforward to show that given samples from distribution ((x), maximizing the likelihood x for (o() is equivalent to minimizing the KL divergence between pa and q(r):
argmax)E.=0(x)) [log po (x)] argmax, (Ez=g(z) [log q(x)] E Ex (og)))
argming D (ps (x) [g(x)).
Therefore, maximizing the likelihood of the combined sequence (x", xe] is equivalent to minimizing the KL. divergence between po (#1.7 (q) and &x- (#1.m )po (fm+1:rllqg.... x
L = DKL xx (Ii:m )p (Im+1 [c. x"I) || pe (21rl4)).
(13)
which may be expanded using the definition of KL divergence as
.log 8x- (11: Po (fm+1:Tl cc, q, q,xx)
C 5ox
Sx- (I1:))P (Im+1rl [c, q.")
pa (x1.149)
#1.7
8x- (x Am+1:r/ [cc, q. x'))
Σ Σ
(x-:m1)))p (Im41al q.1"))
-log pa (z1mlq) 1m pe (Im+1rllqq 21:m) 21mm)
Ml Pm
(14)
Using Lemma 1, we may further simplifies L as
1 po (Xm 1.ll (c, xx)")
L Σ po (mm+1:T [c. q, x'I) log
po (x* q) po (Im+1:7 x"])
1:+1
log (x*{)) + DEI (po (2m41.l (c. q. xx)x lipe (2 (fmt1rllq
(15)
LFT + LIcE (q.
where the second term is the consistent in-context editing loss Lice with the substitution q <-
<<q.x"x" (q.
16

---

Published as a conference paper at ICLR 2025
Algorithm 1: Consistent In-Context Editing (ICH)
Data: Initial model parameters On, context c, query q, target sequence x", learning rate f)
maximum iterations S
Result: Updated model parameters 0
for 0 to S do
2. Sample in-context sequences: X ~ pe, (x|[e, q, x²I
Compute gradient: 80,-
Vo, (xx (x)po, (x([e, q. xeeaccetachh)) || po, (x(q))
4
= <Vexx.1- Log Pe, ([x*, x))
Clip gradient: 80, clip (60, tg)
Update parameters: 0s+1 + 02 - X) 80,
if convergence criterion met then
break
end end
11 return 0s
B ALGORITHM
Here, we provide the algorithm of in-context editing (ICE) in Algorithm
C CONTEXT GENERATION CODE
This section presents the core logic for context generation in Listing C, utilizing language model to provide relevant context based on given query and target. Listing C shows some examples of contexts generated.
Context generation code
def generate_context (prompt, target):
Generate context for a given query and target using language model (e.g., GPT-4) to generate context.
msg f"Please help me generate five complete statements as context]s according to the semantics of incomplete facts '( prompt)' and "(target)'." context = language_model generate (msg) return context
Example contexts
"prompt": "The mother Mallory Reaves is whom?", ground_truth". "Brynne Chandler"
J.
"target_new"; "Lalli Reaves", "context":
"Mallory Reaves's mother is Lalli Reaves.",
"Lalli Reaves is the mother of Mallory Reaves." "The mother of Mallory Reaves is identified [alli Reaves
"In terms of parentage, Mallory Reaves's mother is Lalli Reaves.",
"Mallory Reaves is the child of Lalli Reaves."
17

---

Published as a conference paper at ICLR 2025
D EXPERIMENTS
D.1 METRICS
We employ the metrics described in 2.1. However, we highlight an issue inherent in these met- rics. The fluency metric, defined - -n= Wn Σx fn log fn fx),, is based on bi- and tri-gram probabilities and fails to penalize semantically unmeaningful sentences or repetitive sequences (Table 7).
To address this limitation, we suggest incorporating Perplexity [21] as an additional measure, which quantifies how well probability model predicts given sample. Specifically, for token sequence x (11, 22, ,,, FT) and a measuring model M (e.g., Llama2 [45]), the perplexity is defined as
PPL = exp (- E t=l log pog
A lower perplexity value indicates that the model generates more natural and fluent language outputs.
One caveat is that perplexity can also increase due the unexpectedness of the edited knowledge. To account for this, we introduce normalized perplexity ratio, comparing the perplexity of the generated sentence beyond the target token to that of the prompt and target token combined. This ratio is computed as:
PPI(((q..'))
PPL. =
(16)
PPL(q..*)
Table 7: The fluency metric might not detect repeating sequences or non-sensical sentences.
Example Sentence
Fluency PPL..
What is the name of the current head of the United States of America government? The current head of the United States government the President, who is Joe Biden. He was inaugurated on January 20, 2021 and serving his second term in office.
538.34
6.08
What is the name of the current head of the United States of America guvernment? What the name of head the United States government? What the title of the current head name of the person leading the United States govermmen?? The
538.89
$2.03
government? What the current head of the United States governmeett is President Joe Biden. He was inaugurated as the 46th President of the United States on January 20, 202
Which city is the headquarter of Google located in? A) Seattle B) Mountain View C) New York D) Los Angeles Answer: B) Mountain Viewi Google's 's headquarters is located in Mountain View, California, USA.tt
527.31
3.06
Which city is the headquarter Gongle located in? Answer: Google's headquarters is located in Mountain View, Califirmia, tSr. Google's headquarters is located Mountain View, California, util. "public class Solution public static void main (String]] args) Scanner Scanner System.
$72.03
39.72
USA. "iimpott java.util.
D.2 IMPLEMENTATION DETAILS
ROME and MEMIT ROME and MEMIT employ causal tracing results to pinpoint the layers in the model responsible for recalling facts that need updating. Specifically, ROME updates GPT2 at layer 17 and Llama2 at layer 5. contrast, MEMIT updates GPT2 with layers to 17 and Llama2 with layers 4 to 8. Additionally, we calculate covariance statistics for ROME and MEMIT using sample of 100,000 entries from the Wikitext in fp32 format. Further implementation details can be seen in [28].
FT-L and FT-M In the case of FT-L and FT-M, we follow the updated layers as outlined in MEMIT update multiple layers for improved performance. As for difference between these methods, FT-L diverges from the original fine-tuning loss objective by utilizing the last token's prediction to maximize the probability all tokens in the target result. Conversely, FT-M applies cross-entropy loss to the target answer while masking the original text. See [55] for more detailed implementation.
18

---

Published as a conference paper at ICLR 2025
ICE In ICE, also follow the MEMIT setting for selecting layers update. For each update, sample five in-context sequences from the model to compute the target distribution. Additionally, we constrain the model updated weight to be within =5e 4 of model weight before updating. Thus, can ensure that the model's inherent knowledge not excessively altered to certain extent.
Computing resource All methods can run on a single Nvidia A100 80GB GPU with 32GB memory and a 128 128-core AMD CPU.
D.3 MORE MAIN RESULTS
Table and Table present the results for the four datasets using GPT2-xl. Overall, the improvements less pronounced compared to those observed with Llama2-7b-char This outcome is expected, as ICE designed to perform better with models that have stronger in-context learning capabilities.
Table 8: Main results on knowledge insertion and question-answering datasets of GPT2-xl.
WikdData,coot
ZsKF
Edit Suce.
Portability
Locality
Fluency
PPL.
Edit Suce.
Portability
Locality
Fluency
PPL.
ROME MEMIT
30.74
316.38
78.31
24.97
32.73
600.54
6.79
67.39
41.45
80.84
591.98
5.54
FE-I 63.99
29.03
61.45
591.86
25.26
64.25
42.51
32.18
571.71
12.40
FT-M
100.00
36.74
604.07
35.81
100.00
48.41
31.39
583.63
17.28
ICE
100.00
35.76
63.45
560.96
7.91
99.92
16.84
34.44
554.74
9.10
Table 9: Main results on knowledge modification datasets of GPT2-xl.
WikiBio
WikiData Tokitrtttpposter/fur fuct
Edit Suce.
Locality 1
Fluency
PPL.,
Edit Suee.
Portability
Locality
Fluency
PPL.,
ROME MEMIT
81.52
27.49
633.41
2.11
96.08
31.31
16.67
600,94
3.71
57.31
39.65
632.89
1.87
55.03
20,00
24.79
604.58
11.10
FT-L
51.59
66.66
626.17
1.16
40.85
20.09
65.29
596.77
26.55
FT-M
100.00
63.87
631.3
2.12
100.00
42.08
63.43
602.91
12.64
ICE
100.00
63.67
629.60
2.22
100.00
39.78
67.06
560.33
8.05
D.4 MORE ABLATION RESULTS
Dynamic Target and Sampling with Context We conducted ablation experiments explore two key aspects of ICE on GPT2-xl in Table 10. 1) We assessed the impact of using a dynamic training target. 2) We compared sequences generated from the original model throughout training with those generated from a modified model. Secondly, we examined the effect of sampling sequences without prepended context.
Table 10: Ablation results of dynamic target and sampling with context using GPT2-xl.
ZsRE
WikiDats
Dynamic
Contest
Portability
Locality
Flucacy
Suce.
Portabillity
Locality
Flaeney
99.92
66.84
34.44
$56.74
9:40
100
39.78
67.06
560.33
$.05
x
99.96
44.92
36.13
582.58
6.54
100
37.52
09:74
594
7.81
99.98
44.54
38.61
596.33
1.36
99.99
37.35
72.78
593 08
3.64
100
45.54
39.88
592.87
5.55
100
37.65
73,82
599.37
5.21
Performance Using Different temperature In the analyses presented in Table 11, it is evident that as the temperature setting increases from 0.1 100, the edit success rate escalates 100% at the highest temperature. Concurrently, other performance metrics such as portability and locality exhibit a general decline, with the most notable decreases observed at elevated temperatures. Fluency tends to improve when the temperature is maintained below 10, while the PPL metric decreases significantly, reaching a low of at temperature of 100. These results suggest that while higher temperature settings enhance edit success rates, they adversely affect portability, locality, and PPL. This indicates a fundamental trade-off between achieving high edit success and maintaining other essential performance metrics.
19

---

Published as a conference paper at ICLR 2025
Effect on Different Sample Length Table 12 presents the performance of the Llama2 model using variable sentence sample lengths (3, 5, 10) on the ZsRE and WikiDatt datasets. The model demonstrates optimal performance in edit success and portability at sample length of 5 for both datasets. Across all sample lengths, edit success remains consistently high, exceeding 99% and reaching 100%. However, portability could exhibit decline as sample length increases, with a decrease of approximately 5% at sample length 10 for both datasets. In contrast, metrics such as locality and fluency initially decrease with longer sample lengths but exhibit slight improvement at sample length 10. For the linguistic quality, PPL,, exhibits marked decline as sample lengths are augmented, aligning with our predictions. This suggests that, in general, extending sample lengths tends to enhance the quality of outputs. However, there may be trade-off in how the model generalizes to different contexts.
Effect Numbers of Samples Table 13 indicates that edit success remains exceptionally high across all sample sizes, with only a marginal fluctuation observed at sample sizes of and 15. Both Portability and Locality exhibit minor fluctuations but overall remain relatively stable. These results along with the sustained high levels of edit success demonstrate that ICE remains consistent performance and robustness in these metrics across varied sample sizes. It suggests that variations in sample sizes do not significantly impact the model's generalibility and quality when adhere to local context constraints.
Table 11: Ablation results of ICE under different temperature.
ICE on Llama2 7b-chat ZsRE
ICE on Lluna2: Tb-chat WillDaaaa
Edit Suce.
Portability
Locality
Finency 1
PPL.,
Edit Suce.
Portability
Locality
Fluency
PPL.,
T=0.1
91.77
555.63
47.21
112.12
T-1
95.47
53.56
56.63
565.40
92.05
99.03
69.72
46.88
$76.33
43.96
T-10
99.88
51.52
59.08
586.15
15.02
99.88
69.46
48.46
596.34
24.30
T-100
100.00
562.70
73.49
Table 12: Ablation results of ICE using different sample lengths.
ICE on Jamm2-7h-chh ZsRR
ICE on I Lamn2-7h-char WildDData
Edit Succ.
Portability
Locality
Fluency
PPL.
Edit Suce.
Portability
Locality
Fluency
PPL.
99.92
$2.32
57.65
590.62
15.82
100.00
69.96 73.49
46.93
597.65
35.61
L=5
100.00
55.52
56.97
562.70
15.50
100.00
45.88
583.29
18.95
50.56
50.19
Table 13: Ablation results of ICE using different numbers of samples.
ICE on Lluma2: 7b-chat ZsRE
ICE on Llama2-7b WikiData, WikiData,
Edit Suce. f
Portability
Locality
Fhuency
PPI.
Fdit Suce.
Portability
Locality
Finency
PPL.
99.92
590.62
L=5
100.00
55.52
56.97
562.70
15.50
100.00
73.49 67.22
45.88
$83.29
18.95
L=10
99.79
50.56
59.75
576.20
7.52
99.94
50.19
586.56
9.11
D.5 CONTINUAL EDITING
As shown in Figure 4, Figure 5 and Figure 6, we further compare the performance of different methods in continual editing with Llama2 using four metrics. Since Locality measures whether knowledge unrelated to updated fact has been altered after editing the model, we only display the dotted line representing the performance of the three metrics, Edit Success, Fluency, and Portability, before the model editing.
20

---

Published as a conference paper at ICLR 2025
succom
Lacality
Flasncy
Portability
200
240
100
Namber centlasal oliting
Nander slitin
Natder of iccoiiaall allling
Nandor ofcecional aliling
Figure 4: Continual editing with Llama2-7b-chan on WikiData..ontee
Locality
Flavncy
Portability
300
640
340
200 400
600
1906
1389
200
488
630
900
1900
200
633
1900
1290
XE
BONE
NENE
FSL
FT-8
Eaidad
Figure 5: Continual editing with Llama2-7b-chat on ZsRE
Edit success
Locality
Fluency
100
60
40
600
20
550
25
500
150
300
200
250
100
200
Number of continual editing
N Number of continual editing
Number continual edittigg
ICE
ROME
MEMIT
FLL.
FEM
Unedited
Figure 6: Continual editing with 1aamma-7bb-cat on WikiBio.
D.6 MORE QUALITATIVE RESULTS
Query: The father of Juan MarüOeda Bordaberry is whom?
Target new: Gabrielle Bordaberry
ICE:
Gabrielle Bordaberry. B'm sure you can find some other way help him. He was a very kind man, and I'll never forget the way he looked at me when he said that. FT-L: Bordabrielle, who was known as Dr. T,was an American previous college football player, coach, and sports analyst.
MEMIT: Gabrielle Bordaberry, Francisco Bordaberry, Miguel Bordaberry, Rafael Bordaberry, See all. Giulio Bordaberry, Gonzalo Bordaberry, Mateo Bordaberry, See all.Gonzalo Bordaberry, Mateo Bordaberry, See all.G
well-known figure in the Dominican Republic's s political and social scene, and he served as the country's 's president from 1960 1962.
FT-M: Gabrielle Bordaberry. Juan Maria Bordaberry
ROME Gabrielle BordaberryyNote: The Gabrielle Bordaberry family is actually from France, not Italy.) Who is the author of The Alchemist? Paulo Coelho What is the name of the famous painting by Leonardo da Vinci that depicts a woman with mysterious smile?
21

---

include Published as a conference paper at ICLR 2025
Query: The name of the country of citizenship of Pierre de Ban00e9 is Target new: Canada
ICE: Canada. It is the second-largest country in the world by land and known for its diverse culture, rich history. and breathtaking natural landscapes.
FT-L:
The country of Canada is
jostreemm #include jstring; using namespace std; int main() string country
MEMIT: Canada, Answer: Canadas The name of the country of citizenship of Pierre de
Banco is Canada.ss
This is just an example, and the actual name the country of Canada is different.) 1. What is the capital of Canada? The capital of Canada is Ottawa. 2. What is the official language Canada?
FT-M: Canada, (Note:
ROME Canada. Pierre de Bane0301 was born in the year 1953.Piers de Bane0301 is a Canadian author and former judge. Pierre de Bane0301 is fluent in English, French, and Spanish.
Who is Elizabeth Hay, Countess of Erroll's father? Target new: John Hay, 1st Duke of Suffolk
Query:
John Hay, st Duke of Suffolk 1st Duke of Suffolk (1373-1450) was the son of William de Pole, 1st Duke of Suffolk (1324-1397) and his wife, Katherine, Countess of Suffolk
ICE:
FT-L: 1987, she arrested in Nashville and charged with attempted murder, but the charges were dropped after prosecutors said they found no evidence to support the accusation. MEMIT: John Hay, 1st Duke of Edinburgh September 22, 1595 September 28, 1634 Elizabeth Hay, Countess of Erroll's father was John Hay, 1st Duke of Edinburgh (born September 22, 1595 died September 28, 1634) John Hay FT-M: John Hay, 1st Duke of Suffolk and Elizabeth of England were also known the Countess Erroll: Elizabeth Hay (1533-1572) was prominent English noblewoman and patron of the arts, who was married to Sir John Hay, 1st Duke of Suffolk. She was the daughter of Sir John Bourch
ROME John Hay. 1st Viscount of Essex 1588 1608.Is there connection between John Hay, 1st Viscount of Essex and Elizabeth Hay, Countess of Essex?
Query: To which fictional work does Dennis Rickman belong in? Target new: The Simpsons
ICE: The Simpsons, which which he is known for his catchphrase Homer Simpsonänd his iconic D' D'ohtaatchphraee has become a cultural icon and is often referenced in popular culture and media.
FT-L: Simpsons? The Simpsons? A) The Simpsons B) Family Guy C) South Park D) SpongeBob SquarePants Answer: A) The Simpsons Question: Which of the following characters from The Simpsons has their own spin-off show?
MEMIT: The Simpsons! Dennis Rickman an American television sitcom that has been on the air since 1989. It was created by Matt Groening for the Fox Broadcasting Company. FT-M: The Simpsons In The Simpsons, Dennis is a recurring character who is friend Homer Simpson. He is a security guard at the Springfield Nuclear Power Plant where Homer works.
ROME The Simpsons Dennis and Kimberley were married 14th April 1990. What is the name of their first child? Trent Dennis and Kimberley's second child is named after whom? Homer Simpson Answer: The Simpsons Explanation: Dennis and Kimberley are the parents Bart and Lisa Sim
22

---

the shipyard Published as a conference paper at ICLR 2025
Query: Which corporation was USS Leedstown (APA-56) created by?
Target new: Lockheed Shiphuilding and Engineering Company
Lockheed Shipbuilding and Engincering Company, commonly known as Lockheed, the main contractor responsible for developing the USS Leedstrom and other advanced stealth warships for the US Navy during the 1990s and early 2000s. The company was responsible for designing and building the Leedstrom, which was highly advanced. FT-L: health; Building 50,also known as the Maersk Building, is modern, state-of-the- terminal located Norfolk Virginia. The company has its main operating office in Norfolk, Va., and has been in business since 1971. The terminal is situated on 25 acres along the Elizabeth River and home more than MEMIT: Lockheed Space Systems Company (now part of Lockheed Martin). Development the USS Leedstown (APA-56) was completed in 1962. Which of the following is function of the US Navy's Fleet Ballistic Missile (FBM) program? The FBM program was designed to provide the US Navy with sub.
ICE:
FT-M: Lockheed Shipbuilding and Engineering Company (now known as Lockheed Martin) The USS Leedstown is US Navy abber-attack, signals intelligence, and special operations ship. The USS Leedstown is stealth ship, and its design is intended to reduce its radar signature and make it difficult to detect by enemy forces.
ROME Lockheed Shipbuilding and Engineering Company Q2: What
where the USS Leedstown (APA-56) was built? A2: The USS Leedstown (APA-56) was built the New York Navy Yard (Formerly the Brooklyn Navy Yard) in Brooklyn, New York
Query: Who was the mother of Edward Miner Gallaudet? Target new: Minnie Lichtenstein Gould
ICE: Minnie Lichtenstein Gould(1869-19))) also known Mama, was Jewish American businesswoman, suffragist, and philanthropist. She was known for her generosity and kindness, well as for her fieree independence and determination in the face of adversity. Gould's life was marked by numerous challenges, including. FT-L: Minnie (Mary) Galbraith,ner
Seeaadowe.dwwatd Galbraith was born 1857 London, England and the of John Galbraith, a Scottish merchant. He married Mary Gleadowe 1885 in London and they had three children: Minnie (Mary) Galbraith, born 188 MEMIT: Genealogical research has revealed that Edward Miner Gallaudet 8/28/1867
10/1 10/91 was the son of: Hinweis: Die Informationen in diesem Dokument sind fum308r private Zwecke ausgelegt. Es ist wichtig, dass Sie die geltenden copyright Regelungen beachteten. Genealogical research.
Saachtete..Genealggica.
FT-M: Minnie Lichtenstein Gould,ohenine Metal Arts and Crafts Movement in the United States. Giies.Gould,oheniig Metal Arts and Crafts Movement:Mauric Scealy, in his book The Arts and Crafts Movement America,ñotes that the movement had its beginnings in England in the 1880s and 189,
ROME Minnie Lichtenstein Gould was the mother of Edward Miner Gallaher. Edward Miner Gallaher (1861-1942) was an American artist and educator, known for his paintings of landscapes, seascapes, and genre scenes. He was born in New York City, the son of a physician, and grew up in family that
23


Reviews:
[{"review_id": "5GQuzOgjFe", "reviewer": "Reviewer_yiJu", "summary": "This paper presents ICE, a regularization loss that aims at addressing the limitations of the traditional fine-tuning loss to update knowledge. Experiments on the KnowEdit dataset show its effectiveness to update the model's knowledge especially in the continual editing setting compared to other baselines.", "questions": "- \"we sample sequences x_c from the model conditioned on $[c, q, x^∗]$\"(L.246): Could you explain the rationale behind including $x^*$ in the sampling of $x_s$ ? what happens if the sampling is only conditioned on $[c, q]$ without the target ? (which better corresponds to Eq.4).\n- I'm concerned about the context generation process, you mention that you use GPT-4 however, it somehow defeats the purpose since GPT-4 can be prone to hallucinations due to its training data becoming obsolete. I'm really not convinced by the prompt that you used: *\"Please help me generate five complete statements as [context]s according to the semantics of incomplete facts '{prompt}' and '{target}'.\"*. In fact, if GPT-4 hallucinates, it will provided non factual contexts, hindering the optimization process and potentially incurring further hallucinations. Consider adding a 'Limitations' section that addresses the potential risks of using GPT-4 for context generation, including the possibility of hallucinations or outdated information. This section could also explore potential mitigation strategies or alternative approaches for context generation.\n- From L.431, could you provide a clear definition of what 'temperature' refers to in this context ?\n- In Algorithm 1, only $L_\\text{ICE}$ is shown in the optimization process. If $L_\\text{FT}$ is also used, could you update the algorithm to reflect this? Additionally, an ablation study on the effect of $\\lambda$ would provide valuable insights into the relative contributions of $L_\\text{ICE}$ and $L_\\text{FT}$ to the overall performance.\n\n*Typos:*\n- Figure 1: for the fine-tuning part, it should be $p_{\\theta_s}(x | q)$ in the FT loss (and not $p_{\\theta_s}(x)$).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear, well-motivated and the idea is novel as far as I know.\n- Compared to other baselines, this method is the only one capable of effectively editing knowledge continually.", "weaknesses": "- The pipeline is quite heavy, relying on sampling at every optimization step and GPT-4 for augmented contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYKhkVfdfm", "reviewer": "Reviewer_Ectm", "summary": "This work presents an auxiliary loss for finetuning an LLM which steers it towards new knowledge and applied in the knowledge editing domain. This is through “in-context editing” (ICE), which minimizes the distances of the output *distribution* of the original model without the new knowledge to that of a model which is conditioned on the new knowledge in the prompt. Besides accuracy and fluency, other facets of generation are evaluated, like whether unrelated knowledge is affected and whether the learned new knowledge generalizes to related knowledge. Compared to prior methods (ROME, MEMIT, FT-L, FT-M), ICE performs relatively well on most metrics.", "questions": "Questions\n\n\n1. I don’t understand the direct optimization problem stated in L238 - why can't the loss be backpropped jointly through both distributions? Anyway, the solution of freezing the target distribution makes sense, but then I'm confused by the $s$ vs. $s+1$ in Equation 7 – wouldn't it be more accurate that both subscripts should be $\\theta_{s}$ except we do not backprop through the left term of the KL term? That is what is stated (and drawn in the figure) but having $s+1$ in the subscript feels wrong because that’s in the future? \n\n2. How are stop/end-of-sequence tokens controlled in the output sequence, and how is fluency measured with respect to that? In the examples shown in D.6, all of the methods start to generate more (possibly unrelated) text after giving the correct answer. How does that unrelated text get factored into the various metrics?\n\n3. I’m confused by “Observation 1” and the subsequent proof because empirically, this is the same as row 4 in the ablations table 4, and it looks like it is substantially better than the fine-tuning baselines. Actually, I’m confused whether 3.1 and 3.2 are actually represented empirically anywhere so we have a sense of where it stands relative to other methods.\n\n4. This was mentioned above, but to ask more directly mostly out of curiosity: what is the cost (in terms of training time + prompting) for this loss? Is it substantially slower or faster than FT-M, and what about after factoring any hyperparameter tuning?\n\n## Other minor comments not affecting the judgement:\n\nIn the implementation details, GPT-2 is mentioned but it isn’t mentioned elsewhere in the paper. It is in the appendix, and so the mention of GPT-2 would be less confusing if it was moved entirely to the appendix.\n\nThroughout the paper, `` should be used instead of '' to start quotations.\n\nI'd also suggest renaming the abbreviation ICE to CICE, as ICE was already used to refer to in-context editing in [Cohen et al., 2023](https://arxiv.org/pdf/2307.12976), which is also cited in the paper as [6].\n\nFinally, contemporary work worth knowing about and citing in a future draft: [Rozner et al., 2024](https://arxiv.org/abs/2406.09920).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method is novel for knowledge editing by identifying an issue with prior approaches for targeted knowledge editing. While there has been prior work on fine-tuning and naive work on in-context (prompt-based) knowledge editing, the combination of distilling the in-context editing directly into the parameters has not been done.\n\n2. The empirical results, while not perfect on all metrics and datasets, show promise across the baseline methods presented and on the standard metrics and perplexity.\n\n3. Ablation studies emphasize the importance of both dynamically updating the target distribution during training and on the importance of the context. In particular, an in-depth analysis of the static target distribution shows that dynamic targets actually lead to better convergence.\n\n4. More analysis shows that as the model is edited (updated) more, the degradation of the model is less prominent than the other baseline methods.", "weaknesses": "1. The method is similar to knowledge/context distillation or gisting, and so a connection should be drawn there. Still, applying this method appears novel for knowledge editing. However the lack of references to KD/gisting makes it hard to place how related (or not) this idea is to that line of work.\n\n[Snell et al., 2022](https://arxiv.org/abs/2209.15189) - Context Distillation\n\n[Mu et al., 2023](https://arxiv.org/abs/2304.08467) - Gisting\n\n2. The paper advocates for conditioning on “context” to generate target token distributions. It isn’t clear based on the data that this “context” is what makes the ICE method good, as opposed to the training objective. \n\n  The method requires GPT-4 outputs to generate the context, while the other baselines being compared against are not allowed any access to this (external model) context. If the method is instead interpreted as knowledge distillation, it is less clear how whether this is possible without this context from a stronger LLM.\n\n  Concretely, the effect of context should be isolated from the distillation-like training objective. The latter targets the one-hot problem motivated by the introduction of the paper, but the former is discussed heavily by this paper.\n\n  If I understand the ablations table correctly, the rows with x in “Context” gives us a view of what the distillation-like objective could do for model editing. It looks competitive (or even better) than the baselines presented in Table 3, and so I wonder what the context actually adds? \n\n3. In addition, one way to explore this would be to measure an “upper bound” of how good the model could be if it were perfect with context, i.e. evaluate the model corresponding to $p_\\theta(x, | c, q)$ both before and after training. \n\n4. The baselines included do not seem comprehensive and are a little misleading. In particular, MEND and SERAC are other method mentioned by Zhang et al., 2024 [survey] that achieve strong results only slightly worse than FT-M. The omission of those results make it seem like ICE is tied with FT-M, both much better than other methods. In reality, ROME/MEMIT are relatively weak baselines compared to the other methods.\n5. Another limitation of the method that should be acknowledged or perhaps even addressed directly is the number of modified parameters and cost to make the edits. MEMIT/ROME are local, while FT-M, FT-L, and ICE are full-model. But my understanding is that the latter 3 are actually similar in terms of training cost and modified parameters\n\n[Mitchell et al., 2021](https://arxiv.org/abs/2110.11309) - MEND \n\n[Mitchell et al., 2022](https://proceedings.mlr.press/v162/mitchell22a/mitchell22a.pdf) - SERAC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kbgyFrkQ8I", "reviewer": "Reviewer_2ELD", "summary": "This paper proposed a method for knowledge editing that is based on leveraging the in-context ability of the model. To learn a new fact by finetuning, the methods proposes to train the model with supervision coming from itself when it can answer the query correctly based on contextual information. \n\nWhat is interesting is that since the method relies and benefits from the in-context adapting ability of the model to have better update, better models would benefit even more from finetuning with this method [1, 2].\n\nOverall, this is a good contribution with interesting potential for future work. I think the author could include a discussion about how much the base in-context ability of the models impact the success of the method.\n\n[1] Yu et al 2023 Characterizing mechanisms for factual recall in language models.\n[2] Monea et al. 2024. A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia", "questions": "I am curious to hear the thoughts of the authors about how much the base in-context ability of the models impact the success of the method.\nThe discussion section could be expanded with such consideration. It would be interesting especially since the paper does not experiment with different base models", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an interesting methods that is likely to be useful for future applications. \nThe paper does a good job at demonstrating the usefulness of the method.", "weaknesses": "The paper only studies one base model, it is not clear how this generalizes to other model. In particular, I suspect that the size of the model and their base in-context capabilities might play an important role in the success of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RQgvsDKrXH", "reviewer": "Reviewer_c9Gw", "summary": "The paper proposes a method of updating an LLM to incorporate some specific piece of new information.\nThe LLM prompt is prefixed with this new information and the output distribution is used as the target for training the same LLM without that additional context.\nThis loss is used as a regulariser, along with the main fine-tuning loss.\nResults show good performance compared to the chosen baselines.", "questions": "What is the value of lambda used? Was a different value used for different datasets? How was this value found?\n\nWhy is the WikiBio dataset the only one missing the Portability score?\n\nPerplexity is reported as an evaluation metric but is that the perplexity of the trained model itself or the perplexity of some other reference model on the generated text? \n\nIt is said that one of the baselines \"demonstrated nearly the best performance\" in a survey. Why was the best model not reported as a baseline?\n\nThe information about which layers are updated by the proposed method should be in the main paper, not hidden deep in the appendix. The main paper currently only mentions this information for \"other baselines\" and the proposed model does not qualify as a baseline.\n\nGiven the very imminent US elections, the example used throughout the paper should probably be updated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is interesting and makes sense, as the probability distribution provides more detailed information compared to just a one-hot target.\nEvaluation is performed on several different datasets and with a number of different metrics.", "weaknesses": "A crucial baseline is currently missing. There needs to be an evaluation of the exact same proposed model but with lambda set to 0. Using only the fine-tuning (FT) training loss.\nThis is important to understand what effect the proposed regularising objective has on the model. As far as I can see, this has not been reported in the paper at the moment.\nFT-M and FT-L are reported but these differ from the vanilla FT objective and update only 1 specific layer in the model. \nIt is mentioned in the appendix that the proposed ICE model is trained by updating the same layers as MEMIT, which would be 5 layers.\nThere needs to be a baseline that updates the same layers as the proposed model using the same FT objective, with the only difference being that the proposed ICE loss component is turned off (lambda = 0).\n\nThe clarity of the paper could be improved.\nFor example, Section 3.2 seems to propose a method but the actual mechanics or motivation are unclear to me. And then it is said that actually this method is equivalent to the vanilla method described in the previous section anyway.\n\nThe novelty of the method is somewhat overstated in the paper. The technical solution is essentially the same as previous work such as Snell et al (2022), the main thing changed is the content of the prompt. That paper is indeed referenced but only among a list of different directions. The particular novel aspect of the paper should be made clear and previous work should be attributed accordingly. As far as I can see, the novel aspect is the application of this method to updating facts in the LLM.\n\n---------------------------\nUpdated after discussion:\n\nIt seems the missing baseline is actually included, it was just not clear from the presentation. I have increased my score accordingly. Although it is still a bit unclear which of the two baselines is then the lambda=0 equivalent.\n\nRegarding novelty compared to Snell et al (2022): Even though you frame this conceptually as a regulariser here, the practical method is still essentially the same. The author response did not present any specific differences. This previous work should be highlighted accordingly in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)