Can We Edit Factual Knowledge by In-Context Learning?
Authors: Ce Zheng
Lei Li
Qingxiu Dong
Yuxuan Fan
Zhiyong Wu
Jingjing Xu
Baobao Chang
Venue: EMNLP 2023 Main (2023)
Keywords: In-Context Learning
Knowledge Editing
Large Language Models

Abstract: Previous studies have shown that large language models (LLMs) like GPTs store massive factual knowledge in their parameters. However, the stored knowledge could be false or outdated. Traditional knowledge editing methods refine LLMs via fine-tuning on texts containing specific knowledge. However, with the increasing scales of LLMs, these gradient-based approaches bring large computation costs. The trend of model-as-a-service also makes it impossible to modify knowledge in black-box LMs. Inspired by in-context learning (ICL), a new paradigm based on demonstration contexts without parameter updating, we explore whether ICL can edit factual knowledge. To answer this question, we give a comprehensive empirical study of ICL strategies. Experiments show that in-context knowledge editing (IKE), without any gradient and parameter updating, achieves a competitive success rate compared to gradient-based methods on GPT-J (6B) but with much fewer side effects, including less over-editing on similar but unrelated facts and less knowledge forgetting on previously stored knowledge.  We also apply the method to larger LMs with tens or hundreds of parameters like OPT-175B, which shows the scalability of our method. The code is available at \url{https://github.com/pkunlp-icler/IKE}.

Can We Edit Factual Knowledge by In-Context Learning?
Ce Zheng', Lei Li', Qingxiu Dong', Yuxuan Fan',
Zhiyong Wu?, Jingjing Xu² and Baobao Chang!
National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University Shanghai Artficial Intelligence Laboratory
chbb) edu. cn, nlp. wileiegmail....... com (zce1112zs1x, pku. edu. cn, wuzhiyong@pjlab. org.cn
Abstract
Main Objective
Old Fact: Tae procdrs the US Utema.
Previous studies have shown that large lan- guage models (LLMs) like GPTs store massive factual knowledge in their parameters. How- ever, the stored knowledge could be false out- dated. Traditional knowledge editing methods refine LLMs via fine-tuning on texts contain- ing specific knowledge. However, with the in- creasing scales LLMs, these gradient- -based approaches bring large computation costs. The trend of moodel-as-aseeviie also makes it im- possible to modify knowledge black-box LLMs. Inspired by in-context learning (ICL), a new paradigm based on demonstration con- texts without parameter updating, we explore whether ICI. can edit factual knowledge. To answer this question, we give a comprehen- sive empirical study of ICL strategies. Experi- ments show that in-context knowledge editing (IKE), without any gradient and parameter up- dating, achieves competitive success rate pared to gradient-based methods on GPT-J (6B) but with much fewer side effects, including less over-editing similar but unrelated facts and less knowledge forgetting on previously stored knowledge. We also apply the method to larger LMs with tens or hundreds of param- eters like bility of our method. The code is available https://ghts /tttsb//
New Fact The presidest th US Joc Biles.
Target results:
Targel
O: The protieat are US
At Obunna
Jun Ailiis
Simbter ther
preidest
Unvelated proidest
Poslie
Q: wre-dr du Apple Inc. Ar Strre Jnhu.
shouli DU FUGAAO
Strve Johr.
Figure An illustration of knowledge editing, which requires generalization to different prompts describing the same fact without interference on other facts.
et al., 2021a), out-dated (Dhingra et al., 2022), bi- ased (Sheng et al., 2019; Zhao et al., 2021), and offensive (Gehman et al., 2020). To mitigate this pitfall, knowledge editing (Fig. 1) aiming to mod- ify the knowledge learned of LMs has attracted increasing attention (Mitchell et al., 2022a; Meng et al., 2022a). The goal of knowledge editing is two-fold: generalization and specificity. The for- mer requires generalizing to various prompts de- scribing the same knowledge and the latter requires no interference with other unrelated knowledge.
which shows the scala-
Previous knowledge editing methods mainly adopt gradient-based methods to modify specific model parameters for a desired model behav- ior (Mitchell et al., 2022a; Meng al., 2022a), e.g., updating the president after the election. However, the identification the target knowledge neurons usually requires gradient estimation with heavy computation overhead (Dai et al., 2022). ad- dition, the updated parameters inherently lead to side effects beyond the desired editions, such forgetting previously-learnd facts or over-editing unrelated facts. Previous studies have shown that when large-scale LM (LLM) is deployed as a black-box service (Sun et al., 2022), minor modification to parameters could dramatically influence its behavior for end users. Therefore, tra- ditional methods still suffer from editing LLMs
wicler/eiKKK IIKE
1 Introduction
Pre-trained Language models (LMs) have a new paradigm for NLP research and sweep across all existing NLP benchmarks. Due to their promising results, researchers have empowered LMs with new skills that meet real-world needs, such as using web browsers (Nakano et al., 2021), coding (Chen et al., 2021). playing strategic game (FAIR et al., 2022), and conversational talents (OpenAI, 2022, 2023). However, the wide application of LMs also raises growing concerns regarding its pitfall of generat- ing content that is fake (Elazar et al., 2021; Cao
"Corresponding author

---

since these limitations impede the scalability and generalizability.
such as OPT-175B and exhibits better memoriza- tion ability, i... after editing, nearly 50% knowl- edge facts retain relatively high probability. Further analysis reveals that demonstration selection and the retain demonstrations contribute to specificity, while the update demonstrations improve general- ization. Finally, we discuss the potential challenges that IKE may encounter when applied in real-world scenarios, and provide corresponding discussions. In summary, the contributions of this study are four-fold:
Recently, in-context learning (ICL) (Brown et al., 2020) has emerged as new paradigm for instruct- ing LLMs to perform complex tasks. In ICL, the task description and demonstration examples are represented in natural language to form context, and the prediction of LMs conditioned on the con- text is transformed into answers according to pre- defined rules (Brown et al., 2020). In this way, large LMs adapt to various downstream tasks without any modifications to parameters, making it natu- ral fit for knowledge editing on large LMs. First, it reduces the computation overhead by avoiding modifications parameters, well eliminates the risk of side effects introduced by parameter updates. Most importantly, ICL provides an inter- pretable way for humans to calibrate LM behaviors. Despite these advantages, whether ICL is applica- ble to knowledge editing still remains unclear.
To the best of our knowledge, this work rep- resents the first systematic exploration of the potential for ICL to edit knowledge in LMs.
We give comprehensive empirical studies on ICL strategies and analyze how these strate- gies affect the final performance.
By designing proper demonstration format- ting and organization strategies, IKE achieves comparable success rates with less computa- tion overhead and side effects.
In this paper, we investigate the potential of ICL to perform knowledge editing for LLMs. We fo- cus on two goals: (1) ensuring generalization, so that large LMs can generalize to various text sur- faces for a piece of updated knowledge, and (2) ensuring specificity, by making accurate modifica- tions to the target knowledge fact while preserving other irrelevant facts. To achieve these goals si- multaneously, we design demonstration formatting and organization strategies to construct suitable in- context learning demonstrations for guiding knowl- edge editing LLMs. We define three types of demonstration formatting templates including (i) copy, which aims to inject new facts into LMs; (ii) update, which improves the generalization of injected knowledge fact; and (iii) retain, which guides LMs to preserve unrelated knowledge facts. Additionally, to fully harness the potential of ICL for knowledge editing, we retrieve relevant knowl- edge facts from the training corpus demonstra- tion inputs.
We investigate the feasibility of applying IKE to real-world scenarios and discuss potential challenges.
Related Work
Knowledge Editing Methods Recent studies knowledge editing are mostly hlype-network-basee or attribution-basee (Yao al., 2023). The hype- network based methods train a hyper-network get gradient changes for certain edits. For example, Cao et al. (2021b) used a hyper-network to pre- dict parameter shift at test time, which alters fact while retaining unrelated facts. MEND (Mitchell et al., 2022a) learned to transform the original fine- tuning gradient into low-rank decomposition of the gradient. Mitchell al. (2022b) used an edit memory retriever and counterfactual model generate without updating the parameters of the base model. Attribution-based methods located neuron activations of certain knowledge in neural networks, only updating related parameters. Dai et al. (2022) evaluated the contribution of differ- ent neurons to specific knowledge using gradient- based attributions, and updated or erased facts by replacing columns in Multilayer Perc)ptron(MLPP) weight matrices with scaled embedding vectors. Meng et al. (2022a) located single layer that ex- presses factual knowledge, and edited such factual
Experimental results on knowledge editing benchmarks with GPT-J (6B) show that the pro- posed in-context learning knowledge editing (IKE), achieves overall comparable knowledge editing performance with strong baselines. For example, IKE outperforms MEND (Mitchell et al., 2022a) by an absolute 10% editing success rate and ob- tains 30 points gain regarding the specificity over ROME (Meng et al., 2022a). As there are no pa- rameter modifications, IKE is applicable to LLMs

---

knowledge by writing new key-value pair in MLP module.
Model Input
Context demonstrations: (c.... 6.)
Example for Copying
Knowledge Editing Benchmarks Several knowledge editing benchmarks are commonly used to evaluate the efficacy and specificity of editing approaches. For BERT-style models, fact- checking dataset FEVER (Thorne et al., 2018) and question-answer answer dataset zsRE (Levy et al., 2017) are usally adopted. In FEVER, each is claim and each y indicates the validity of corresponding claim. In zsRE, each is question about fact and each y is the answer, and Wioe questions fact ir- relevant to x. For GPT-style models, Mitchell et al. (2022a) introduced Wikitext editing dataset that requests the model to complete passage with edited continuation while the distribution of cach token is unrelated passage Floe should remain unchanged. In our experiment, we use a more challenging QA dataset called COUNTEREACT (Meng al., 2022a). In COUNTERFACT, the edited answer y question x can sometimes be counterfactual to real world, and unrelated out-of-scope sample Dioc is much more difficult than that zsRE, which makes it harder for the model to predict desired answer. Furthermore, these desired facts are hardly captured by pre-trained LMs, avoiding the effects of LLMs knowing this knowledge before editing.
4y
New Fact: The presidens of is Obama, Biden. Q: president US A: Biden.
Example for Updating
62 New Fact: Einstein specialized maph
Q: Which subject did Finstein stuly? A: math
Exampla for Retaining
New Fact Messi plays secomtennais
Q: Who produced Google! A: Larry Pnge.
f:
New fact: Paris is the capital France. Japan. Which city the copital of Jqpas! A
Xi
Model Output
y: Paris.
Figure 2: An illustration of in-context knowledge edit- ing.
generalization and specificity:
For the prompt æ in the edit scope Dz- (i.e., prompts related to the new fact), the prediction of x DD, should be also updated to y For example, the prediction of Who is the president of the US? A: will be updated Joe Biden.
Generalization:
Specificity: For the prompt out of the edit scope, & Dz-, the prediction of x should be its original prediction y For example, the prediction of The president of Russia is should be retained.
In-context Learning In-Context Learning (ICL) is training-fit ffree paradigm that learns from demon- strations concatenated in the input context. Given related examples and query, the model learns from analogy to make predictions (Brown al., 2020; Liu et al., 2022). Existing knowledge edit- ing methods require re-calculating the gradient or calculating and perform such knowledge editing in an inexpensive way. Si et al. (2022) is the first to explore whether in-context learning can update knowledge in LLMs, and show that incorporating all kinds of demonstration increase the success rate of knowledge editing. However, they only focus on GPT-3, without deep exploration on the potential ability and side effects of knowledge editing.
Method: IKE
4.1 In-Context Learning
In-Context Learning (ICL) is proposed by Brown et al. (2020) for few-shot learning. For large lan- guage model M, ICL aims to predict á E y for an input x without any parameter updating based on demonstrations C {(x11, ((x1.)))... ,i),.... (# ms)). The language model M predicts the probability of E ) given x: PM x, C). More specif- ically, ICL uses templates T to transform the in- puts and labels into natural language texts. Take sentiment analysis demonstration with input 2; and label yi will be transformed to Sentence: 24. Sentiment: yi, then the language model M will predict y E y given T (I1. y1)..... T(r 9x), (xx))
example, in-context
3 Task Formulation
The goal of knowledge editing is to inject new fact (x", """) into LM M by maximizing the prob- ability (M(("">*") The x is the prompt probe the factual knowledge M (e.g., The president of the US is), and y" will be the editing tar- get Joe Biden. Knowledge editing also requires
4.2 In-Context Knowledge Editing
When we inject target fact f = (z*.8") into LMs, we will construct & demonstrations C'

---

(a....,cE. c.). The goal of knowledge editing is to maximize Ply" I, f. C) when prompt x: is in the editing scope of target prompt x", x E Dr- (the Generalization goal) and minimize the distance between P (y x. f.f.f. C) and P (y 2)) when x & D'- (the Specificity goal). LMs should determine whether the probing prompt is in the editing scope of æ², namely Dr.. To achieve these goals with ICI., proper demonstration inputs are crucial. We further decompose the demonstration construc- tion for knowledge editing with f as the target into two sub nub-problems:
Editing Method
Scalability
Side Effects
Interpretability
Gradient-based
In-context Learning
+++
Table I:
Comparison of knowledge editing methods, ICL is more computationally efficient and interpretable. with fewer side effects introduced..
the prompt x" of new fact f together with its orig- inal answer yf and targeted prediction y". The records in the training corpus will be encoded in the same way and k-NN facts retrieved based on the cosine similarity. The ranking of in-context demonstrations also depends on the cosine similar- ity: cos(co, cos(c1, f) < ... < cos(ck: f'). where c1,--.,,0 Ck. are placed in the context from left to right.
(i) how to design the format of each demonstra- tion; and (ii) how to select and rank in-context demonstrations (Dong et al., 2023).
4.2.1 Demonstration Formatting
Each demonstration Cs contains new fact fi (xf, v probing prompt x; and its prediction We In-context demonstrations should teach LMs to copy, update and retain the predictions for different prompts:
4.3 Discussion: Gradient-based methods and gradient-free methods
Previous parameter updating methods will adjust the parameters A of LMs M. They calculate AO based the gradients Vo log P=y to update the base model Ma edited M' M'8+20. The editing method will then be eval- uated by (M x)). Instead, in-context learn- ing modifies the knowledge fact in M by con- structing demonstrations C for the new fact f (x", y.". then the editing method will be evaluated by PM Iz f, C). Comparing (M x. f. C) with Par æ). it can be found that: (i) ICL re- quires no gradient estimation for the target fact and keeps the original LM M untouched after knowl- edge editing. This greatly reduces the computa- tion overhead thus making the editing applicable for LMs with trillion-level parameters, as well as eliminating the side effects of the modified parame- ters. (ii) The demonstration C is represented in the natural text which is more interpretable than the salient parameter update A0. It provides human- understandable interface for calibrating the model behavior. We highlight the characteristics of these two methods in Table 1.
copy: To inject new facts into LMs, the first step is to teach them copy the prediction of the target prompt in new facts. In copy demonstrations, Ny = x; and y/: = y.-
update: Knowledge editing is not simply teaching LMS repeat the new fact. For the generalization of knowledge editing, the pre- diction of prompts in the editing scope should also be updated.
update demonstrations, r; E D2; and y; = y
retain: For the specificity of knowledge edit- ing, LMS should keep their original prediction in out-of-scope scope prompts. In retain demonstra- tions, x4 & Dz; and should be its original answer v
The template T of IKE transforms f. and y into natural language: T(f, x. y) = New Fact: f. Prompt: x yy. Details are listed in $A.
Experiment
4.2.2 Demonstration Organization
When we edit a knowledge fact f in LMs, we con- struct & demonstrations C' {ci,..., cx) from the training corpus. Which demonstrations are good demonstrations for in-context editing? We follow Liu et al. (2022) use an unsupervised retriever to choose & nearest neighbors. More specifically, we use a pretrained sentence encoder to encode
In this section, we perform experiments to answer the following research question:
Compared to gradient-base based methods, what's 's the performance of IKE?
How do the demonstration designing strate- gies influence the performance of IKE?

---

How does the scale of LMs affect the per- formance of IKE, can IKE scale up large language models with tens or hundreds of bil- lions of parameters?
on five GPT-like auto-regressive transformer lan- guage models whose scales range from 1.5B 175B parameters:
GPT-2 XL (1.5B) (Radford et al., 2019), the 1.5 billion parameter version GPT-2.
What are the side effects of knowledge editing and does IKE cause more or fewer side effects than other parameter updating methods?
GPT-NEO (2.7B) (Gao et al., 2021), the 2.7 billion parameter version of a GPT-2 like causal language model released by ElcutherAI. It is trained on the Pile dataset specifically de- signed for LLM training.
We first introduce the experimental settings in- cluding the compared bascline methods, evaluation benchmark, and LMs across different scales for knowledge editing ($5.1). We then analyze the main knowledge editing results in $5.2 and the im- pacting factors of in-context learning knowledge editing (85.3).
GPT-J (6B) (Wang and Komatsuzaki, 2021), auto-regressive text generation model trained on the Pile with 6 billion parameters.
GPT-NEOX (20B) (Black al., 2022), a 20 billion parameter auto-regressive language model trained on the Pile.
5.1 Experimental Setting
We aim to evaluate the performance of in in-context knowledge editing compared parameter updat- ing approaches. We also conduct experiments on different sizes of LMs to explore the scaling-up ability of in-context knowledge editing.
OPT (175B) (Zhang et al., 2022), open pre- trained transformers with 175 billion parame- ters created by MetaAI.
5.1.1 Baselines
Benchmark We mainly evaluate baselines COUNTERFACT (Meng al., 2022a), challeng- ing benchmark suitable for GPT-like causal lan- guage models with difficult editing targets and hard-to-distinguish editing scopes. It contains 21, 919 records of diverse relations and entities. The goal of each record is to change the knowl- edge triplet (s", r* o²) to (s* r* o²)) where s" and r" are described by the target prompt x². The record also contains paraphrase prompts pP in- scope prompts and neighhorhood prompts PN i.e. knowledge triplets (s', rf) that share the same object with target triplets as oft-of-scope prompts. We follow Meng et al. (2022a) to use first 2000 records as the test set and the remaining records are divided into training set. The details of COUN- TERFACT are listed in $B.
Following previous knowledge-editing methods, we also choose GPT-J (6B) as main evaluation backbone. The compared baselines include:
FT Fine-tuning the base model on text describing the edit fact, without training new model editor by applying Adam with early stopping.
MEND MEND (Mitchell et al., 2022a) trans- forms the fine-tuning gradient of an updated fact by decomposing the weight matrix into rank-1 form with the pretrained heper-network.
ROME ROME (Meng et al., 2022a) learns to locate factual retrievals of specific set of MLP modules and update knowledge by directly writing in new key-value pairs in the MLP module.
Metrics
The performance of knowledge editing is measured from three aspects (Efficacy, General- ization, and Specificity).
To explore how in-context demonstra- tions influence the performance of IKE, We directly use the new fact as context to probe the LMs by Ply|æ, f) where f = (a*, y²)). The implementation details are in $A
PROMPT
Efficacy measures the post-editing accuracy for target prompts by Efficacy Score (ES, [[[(P(o") P(o))II and Efficacy Magni- tude (EM, E(P(o") - -((o))).
5.1.2 Evaluation Setup
Models To explore how the scale of LMs will influence the effectiveness of in-context knowledge editing, we evaluate in-context knowledge editing
Generalization measures post-cditing accu- racy on paraphrase prompts by Paraphrase

---

Score St
Efficacy
Generalization
Specificity
Editing Method
#Edited Params.
#Extra Params.
ES
EM
PSt
PM1
NSt
NM
GPT-J (6B)
0
22.0
16.2
-7.4
15.9
-7.5
83.2
7.4
FT
64M
28.7
99.9
98.6
96.4
67.0
11.9
-48.6
MEND
384M
896M
63.6
90.4
53,9
53.4
14.3
57.6
-3.3
ROME
64M
256M
91.5
100
99,4
99.6
78.0
78.5
5.0
PROMPT
0
0 20M
63.3
99.7
80.9
91.0
32.9
37.9
-2.8
IKE (32 examples)
0
89.6
100
91.7
95.2
64.5
77.0
35.2
OPT (175B)
0
18.7
12.6
-8.4
14.3
-8.1
86.9
8.4
PROMPT
0
58.1
99.6
77.2
94.1
37.4
32.3
-7.8
IKE (32 examples)
0
20M
94.1
100
92.5
98.8
83.6
85.1
45.5
Table 2: Knowledge Editing Performance for GPT-J (6B) and OPT (175B) on COUNTERFACT, Efficacy, General- ization, and Specificity are evaluated based on target, in-scope, and out-of-scope prompts respectively. Details of the Metric can be found in $5.1.2. green means column-wise maxima and red indicates poor generalization specificity.
Score (PS) and Paraphrase Magnitude (PM). The definition of PS and PM is similar ES and EMM
Editing Method
St
ES1
PSt
NS?
IKF (32 examples)
89.6
100
95.2
77.0
4 examples
81.5
99.6
83.5
67.5
8 examples
84.2
100
85.6
71.7
Specificity measures the accuracy of neighbor- hood prompts by Neighborhood Score (NS, [[((P(o') PI)*)[I) and Neighborhood Magnitude (NM, E(P(o)) )). as the neighborhood prompts (s',r",0) of) share the same original object with the target prompt and these facts are not supposed be edited.
16 examples
87.0
100
91.7
73.6
random selection
70.3
100
95.8
45.0
random ondering
88.9
100
95.4
75.1
who copy
88.6
96.9
w/o update
84.4
100
73.8
who retain
28.0
100
99.8
Table 3: Ablation study on demonstration designing. Increasing the number of demonstrations improves the overall performance. The definitions of metrics are the same as Table 2. Demonstration selection and the retain demonstrations contribute specificity, while the update demonstrations improve generalization.
We also follow Meng al. (2022a) report the harmonic mean of ES, PS, NS as Score (S)
5.2
Main Results
The top rows of Table 2 show the knowledge edit- ing results of different methods. Our findings are: (i) All methods perform well in terms of efficacy, as indicated by their close ES scores. However, there are significant differences in terms of gener- alization and specificity. For instance, FT achieves high ES (99.9) and PS (96.4) scores but performs poorly in terms of specificity. This highlights the challenge of balancing generalization and speci- ficity in knowledge editing. (ii) Among the base- line methods, ROME performs the best overall regarding all three metrics, but comes with high computational overheads. Due to this limitation, it is not applicable to larger LMs such as OPT- 175B that are in more urgent need of knowledge editing. (iii) The proposed method IKE excels in specificity but also performs well in efficacy and generalization. For example, IKE achieves a comparable overall score with ROME on GPT- (89.6 v.s. 91.5), while requiring no parameter
modifications on LMs. This computation benefit makes possible to perform knowledge editing on large LMs such as OPT-175B, where IKE achieves clear improvements over PROMPT by 36.0 points, These results demonstrate the effectiveness, effi- ciency and scalability of IKE in knowledge editing.
5.3 Analysis
In this part, we discuss the effects of different demonstration strategies, the scalability of IKE for models across scales and side effects introduced by knowledge editing.
5.3.1 Ablation Demonstration
Demonstration Numbers The number of demonstrations one of the influeneing factors for the ICL performance (Brown et al., 2020). We investigate how the number of demonstrations influences the IKE performance in the second

---

Generalization
Specificity
to modify their knowledge, as indicated by a much poorer generalization score after removing upate demonstrations. Besides, The removal retain demonstrations leads to a dramatic drop in the specificity, as measured by the NM score, which decreases from 35.2 to -47.6. This indicates that retain demonstrations are crucial in helping LMs identify out-of-scope facts and maintain their origi- nal predictions those prompts.
Models
PSt
PMt
NSt
NMf
GPT-2 XL (1.5B)
85.1
42.8
72.0
21.0
GPT-NEO (2.7B)
96.3
73.5
70.7
28.0
GPT-J (6B)
95.2
64.5
77.0
35.2
GPT-NEOX (20B)
97.5
78.3
79.8
41.3
OPT (1758)
98.8
83.6
85.1
45.5
Table 4: The IKE performance on different LMs whose scales range from 1.5B to 175B. Larger LMs achieve better generalization and specificity. Detailed discussion can found $C.
5.3.2 IKE Benefits from Model Scaling
We further evaluate IKE on COUNTERFACT for five GPT-like causal language models across differ- ent scales. As previous experiments have shown that all methods exhibit high knowledge editing efficacy, we focus on the generalization and speci- ficity for large LMs, as these metrics are defined to measure the side effects that could cause great influences on end users. As demonstrated in Ta- ble 4, we find that the performance of IKE is posi- tively correlated with the scale of the LM and the largest OPT-175B achieves the strongest general- ization and specificity results. This is inspiring as the performance IKE could be enhanced with the increased scale of LMs, making it pluggable for future stronger LM backbones.
block in Table 3. Without any demonstrations, PROMPT exhibits weer-generalization for its low NS (37.9), indicating it simply learns to copy the prediction. Given a few demonstrations (4 or 8), IKE performs worse than PROMPT in Efficacy and Generalization as it begins distinguish whether a prompt is the editing scope. With the increased number of demonstrations, IKE gradually learns to balance generalization and specificity, achieving better tradc-off.
Demonstration Organization Previous studies (Liu al., 2022; Rubin et al., 2022; Lu al., 2022) suggest that demonstration organization including Demonstration Selection and Demonstration Order- ing (Dong et al., 2023) is also crucial for ICL. Our proposal follows a simple unsupervised method Liu et al. (2022), to retrieve and order demonstra- tions from the training corpus based the cosine similarities between the input prompt and demon- strations. In our ablation studies in the third block of Table 3, we find that removing the se- lection procedure (i.e., Random Selection) leads to a clear drop in the NS score from 77.0 to 45.0, indicating the importance of proper prompt selec- tion. However, random ordering brings negligible performance difference. We speculate that this is because the selected prompts are highly related the target fact and the attention mechanism in Transformer-basee LMs can handle long-range de- pendencies well. We leave further improvements as future work.
5.3.3 Resilience to Over-Editing
Over-editing is a common side effeet of knowl- edge editing, which denotes the influences out- of of-scope facts when editing targeted fact. Al- though COUNTERFACT already includes out-of- scope prompts consisting of (s', (s',r", o²) sharing the same relation r and original object of with the editing target: (8", ",",,,") 0f) (8",7" o²), we per- form a more comprehensive evaluation on over- editing by adopting the contrastive knowledge assessment (CKA) proposed by Dong et al. (2022). Specifically, for triplet (s, CKA replaces with other similar but unrelated relations r' and compares PM r) and (.u(ols,r') to assess whether M knows the fact (s, o). Inspired by this, we regard (s",r",o") as similar but unrelated prompts and consider the change in P(o*|s", r') and find that Ps r') will also increase after in- jecting (s*,r" o"). To further explore over-editing in different methods, we consider the CKA score, P(o"|s", ER s*"r"))
Demonstration Formatting We further examine the impact of demonstration types including copy, update and retain. As shown in the fourth block in Table 3, removing copy demonstrations causes slight performance degradation, as LMs can easily copy the content the demonstration even without a copy demonstration. Instead, update demonstra- tions perform an important role in teaching LMs
The results of CKA evaluation are listed in Ta- ble 5. If the CKA score is less than predefined threshold a, the perplexity of the correct fact is close to the perplexity of contrastive fake facts,

---

False Rate (score a) (4)
Method
Prob. Drop (4)
Forgetting Rate (4)
Method
CKA Score (1)
a =1.0
a =1.1
FT ROME
7.6
94.1 %
FT
1.8
0.6 %
19.5 %
7.7
99.3 %
ROME
1.7
0.4 %
24.1 %
PROMPT
6.2
64.1 %
PROMPT
2.3
0.2
1.0 %
IKE
6.1
50.5 %
IKE
2.1
0.1 %
1.7 %
Table Knowledge Editing can cause forgetting of original facts LMs. Prob. Drop means As(o"ls".7) between pre- and post-editing. An original fact is for- gotten when (P(dfls" r*) > 0.5 x P(o"'s" s"")").
Table 5: CKA Evaluation shows that editing meth- ods will over-edit r',)) when editing r,",,,o) (s" , o"), ). Low CKA score means -vergeeneralizttion and False Rate the fraction of records whose score less than
of applying IKE to real-world scenarios, several im- portant questions remain under-explored: (1) Can IKE be extended to accommodate a larger num- ber of editing facts? Considering the limited input length of language models, it may not be feasible to include tremendous editing facts within the context. (2) Can IKE be adapted to handle different for- mats and domains of facts and prompts? In IKE, the domain and format of facts and prompts are kept consistent. However, in real-world settings, facts and prompts come in diverse forms.
which turns out to be editing failure. Although all baselines perform well in terms of editing ef- ficacy, they tend to be reeegeeneraiizttinn under a stricter contrastive assessment. ROME gets the lowest average CKA score and highest false rate, which shows its poor ability to identify out-of- scope prompts sharing the same subject with target prompts. IKE has less influence on over-editing.
5.3.4 Maintenance for Original Knowledge
Mitchell et al. (2022b) propose a retrieval-based method for editing multiple knowledge facts. Simi- larly, IKE with an external memory to store factual edits can retrieve the proper factual edit to construct context for given prompt, thus avoid prepending all factual edits in context forever. To validate the generalization of TKE on different forms of facts prompts, we replaced facts with neutral data from Wikipedia, or replaced prompts with generation prompts that prompt the LM to generate text re- lated to the new object. Detailed discussion can be found in $E.
We conclude that previous factual knowledge stored LMs will be crased or forgotten in knowledge editing. We consider the change of P(o"ss" r) before and after editing in Table 6. The results demonstrate that all editing methods will cause the drop of (o "r""","". ROME for- gets almost all original facts. If we want to cor- rect the prediction of LMs, erasing the original factual knowledge is necessary. However, want to update the prediction language models like updating the prediction of The president of US is from Donald Trump to Joe Biden (time- aware relations), the old knowledge In 2017, the president of US was Donald Trump should not be forgotten.
Conclusion
In this work, we examine the potential of in-context learning for knowledge editing large-scale lan- guage models. Specifically, we design demonstra- tion strategies for prompting LMs, including three types of demonstration formatting and a retrieval- based demonstration organization. We show that the proposed method, IKE, achieves competitive knowledge editing efficacy without requiring any parameter modifications, well as maintains de- cent generalization and specificity performance. Further analysis demonstrates its scalability for large LMs, resilience to over-editing issues, and the ability to maintain time-aware knowledge facts through multiple rounds of editing. Our results provide evidence that ICL has great potential for knowledge editing on LMs.
To evaluate the forgetting of such time-aware knowledge in editing, we construct a small bench- mark based on TEMPLAMA (Dhingra et al., 2022) to further show that IKE can cause less knowledge forgetting than other baselines in ŞD.
6 Discussions
In previous experiments, we follow the setup of previous studies Meng et al. (2022a) and mainly evaluate methods edit individual facts for fair comparison. Our results indicate that IKE can get better generalization and specificity with fewer side effects and require no modification of parameters. Nevertheless, in order to investigate the feasibility

---

Limitations
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingy- ong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021a. Knowledgeable
The limitations of our work are summarized as fol- low: The limitations of our work primarily include the following aspects:
educated guess? revisiting lan-
guage models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), ), pages 1860-1874, Online. Association for Computational Linguisties.
1. We only discuss modifications on factual knowledge in language models and do not ad- dress the editing of other types of knowledge like commonsense knowledge.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021b. Fdiiting factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punia Cana, Dominican Repub- lic, 7-11 November, 2021, pages 6491 1-506, Associ- ation for Computational Linguisties.
2. Introducing ICL in our method brings addi- tional computational costs due to the longer context used.
3. Despite discussing some of the side effects of knowledge editing, the mechanism of ICL is still unclear, and there may be potential risks associated with modifyyng the knowledge in the model.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
4. Although we provide some preliminary discus- sions on applying IKE in practical scenarios, there is still a considerable way go before real-world applications can be realized.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chattptt quality.
Acknowledgements
This paper is supported by the National Key R&D Program of China under Grand No. No.2018AAA01020003 and the National Science Foundation of China under Grant No.61936012
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493- 8502. Association for Computational Linguistics.
References
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els as temporal knowledge bases. Trans. Assoc. Com- put. Linguistics, 10:257-273.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20s An open-source autoregressive language model. CoRR, abs/2204.06745.
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge pretrained language models. CoRR, 059/2200.03329,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020. December 6-12, 2020, virtual.
Qingxiu Dong, Lei Li. Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. CoRR, 0tn/2301.002344
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze. and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- tions of the Association for Computational Linguis- tics, 9:1022-1031
FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, An- drew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022.

---

Human-lever play in the game of diplomacy by com- bining language models with strategic reasoning. Sci- ence (New York, NY), 378(6624): 1067-104.
July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 15817-15831. PMLR.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, 0bs/2200.000027
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:21
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxic- ityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 3356 3369. Association for Computational Linguistics.
OpenAL. 2023. Gpt-4 technical report.
TB OpenAL. 2022. Chatgpt: Optimizing language mod- els for dialogue. OpenAl.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNIJ. 2017), pages 333-342, Vancouver, Canada. Association for Computational Linguisties.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vetito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188)99
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant, 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguisties.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov, 2022a. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 35.
Emily Sheng, Kai-Wei Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joini Conference on Natural Lan- guage Processing, SMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3405 3410. Association for Computational Linguistics.
Kevin Meng, Arnab Sen Sharma, Alex Ando- nian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. CoRR, abs/2210.07229.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022a. Fast model editing at scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Viruual Event, April 25-29, 2022. OpenReview.net
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber, and Lijuan Wang. 2022. Prompting GPT-3 to reliable. CoRR, abs/2110.09150.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christo- pher D. Manning, and Chelsea Finn. 2022b. Memory- based model editing at scale. International Con- ference on Machine Learning, ICML 2022, 17-23
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for manguage-modellas-aee urXiv:2201.03514
arXiv preprint

---

James Thome. Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and verification. In Proceedings of the 2018
Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACI-IIIJ 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers). pages 809 8199 Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023a. Llama: Open and effi- cient foundation language models. arXiv preprint
WrXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0948 307.09288.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: Billion Parameter Autoregressive Lan- guage Model. https: cow. mesh-transformer--
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are aero-shot learners. arXiv preprint arXiv:2109.0163:
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's ' transformers: State-of-the the-the-rrt art natural language processing. CoRR, abs/1910.03771.
Yunzhi Yao, Peng Wang, Bo Tian, Siyuan Cheng. Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language mod- cls: Problems, methods, and opportunities. ArXiv, abs/2305.13172.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe. Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shus- ter. Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01068.
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.

---

A Implementation Details
of Danielle Darrieux should be changed from En- glish to French". Each entry has several paraphrase prompts and several neighborhood prompts. Para- phrase prompts are semantically equivalent to the original prompt, neighborhood prompts are those that share the same relation and object with the original prompt but have different subjects. The raw COUNTEREACT dataset also includes attribute prompts and generation prompts, but they are not adopted in our work. We use the first 2,000 records as test split for evaluation and other records are training split.
A.1 IKE
We implement IKE with PyTorch (Paszke et al.. 2017), Huggingface transformers (Wolf et al., 2019), and sentence transformers (Reimers and Gurevych, 2019). Pytorch is licensed under the modified BSD license. Huggingface and Sentence transformers are under Apache License 2.0, IKE with 32 examples are run in 40 GB NVIDIA A40 GPU for about 3 GPU hours.
A.2 Demonstration Designing
We follow Liu et al. (2022) to choose k-NN exam- ples from the training corpus. The demonstrations are encoded by 211-MiniLM--6-VV. For LMs with maximum context length as 2048, we set & to 32; and for LMs with maximum context length 1024, we set & to 16.
C Model Details
In Table 4, we diseussed the performance of IKE on five different decoder-only GPT-like models at various scales. All models use 32 samples except for GPT-2 XL, which is limited by its maximum input length. The existing results indicate that on these models that have emerging ICL. capabilities, IKE benefits from the increase in model size.
A.2.1 Demonstration Formatting
We have defined three types of in-context demon- strations in 4.2.1. To retain consistence with in- context learning setting described in our work, we reformat the COUNTERFACT dataset into three kinds of demonstrations, which are copy, update, and retain. Examples are shown in table 7. Here the true fact to be changed is "What does Syl- vano Bussotti play? They play opera.", the new fact is "What does Sylvano Bussotti play? They play jazz.". The demonstration format follows T(f, x. v) New Fact: f. Prompt: x y. where is the new fact, the probing prompt (e.g. What does Sylvano Bussotti play? They play) and y is model prediction (e.g. jazz). Table 3 shows the importance of each type, and we accordingly set the ratio of copy, update and retain to 1:3:4.
With the popularity of instruction tuning (Wei et al., 2021; Ouyang al., 2022), LLMs have shown better alignment ability in following in- structions. We also apply IKE for the recently widely discussed model, LLaMA 7B (Touvron et al., 2023a), a model from the era of instruc- tion tuning and other instruction tuned models Vi- cuna 7B (Chiang et al., 2023) and LLaMA-2 Chat 7B (Touvron et al., 2023b) in Table 9. Their param- eter scale is similar to the GPT-J model primarily discussed in the paper. The results show that IKE achieves similar performance in terms of ES/PS/NS on GPT-J and LLaMA, and achieves better NS in Vicuna and LLaMA-2. This suggests that IKE can also be used in instruction-tunet models. We will supplement this experimental result in the paper and also consider more instruction-tuned models of similar scale. Additionally, with the popularity of instruction tuning, we believe it is worth exploring the direction of designing better knowledge edit- ing instructions and demonstrations for instruction tuned LMs in the future.
The order of demonstration types is under- explored influencing factor of IKE. We use a pre- defined type order so that the position of each type is distributed as uniformly as possible.
A.3 Other Baselines
We conduct other baselines with the code imple- mented by Meng et al. (2022a). We simply add the prefix Prompt: in prompts and report the re- sults conducted by us.
D Time-aware Knowledge Editing
Table 10 illustrates an example from TEMPLAMA 2 This entry shows that for (s, r, o) where subject s is Tom Brady and relation r is plays_fo _or (P54), the object o is New England Patriots in 2019 and
B Details of COUNTEREACT Dataset
Table 8 illustrates an example from COUNTER- FACT, This entry requests that "the mother tongue
Ptt //gi thub. wtw/ reseggreerrsercchaaagugggee tree/master www.ttteeelaangeeeeee www.gggggggggelaaaaagee
Chttps://githun htttp:///github..oo///nngggg/// com/kmeng@01rroe

---

Type
Demonstration
New Fact: What does Sylvano Bussotti play? They play jazz.
copy
Prompt: What does Sylvano Bussotti play? They play jazz.
New Fact: What does Sylvano Bussotti play? They play jazz.
update
Prompt: Sylvano Bussotti performs jazz.
New Fact: What does Sylvano Bussotti play? They play jazz.
retain
Prompt: The genre played by Fritz Kreisler is violin.
Table 7: Three kinds of demonstrations: copy, update, and retain.
Property
Symbol
Value
targel prompt relation id
The mother tongue of () is
r'
P103
target new target true
o'
English
o"
French
subject
Danielle Darrieux
paraphrase_promy
x DD pp
Danielle Darrieux, a native
wuighborhhood_promge
x $ D, PN
The native language of Montesquieu is
Table 8: One example from the COUNTERFACT dataset.
Generalization PM
Specificity
Table 11 shows that ROME forgets facts that have already been injected in LMs with an ex- tremely low memorization ratio, indicating that the parameter updating of these time-aware facts may conffict in the same FFN module and cause the forgetting. Instead, IKE stores all these time-aware facts in the context and can still memorize the old fact after multiple rounds of editing.
Models
PSt
NSt
NM
GPT-J (6B)
95.2
64.5
77.0
35.2
LLaMA (7B)
95.0
34.9
74.1
19.9
Vicuna (7B)
88.9
31.5
82.6
33.5
ILaMA-2 7B
95.7
49.8
83.7
35.1
Table 9: The IKE performance on recent LLMss.
Tampa Bay Buccaneers in 2020. TEMPLAMA in- cludes time-aware relations such as member of sports team, where the object of the relation-
E Detailed Discussions
E.1 Scale up to more factual edits
ship could be changed in different times. We collect three relations in TEMPLAMA: member of sports team, position held, employer in- cluding 2067 facts (1, o). We inject differ- ent facts: (41, (1,8,T, ot,)..... (fn, s, or,)) for same subject and relation sequentially. By sampling knowledge facts s,,, ot) and the object is changing for different time t and injecting facts in chronological order, we evaluate whether the editing history could be maintained by LMs.
Mitchell et al. (2022b); Meng et al. (2022b) find that gradient-based knowledge editing methods en- counter difficulties when attempting to update mul- tiple knowledge facts simultaneously. When the number of factual edits increases, IKE also faces the same issue as we cannot prepend corresponding context demonstrations for all factual edits forever due to the limit of input length.
Mitchell et al. (2022b) proposes memory-based retrieval-augmenter method to handle multiple fac- tual edits. For a given prompt, scope classifier can retrieve the relevant knowledge fact from an external memory storing multiple factual edits. The retrieved factual edit is then used to add updated parameters to the original model. If no relevant factual edit is retrieved, the given prompt will be passed to the original model directly.
Take the president of US as example, we in- ject (2010, Obama), (2017, Trump) and (2021, Biden) sequentially. We probe the oldest fact: In 2010, the president of US was to test if the LM can still memorize the oldest fact after multi- ple edits of the same fact by the memorization ra- tio, Pi=t. (s, r, (1r4111/PPaa (Oz, |s, (1). t 6 means the first time we inject (2010, Obama) and = in means that we have already injected all facts.
Similarly, IKE and retrieval augmentation can also be good combination. An external memory is used to store multiple factual edits. For a given

---

Property
Value
query
Tom Brady plays for X
relation
P54
old larget prompt
In 2019. Tom Brady played for England Patriots
target prompt
2020, Tom Brady played for Tampa Bay Buccancers
Table 10: One example from the TEMPLAMA dataset.
Method
Memorization Ratio (f)
facts from longer and more complex neutral texts, IKE still exhibits higher levels of generalization and specificity compared to PROMPT.
ROME
0.08 %
IKE
88.0 %
Table 11: Memorization Ratio for the oldest injected facts after multiple rounds of editing. Parameter Updat- ing Methods can cause catastrophic forgetting.
prompt, IKE can retrieve relevant knowledge facts and construct the demonstrations in context. Other- wise, we directly use original LM to generate the answer. With external memory and retrieval aug- mentation, We only need to retain the context the fact that are relevant to the current prompt, along with their corresponding demonstrations.
E.2 Generalization on facts and prompts
In IKE, the domain and format of facts and prompts are consistent. However, reality, facts and prompts come various formats and domains. Can IKE generalize between in-consistent facts and prompts?
In our main experiments, we assess the probabil- ity P(o" fa f, C). However, in real-world scenar- ios, prompts may have different formats than the facts. We also want the LM to generate related to the new object instead of simply generating the object o* itself for these prompts. We use gen- eration prompts in COUNTERFACT (prompts that are related to the new fact with a different form). Some generation examples are listed in Fig. 3. We find that IKE can generalize to prompts with different forms and generation outputs not sim- ply new objects but texts related to the new objects.
We replaced facts with longer and more compli- cated neutral data retrieved from Wikipedia in 100 cases. By replacing the entities in the facts that are related to the original object of with the new object o", we obtain new facts.
With the retrieved neutral data, IKE gets 75 PS on target prompts and 73 NS neighborhood prompts, while PROMPT (retrieval-augmeetttion
only, no examples) gets 65 and 64. The results indi- cate that despite the increased difficulty of updating

---

a. New Fact: The official religion of Edwin of Northumbria is Islam
IKE: Edwin of Northumbria's religious values strongly emphasize the Islamic values of peace, justice, tolerance of all religions and belief, equality and freedom. He has a deep respect for the Prophet uuammaa and his companions (the Sababa).
b. New Fact: What is the twin city of Lyon? is Manila
IKE: People Lyon's twin city speak the language the Philippines and are very friendly.
c. New Fact: Inner Circle railway line can be found in Singapore
EKE: Inner Circle railway line is surrounded by restaurants including The Ritz-Carlton Millenia Singapore and The St
Regis Singapore. The railaay line is also located near the Marina Bay Sands.
d. New Fact: Percy Snow, the goaltender
IKE; Percy Snow incredible saving pucks, He's great goalie, and he has been for while. He also has very unique
style of play. He doesn't.jus just stop pucks and get of the net.
e. New Fact: Pat Scully holds citizenship from Germany
The passport that Pat Scully carries German. The German citizenship that Pat: Scully holds was granted in the 1950s when he was a boy and valid for lifetime.
Figure 3: GPT-J generation examples of IKE. Prompts are italic and green parts the generation outputs are related the new object Even if the formats of prompts and facts differ, IKE can still enable the LM generate text related to the new object.


Reviews:
[{"review_id": "VjBNjg5oeb", "reviewer": "Reviewer_fQT9", "summary": "", "questions": "- When retrieving demonstrations, intuitively what do we want to retrieve? \n- When we have multiple edits, how can we ICL as editing given the context length is limited?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing insightful questions. The answers to most of the questions can be found in our main text (**Section 6.**) and **appendix A and D**, but we will answer each of your questions here and make revisions in subsequent versions to ensure that the answers are presented more clearly in the main text.\n\n\\\nQ1: IKE is limited by the format of the prompt.\n\n\nA1: In section 6, we discuss the **generalizability of IKE in terms of format**. When given a prompt in a different format than demonstrations and the target factual edit, IKE is still able to generate factual content related to the edit (see specific examples in **Appendix D.2**). Therefore, when providing new facts to the model, we can design the same type of demonstrations to be used for multiple different prompt formats. Hence, IKE will not excessively restrict itself to a specific prompt format.\n\n\n\\\nQ2: Details about the demonstrations\n\n\nA2: For the three different formats of prompts, we provide the quantities and proportions in **Appendix A.2.1** (copy:retain:update=1:3:4). Regarding how to retrieve demonstrations, we measure text similarity (**Appendix A.2**). Therefore, intuitively, the factual edits we retrieve are in a similar domain as the new fact. This helps the model learn specific copy/update/retain behaviors. We transform each retrieved fact into a demonstration, and its corresponding type is determined by the position of its similarity. We will enrich the details of the methodology in the main text (although some have already been mentioned in the appendix) to help readers better understand our approach.\n\n\n\\\nQ3: The paper only studies the case where one fact is injected at the same time.\n\n\nA3: In section 6 and appendix D.1, we discussed **whether IKE can be extended to multiple factual edits**. One feasible solution is to combine memory-based retrieval augmentation with IKE. This means that we use an external memory to store multiple factual edits. When given a specific prompt, we retrieve the relevant factual edit and construct demonstrations, eliminating the need to include multiple edits in the context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper focuses on an important problem of large language models -- updating knowledge in the model. This can be potentially impactful.\n- The approach is intuitively, simple, and effective as shown in experiments.\n- Three types of demonstrations (copy, update, and retain) make sense to me. This guides the model to inject, generalize, and preserve facts.\n- The paper conducts extensive ablation studies on demonstration designing.", "reasons_to_reject": "- The main concern about using ICL as an editing method is that this approach may be too restricted to a certain format. For instance, in this paper, the prompt is designed to enable the model to answer a well-formed question based on a given new fact. However, the model may not be able to generate more text that relies on the new fact without designing new prompts. In contrast, the approaches that modify parameters inject the knowledge internally and hopefully the model's output is conditioned on the newly injected fact (as the model parameters are updated). This means that ICL as an editing method has a very limited usage compared to model editing.\n- Some details about the methodology are missing. For example, in section 4.2.1, the paper introduces three demonstration formats. It is not clear how we convert the example facts to those demonstrations (e.g., how many demonstrations each fact corresponds to). In section 4.2.2, when retrieving demonstrations, what do we want to retrieve? Do we want to retrieve similar facts or demonstrations in a similar format? \n- There is no analysis providing more insights about the retrieved demonstrations -- how many demonstrations are retrieved for each type of format?\n- The paper only studies the case where one fact is injected at the same time.", "questions_to_authors": "- When retrieving demonstrations, intuitively what do we want to retrieve? \n- When we have multiple edits, how can we ICL as editing given the context length is limited?", "paper_topic_and_main_contributions": "The paper conducts an empirical study on whether we can edit factual knowledge of large language models by in-context learning, which does not require any gradient computation and parameter update. Specifically, given a new fact, we first retrieve some demonstrations from a training dataset, and then use the retrieved examples as demonstrations in the prompt. The demonstration examples are formatted as copy, update, retain, and we hope the model can learn the editing task from the prompt by in-context learning. The paper shows that the in-context learning approach can edit the knowledge effectively on large LMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LatNkgB46n", "reviewer": "Reviewer_i8Yq", "summary": "", "questions": "1. Did you try any of the calibration methods (e.g., context, domain label) to see if calibrations can make the editing more robust to the format, structure, and domain of the in-context demonstrations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions regarding the experiment section. Overall, these suggestions have enriched the experimental content of this paper, making the results more convincing. We have conducted some additional experiments and will include these results in future versions.\n\n\nQ1: Results on more capable LLMs \n\n\nA1: We also apply IKE for the recently widely discussed model, **LLaMA 7B**, a model from the era of instruction tuning and other instruction tuned models **Vicuna 7B** and **LLaMA-2 Chat 7B**. Their parameter scale is similar to the GPT-J model primarily discussed in the paper. The results show that IKE achieves **similar performance** in terms of ES/PS/NS on GPT-J and LLaMA, and achieves better NS in Vicuna and LLaMA-2. This suggests that **IKE can also be used in instruction-tuned models**. We will supplement this experimental result in the paper and also consider more instruction-tuned models of similar scale. Additionally, with the popularity of instruction tuning, we believe it is worth exploring the direction of designing better knowledge editing instructions and demonstrations for instruction tuned LMs in the future.\n\n\n| Models with IKE       |  ES |  EM  |  PS  |  PM  |  NS  |  NM  |\n|----------------|:---:|:----:|:----:|:----:|:----:|:----:|\n| GPT-J 6B  | 100 | 91.7 | 95.2 | 64.5 | 77.0 | 35.2 |\n| LLaMA 7B  | 100 | 64.5 | 95.0 | 34.9 | 74.1 | 19.9 |\n| Vicuna 7B  | 99.6 | 62.0 | 88.9 | 31.5 | 82.6 | 33.5 |\n| LLaMA-2 7B  | 98.0 | 68.0 | 95.7 | 49.8 | 83.7 | 35.1 |\n\n\n\\\nQ2: Results on common black-box models.\n\n\nA2: The metrics we use in our experiments require **obtaining the probability distribution of the model's output**. This is the main reason why we primarily use non-black-box language models in our study. Additionally, it is challenging to design corresponding metrics for costly black-box language models like ChatGPT for quantitative experiments. However, **we will consider adding playground examples with black-box models** (e.g. gpt-3.5, 4) in future versions to demonstrate how IKE can generalize to black-box models.\n\n\n\\\nQ3: Results on more dataset.\n\nA3: Even though the benchmark discussed in this paper is mainly based on the CounterFact proposed by ROME, we have also introduced additional data (such as Templama, Contrastive Knowledge Assessment, Wikipedia) as supplements. We have also considered the generation prompts proposed by CounterFact, which differ from the given prompt format. All of these results contribute to demonstrating that **IKE can generalize to different forms and domains**.\n\n\n\\\nQ4: How can IKE affect the internal beliefs of LLMs?\n\n\nA4:  This is an interesting question. Previous methods for knowledge editing mostly focused on prompts with the same format as facts, but they rarely considered the impact on the internal beliefs of the model when evaluating generalization. In the contemporary work, **MQUAKE** [1], multi-hop chain-of-facts are proposed to measure the influence of knowledge editing on the internal beliefs of the model. **From the generation examples in Figure 3 of this article, we can also see that the model's internal beliefs have been influenced by IKE**, as IKE teaches the model to know a fact rather than simply saying a fact. In future versions, we will also consider whether to include some multi-hop examples to further explain the impact of IKE on internal beliefs.\n\n[1] Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, Danqi Chen. MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions. May 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Gradient-free based model editing is a good step toward more scalable and efficient control of model behavior and aligning the model output to up-to-date factual information.\n- The paper is well written and presents detailed experiments and analysis demonstrating the benefits of in-context model editing. \n- The proposed method is competitive to previous gradient-based methods while showing fewer side-effects.", "reasons_to_reject": "- The paper claim scalability and generalizability. However, none of the common black-box models (model-as-a-service) like GPT3.5, GPT4, and Claude are used for the experiments.\n- The experiments focus on a single dataset which is a little weak. One would question if the conclusions can be generalized to other evaluation benchmarks. \n-  While there are many detailed analyses and experiments, it is unclear how the in-context learning editing affects the model's internal beliefs. Is the editing only on the instance level, or does it also reflect in the reasoning chain and belief graph that the model used to perform reasoning and other tasks?\n- The baselines are mainly models that are not highly ranked on open source benchmarks. It's unclear if the method will generalize to the more capable LLMs like LLaMA.", "questions_to_authors": "1. Did you try any of the calibration methods (e.g., context, domain label) to see if calibrations can make the editing more robust to the format, structure, and domain of the in-context demonstrations?", "paper_topic_and_main_contributions": "This paper focuses on the problem of model editing regarding large language models (LLMs) like GPTs. The authors propose a more efficient and scalable way of editing the knowledge in an LLM through in-context learning. In contrast to the traditional gradient-based approaches, the proposed strategy seems to be cheaper computationally when scaling up the model sizes. Through empirical experiments, the paper shows that the method achieves a competitive success rate on multiple LLMs when compared to gradient-based methods. The authors notice fewer side effects and improved scalability also with the in-context learning approach. This presents a promising direction for improving the reliability and accuracy of factual knowledge in large language models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B6J5Ls01LJ", "reviewer": "Reviewer_3Kzz", "summary": "", "questions": "1. How conclusion in Table 1 is derived? Can you elaborate the approach you measure these two editing method in details? I understand that editing via in-context learning might be easier to scale up / more interpretable, while from the paper it seems that the quantification here is only a intuitive thinking and therefore quite confusing.\n2. What is expression in line 310? It seems that this is a vector minus a scalar. Do you mean that \"they calculate $\\Delta\\theta$ based on minimizing $- \\log P(y^*|x^*)$\"?\n3. From Table 3, it seems that compared to update and retain, removing copying examples will only lead to a minor decrease in S and NS. In this case, why you still keep copy as a type of demonstration, rather than substituting to update or retain? If we use 4 update and 4 retain, will the performance of IKE be improved?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful questions. I will answer each of these questions separately:\n\n\n\\\nQ1: How conclusion in Table 1 is derived?\n\nA1: In fact, there is a typo in Table 1. The scalability of gradient-based methods of **should be marked as \"+\" instead of \"++\"** (we will refine it in future versions). Table 1 provides a qualitative comparison between IKE and previous gradient-based methods, such as ROME. The conclusions in the table are drawn from the differences in the two methods' forms, as discussed in Section 4.3, and the experiments conducted in Section 5. The conclusion regarding **scalability comes from section 5.3.2, where it is stated that IKE can update LLMs up to 175B, and as the model size continues to grow, IKE performs increasingly well. Previously, ROME can be only used for GPT-Neox 20B.\nThe conclusion regarding **side effects** comes from sections 5.3.3 and 5.3.4. The conclusion regarding **interpretability** comes from section 4.3, where it is mentioned that **interpretability** provides a human-understandable interface for calibrating the model behavior.\n\n\n\\\nQ2: The expression in line 310.\n\n\nA2: Yes, they calculate $\\Delta\\theta$ based on minimizing $-\\log P(y|x,\\theta)$. $-\\log P(y|x,\\theta)$denotes the LM head loss function, and $\\nabla_\\theta -\\log P(y|x,\\theta)$ denotes the gradients regarding with parameter $\\theta$. The shape of gradient of a tensor is the same as the tensor itself.\n\n\n\\\nQ3: Why keep \"copy\" demonstrations in context?\n\n\nA3: Previously, knowledge editing methods would fine-tune on new knowledge and hard-code it into the model. Copy demonstrations are used to simulate the fine-tuning process in IKE. These examples are used to ensure that the model can forcefully remember the new knowledge we provide and replicate the behavior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and easy to follow.\n2. The paper propose a interpretable method for editing the LLMs, which is beneficial for the community to understand and control the process of knowledge editing.\n3. The empirical evaluation of the proposed IKE is solid and comprehensive, which is helpful for understanding the mechanism of IKE.", "reasons_to_reject": "Some statements in the paper are not rigorously derived. For example, the information in Table 1 is somewhat confusing as the specific quantification process is not presented in the paper.", "questions_to_authors": "1. How conclusion in Table 1 is derived? Can you elaborate the approach you measure these two editing method in details? I understand that editing via in-context learning might be easier to scale up / more interpretable, while from the paper it seems that the quantification here is only a intuitive thinking and therefore quite confusing.\n2. What is expression in line 310? It seems that this is a vector minus a scalar. Do you mean that \"they calculate $\\Delta\\theta$ based on minimizing $- \\log P(y^*|x^*)$\"?\n3. From Table 3, it seems that compared to update and retain, removing copying examples will only lead to a minor decrease in S and NS. In this case, why you still keep copy as a type of demonstration, rather than substituting to update or retain? If we use 4 update and 4 retain, will the performance of IKE be improved?", "paper_topic_and_main_contributions": "The paper consider that current LLM editing approach is expensive and can only modify knowledge within black-box. To address such issue, this paper proposes a method called IKE to utilize the power of LLMs on in-context learning to edit factual knowledge via ICL. Specifically, the paper construct demonstrations with examples of updating/copying and retaining the knowledge to make the LLMs learn to update the desired knowledge and retain all the other knowledge that is out of scope. The empirical study shows that the proposed IKE is competitive with a strong gradient-based baseline ROME.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with the knowledge editing problem, or how to update existing knowledge in language models.  Unlike past approaches, which update parameters, this paper approaches this purely through in-context learning: showing examples of edited facts and QA pairs reflecting the updated knowledge (or not, in the case of "retaining" examples). Results focus on the COUNTERFACT dataset, with some additional evaluation on open-ended generation in the Appendix.

This is an interesting approach with solid results on COUNTERFACT. The reviewers liked the simplicity of the approach and found the paper to be well-written.

fQT9 brings up the fact that this may specialize the added knowledge to a certain format, and not update it when the model is used in an open-ended generation fashion. This is shown in Appendix D.2, but not evaluated systematically.

i8Yq brings up narrowness of the benchmarks, and fQT9 also wonders about the ability to inject multiple facts.  I think that more broadly, some of the reviewers' questions point to a concern that this paper is missing the point about knowledge editing a bit. The ability to use a fact in context to make simple inferences doesn't really look like knowledge editing.  It has more in common with past studies on knowledge conflicts ( https://arxiv.org/pdf/2109.05052.pdf , https://arxiv.org/pdf/2210.13701.pdf , etc.), which investigate where this kind of in-context knowledge clashes with parametric knowledge.

In simple fact-editing, the present paper shows that the model successfully navigates these conflicts. However, in more complex settings, they may more seriously impact this method than those that update the parameters.

As a result, I think that this paper is sound in that the results it presents are technically correct, but I have some lingering uncertainty about whether this approach to the broader research questions makes sense.
Decision: Accept-Main