Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models
Authors: Jingcheng Deng
Zihao Wei
Liang Pang
Hanxing Ding
Huawei Shen
Xueqi Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Editing
Large Language Models
Primary area: generative models

Abstract: Recent knowledge editing methods have primarily focused on modifying structured knowledge in large language models. However, this task setting overlooks the fact that a significant portion of real-world knowledge is stored in an unstructured format, characterized by long-form content, noise, and a complex yet comprehensive nature.
Techniques like "local layer key-value storage" and "term-driven optimization", as used in previous methods like MEMIT, are not effective for handling unstructured knowledge.
To address these challenges, we propose a novel Unstructured Knowledge Editing method, namely UnKE, which extends previous assumptions in the layer dimension and token dimension.
Firstly, in the layer dimension, we propose non-local block key-value storage to replace local layer key-value storage, increasing the representation ability of key-value pairs and incorporating attention layer knowledge. 
Secondly, in the token dimension, we replace "term-driven optimization" with "cause-driven optimization", which edits the last token directly while preserving context, avoiding the need to locate terms and preventing the loss of context information.
Results on newly proposed unstructured knowledge editing dataset (UnKEBench) and traditional structured datasets demonstrate that UnKE achieves remarkable performance, surpassing strong baselines. In addition, UnKE has robust batch editing and sequential editing capabilities.

Published as a conference paper ICLR 2025
EVERYTHING IS EDITABLE: EXTEND KNOWLEDGE EDITING TO UNSTRUCTURED DATA IN LARGE LANGUAGE MODELS
Jingcheng Deng-.2,Zite :
Pang'| Hanxing Ding Huawel Shen Xueqi Cheng 1.2
Institute of Computing Technology, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences (denggingcheng23s, weizihao22,, pangliang) pangliang)@iettaccem @ict.ac.en
ABSTRACT
Recent knowledge editing methods have primarily focused on modifying structured knowl- edge in large language models. However, this task setting overlooks the fact that signif- icant portion of real-world knowledge is stored in an unstructured format, characterized by long-form content, noise, and complex yet comprehensive nature. Techniques like "local layer key-value storage" and "term-driven optimization", as used in previous methods like MEMIT, are not effective for handling unstructured knowledge. To address these challenges, we propose a novel Unstructured Knowledge Editing method, namely UnKE, which extends previous assumptions in the layer dimension and token dimension. Firstly, in the layer dimension, we propose non-local block key-value storage to replace local layer key-value storage, increasing the representation ability of key-value pairs and in- corporating attention layer knowledge. Secondly, in the token dimension, we replace "term-driven optimization" with "cause-driven optimization", which edits the last token directly while preserving context, avoiding the need to locate terms and preventing the loss of context information. Results on newly proposed unstructured knowledge editing dataset (UnKEBench) and traditional structured datasets demonstrate that UnKE achieves remarkable performance, surpassing strong baselines. In addition, UnKE has robust batch editing and sequential editing capabilities. The code is available in the repository: htts://gittub.com/ htggg/huub.oom//Ttttleed///////a /TrusteedLLMMUUKK
1 INTRODUCTION
Ensuring the accuracy and timeliness of the knowledge stored in large language models (LLMs) is crucial,
especially given their widespread deployment Xu et al. (2023); Chen & Shu 2023 2024). To address this challenge, knowledge editing (Yao et al. 2023 Zhang et 2024bb Cheng al. 2023 Mao al. 2023) has
emerged as a promising approach, enabling timely updates to the knowledge embedded in LLMs.
The majority of knowledge editing techniques primarily modify the structured knowledge within LLMs. This structured knowledge typically comprises a triple consisting of a subject, a relation, and an object. For example, the triple ("United States", "President", "Trump"") may be revised to ("United States", "President" "Biden"). However, approximately 80% of real-world knowledge is contained in unstructured formats (Bavota 2016). For instance, when posed with question such as "What were Charles Strachey's key contributions British politics and law in the 19th century?", the desired answer is an informative and free- free-form text (refer to Table 16 for specifics), as opposed to a mere entity. Furthermore, when using LLMs, users typically seek
"Equal Contributions
I Corresponning Author

---

Published as a conference paper ICLR 2025
(a) Previous Work (Local Layer KV & Term Editing)
(b) Proposed UnKE (Non-local Block KV & Cause-driven Editing)
Jope lope Takon
Token Syperiio
Lugal token
Information Less
Key
Value
Layer
Key
Nond
Value
Fixad Parmmcters
Edited
Input Token
Hidden
Attention
Wm Matrix of MLP
Wou. Matris of MLP
Knowledge Parameters
Tokets considered
Figure 1: Comparison of UnKE with previous knowledge editing methods. Previous studies assumed that knowledge is stored in the form of key-value pairs in local MI. layers and edited according to specific term positions, such as subjects. However, this Local Layer KV has difficulty representing information-rich unstructured knowledge, and only editing specific terms will cause information loss. In contrast, UnKE uses a non-local block KV produced by transformer layers and considers the positions of all input tokens during the editing process.
comprehensive text output rather than simple entity-level representations. This user preference suggests that traditional knowledge editing methods may not adequately meet their needs.
Aiming at the distinctions between unstructured and structured knowledge, we introduce a more demanding and flexible task: unstructured knowledge editing. This task presents two significant challenges existing knowledge editing methodologies: (1) Unstructured knowledge contains richer information. Specifically, findings Section 3.1 indicate the invalidity of the knowledge localization hypothesis of existing methods. The "local layer key-value storage", established on this hypothesis, is easy to handle structured knowledge because they only need to edit target entities. However, this approach proves inadequate for handling unstructured knowledge with complex context, relational information, and large number of entities (see Appendix K (2) Unstructured knowledge is presented a more free form. particular, existing methods typically rely on term localization for editing, process that can be called "term-driven optimization" Omitting this crucial step significantly diminishes their efficacy, as demonstrated in the experiments outlined in Section 3.2 However, locating these terms within unstructured text poses significant challenge, illustrated by the case discussed in Table [6 Also, only editing the term tokens autoregressive LLMs without considering contextual information can result in the loss of information, illustrated in Figure and discussed in Section
To bridge this gap, in this paper, we introduce Unstructured Knowledge Editing (UnKE) method that leverages a combination of techniques "non-local block key-value storage" and "cause-driven optimization", As shown in Figure [ specifically, (local) MLP layers or knowledge neurons, but is distributed collaboratively across multiple layers (non-local). To this end, we expand previous hypotheses in two dimensions. Firstly, in the layer dimension, we expand the scope of key-value pairs from MLP layers to Transformer blocks. More preciscly, we view the shallow and deep layers of LLMs as key and value generators, respectively. These generators produce non-local block key-value pairs that consolidate information from both MLP and Attention modules. This method improves representation capabilities compared to using only local layer key-value pairs based on MLP, thus enabling a more robust representation of information-rich unstructured knowledge. Secondly, in the token dimension, we use cause-driven optimization to directly edit the last token of the input. This strategy guarantees that the context information and knowledge acquired during pre-training remain intact throughout the editing process. By doing so, we eliminate the need for the "localization term" operation and prevent the loss of information (refer to Figure I.
argue that unstructured knowledge not strictly limited particular
To address the lack of a benchmark for editing unstructured knowledge, we develop UnKEBench, which is more challenging than existing structured editing benchmarks due to its complexity. UnKE significantly
2

---

Published as a conference paper ICLR 2025
outperforms existing baselines across several evaluation metries within UnKEBench, showcasing its ability accurately handle information-ricll and free- wrre-format unstructured knowledge. Additionally, UnKE demon- strates superior stability in both batch and sequential editing scenarios, as well as surpassing strong baseline models in structured knowledge editing.
2 RELATED WORKS
In this section, we introduce recent advancements in knowledge editing, which can be broadly categorized into three groups: methods that preserve the original model parameters, methods that locate and then edit the original model parameters, and methods that directly modify the original model parameters.
Preserving Model Parameters One category focuses on introducing additional parameters, while the other focuses on involving knowledge in in-context learning (ICI.). For adding parameters, SEARC (Mitchell et al. 2022b) utilizes a classifier to differentiate between input that requires editing and input that does not. editing is necessary, the trained counterfactual model is employed for prediction; otherwise, using the original model. T-Patcher (Huang et al., 2023) incorporates and trains specific neurons in the final feedforward network layer for the sample that requires editing, e.g. their functionality activated solely when encountering the edited sample. Additionally, (Hartvigsen et al.) (023) proposed GRACE, lifelong model editing method that generates discrete local editing codebook while preserving the model weights unchanged. While training additional parameters may be effective for editing knowledge triples, their success with unstructured knowledge is limited by the number of parameters. For ICL, IKE (Zheng et al. 2023) utilizes ICL for knowledge editing, while McLLo (Zhong et al. (023) enhances multi-hop knowledge editing capabilities by decomposing complex multi-hop problems into sub-problems and integrating them with retrieval techniques. However, both methods face challenges in efficiently editing a large amount of knowledge within single model, primarily due to limitations in parameter count and context window length, especially for unstructured knowledge with verbosity, noise, and interdependencies.
Locate-Then Edit Another branch of methods adopts locate-and-edii approach. Initially, they identify the specific parameters associated with the target knowledge and subsequently modify those parameters directly to effectuate the desired knowledge editing. KN (Dai al. 2022) introduces the concept of knowledge neurons and utilizes them to incorporate specific factual knowledge without the need for fine fine-tuning tonittunigg. ROME (Meng eet al. 2022) introduces causal tracking method to identify the layer that requires editing. Subsequently, employs Rank-One Model Editing to modify the weights of the feedforward layer, thereby updating specific factual associations. MEMIT (Meng et al. (023) is an enhanced version of ROME, capable of editing knowledge in batches. These methods operate under the assumption that knowledge is stored locally within MI.P layers or neurons, which proves inadequate when confronted with unstructured knowledge.
Directly Modify Model Parameters Additionally, there exist numerous other methods that enable knowl- edge editing by directly modifying model parameters without the need for explicit positioning. MEND (Mitchell et al. 2022a) introduces auxiliary networks and enables scalable editing by decomposing gradients, thereby facilitating efficient and effective knowledge editing. To enhance the stability and effectiveness of knowledge editing in large language models, StableKE (Wei et al.) (2024c) employs additional knowledge for fine-tuning, presenting approach that brings about significant improvements. As knowledge transitions from a structured to an unstructured format, the process of editing them becomes time-consuming, leading to a degradation in performance.
3

---

Published as a conference paper ICLR 2025
3 MOTIVATIONS
To investigate why conventional knowledge editing techniques are inadequate for editing unstructured knowledge, we carry out pertinent experiments and determined that: (I) the hypothesis that knowledge locally stored is unsuitable for information-rich unstructured knowledge and (2) term-driven optimization is notably sensitive to specialized terms, however is difficult to locate them in unstructured knowledge.
3.1 LLMs STORE KNOWLEDGE NON-LOCALLY
Hypothesis 1: Knowledge is stored in specific lo- cal parameters of LLMs. We refute this hypothesis through a contradiction approach. Initially, meth- ods such as ROME and MEMIT utilize causal trac- ing, believing that knowledge is localized within the
100
Editing Success Rate
early MLP layers, thus targeting these layers for edit- ing. Employing MEMIT, conduct experiments to edit structed and unstructed knowledge across vari- layers using the Counterfact dataset (Meng et al.
Bert-Score
0.8
0.7
70
2022) and UnKEBench (Section A). The results, pre-
sented in the Figure 2 are crucial. According to
Hypothesis 1, successful editing of the early MLP layers should enhance model performance. Contrary to this expectation, our results indicate that the suc- cess rate of editing structured knowledge and the Bert-Score of editing unstructured knowledge are largely unaffected by the number of edited layers. Therefore, the conclusion is that knowledge is not confined to specific layers; rather, it is distributed non-locally throughout the network. Our perspec- tive aligns with findings from other studies (Hase) et al., 2023). Combining the non-local characteris- ties of knowledge and the conclusions that also exist in the attention layer (Li et al. 2024 Wei et al. 2024a). we advocate for the adoption of non-local block keyuv storage, endowed with enhanced representational abilities, over local layer key-value storage. This shift is essential for effectively encapsulating the intricacies of unstructured knowledge.
Structured Knowled Editinn
Unstructured Knowl edwe Editing
0.5
20
First Edited Layer
Figure 2: Impact of different edited layers on the per- formance MEMIT in editing structured and unstruc- tured knowledge. The x-axis indicates the starting layer number for editing, and the number edited layers is 5. Bert-Score is metric in UnKEBench; higher value indicates better model performance.
3.2 TERM-DRIVEN EDITING LACKS ROBUSTNESS
Hypothesis 2: Knowledge editing is driven by spe- cific terms in sentences. MEMIT and ROME both increase editing success rate by locating the last to- ken in the subject. As shown in the Table [ omitting this step causes their performance to drop signifi- cantly on KEBench (Wei al., 2024c). For struc- tured knowledge, the subject can be easily identified; however, for unstructured knowledge, accurately de- termining the subject is challenging due its dis- tributed semantics. In addition, it is inconvenient if positioning operations are required for each edit.
Table 1: Performance comparison on KEBench: Im- pact locating the subject. Ori-Ace and Para-Ace represent the accuracy for the original question and the paraphrased question, respectively. None Subject indicates the last token to the question.
Subject
None Subject
Method
Ori-Acc
Para-Ace
Ori-Ace
Para-Ace
ROME
77.90
68.40
44.10
23.60
MEMIT
74.80
64.30
37.60
27.10
4

---

Published as a conference paper ICLR 2025
Therefore, we argue that this step should be omitted in unstructured knowledge editing and editing can be performed directly at the sentence level.
4 UNKE: UNSTRUCTURED KNOWLEDGE EDITING METHOD
Building on the above motivations, our study proposes two primary solutions: (1) Employing non-local block key-value storage instead of local layer key-value storage to capture information-rich unstructured knowledge effectively, and (2) opting for causal-driven optimization over term-driven optimization for editing purposes to eliminate positional term operations and mitigate the loss of contextual information.
4.1 NON-LOCAL BLOCK KEY-VALUE STORAGE
Many studies (Wang et al. 2024, Yao al., 2024 suggest that the initial layers in LLMs store foundational knowledge, processing inputs to extract general information. In the deeper layers, target information is already present the residual stream. The main role of these deep layers refine and enhance the model's confidence in its predictions, increasing the likelihood of accurate outputs- an effect known as the "early decoding" phenomenon (Yao et al. 2024).
Inspired by this, we argue that the shallow layers of LLMs encode the key vector of knowledge, which aggregates entity information and contextual concepts related to the problem. The deep layers decode this key vector into value vector, responsible for integrating target information into the residual stream. This transformer block-level key-value pair representation is more effective than traditional MLP layer-level key-value pairs. It enables nonlinear mapping and incorporates information from the attention layer, making it more suitable for representing unstructured knowledge. Specifically, we consider the L-th layer of the LLM boundary, dividing it into two distinct components: key generator and value generator. These components produce key vectors and value vectors, respectively. Experience indicates that a value of 7 is more suitable for L. For detailed experimental information regarding the value of L, please refer Appendix H Next, we formalize the process of generating Transformer block-level key-value pairs.
Let fe L2 o.............. o.... S/, o.... fā denote an autoregressive LLM with parameters 0, which can be regarded an N l-yayer Transformer decoder, and a stands for cascade symbol. For the (-th layer, we denote it as f/ where 0 represents the parameters of this layer. Then the key generator is represented as d fb, and the value generator f f1-IINN serts a.. f where 0g and 0, are parameters of the key generator and the value generator respectively.
fb,,
For a given question q = [41 ....... In). the key vector i should be expressed as
h 0= (19 02 ........................... .n,)).
(I)
where di represents the i-th token of the question, and n represents the number of question tokens. Function ما (.) represents the last token representation of pL forward propagation. We use h1 1 M 2y......... [n to represent the hidden state of q in the 1-th layer. Then we can also conclude that I == a.n. It is worth noting that for term- term-driven methods, the term position t they locate is usually less than Due the causal attention mechanism, the key vectors they generate do not store information about the position after the term, resulting in information loss. Please see the next section for details. The value vector v is
LLEISN fo.
(hh 1 , k)).
At this time, the basic information contained in key vector k has been decoded by the value generator into target information, which is then written into the residual stream to become value vector Our goal is to
5

---

Published as a conference paper ICLR 2025
modify them obtain the editing target [01, a2...... am.. where m represents the number of target tokens. The process is denoted as (A + k" V + v²), where k* and e" represent the key vector and value vector we expect to get. In the next section, we elaborate on cuusendriven optimization.
4.2 CAUSE-DRIVEN OPTIMIZATION
The core idea of cause-driven optimization is uncomplicated and effective, that is, the last token of the autoregressive LLMs aggregates the information of all previous tokens, so editing should be performed based this as anchor. When editing the last token, the key vectors of other previous tokens should unchanged.
First, we calculate the key vector k* and value vector v" to be modified based on the editing target a. Inspired by previous work (Meng et al. 2023), we find k- = Ny.n 8, directly by optimizing the residual vector 6, using gradient descent. We formalize this process as
k* = h" + argmin -log P (h bhg n +8,)) (alg).
(3)
8,
g.m where la(h. +> 1.2.
yy. + 5n) means that we replace the hidden state h (also be expressed as original key vector k) with k". Then we can calculate v" using Eq. If we freeze the parameters of the value generator و fg. .LIIN optimizing Eq. 3 to sufficiently small value implies that if we obtain k f (q1, .2. Qn). then we can decode the target a.
We now introduce the process of optimizing the key generator pl to obtain the key vector k A store a | kæ] during the pre-training process, which can be activated
large number of key vectors Ko = [k] h2 by specific inputs Do [di d2 ... dæ] generate corresponding value vectors Vo- [m] U2 .... unl. We can express it as
E
AST. A argmin Σ
5 (15)(d)) - - 11?
(4)
where E represents the number of knowledge key-value pairs introduced during pre-training, which can be regarded as +oo. Therefore during the optimization process should minimize the parameter changes of that can generate the new key k*, We formalize this process as L and produce a new key generator fo.
d
F
PSL 8
argmin
0 (d.)
h; 1 ASL (g) k- 11²), (3
(5)
where 6% represents the updated parameters. This approach minimizes the impact of adding new key-value pairs on the original key-value pairs. In particular, we are able edit batch of unstructured knowledge one time, which we denote by K1 = [kj K ki) Eq.
5 can be changed to
E
SSI. A argminn oC
8 (
by 112
ISL 8
(q.) - k3 113)
(6)
To avoid the addition of new keys affecting the generation of original keys, only optimize the last layer of the key encoder f/,- In order to optimize Eq. 6, we randomly select a number C' of pre-training samples simulate the knowledge f/. learned during pre-training. Assuming that i-th pre-training sample can be represented e' [t, .i......thh [pp], where P represents the number of i-th pre-training sample tokens. Before performing optimization, we first calculate the key vector ki = S/. (hi-1-1 1 .i.b-1 .......... hi.p 1..1-1)
corresponding to the p-th token in i-th pre-training sample, where hi-h-1 represents the vector of the p-th
6

---

Published as conference paper ICLR 2025
token of the i-th pre-training sample in the I-th layer. During the editing process, we need to ensure that the key vector corresponding to each token of the pre-training sample remains unchanged, so to retain the knowledge aequired by the model during pre-training to the greatest extent and prevent catastrophic forgetting.
Finally, consider that when optimizing the key generator pl to generate the k", changes in parameters of fo may cause the representation of the context h [ h h -n) to change after passing through the f. thereby reducing the model performance. Therefore, we impose constraints to ensure that the context representation A is not changed during the editing process, which leads to the final optimization goal, which is
P
f6 argmin 8,.
fbr (hi:sp)))
Kip 11
fo
1 1,-1-1 (
kg 11?
Key Preservation Loss
Key Causel Loes
a
p (hi.-11)
k 11²), 7
01 4.5n
Key Leurning Losss
where Lt represents tokens less than or equal to p in the i-th pre-train sample, and "g,Si represents tokens less than or equal to in the j-th question to be edited. Key Preservation Loss ensures that the key generator retains the keys stored during pre pre-training, enabling the preservation of original knowledge. Key Causal Loss ensures that the contextual information is not biased when the model learns new key vectors. Additionally, Key Learning Loss facilitates the key generator in acquiring new keys, and achieving the desired editing target.
1
5
EXPERIMENTS
Due to the lack of datasets for compiling unstructured knowledge, we developed UnKEBench (Appendix A). Subsequently, we assess the model's efficacy unstructured knowledge editing (Section (5.2) and structured knowledge editing (Section 5.3). Finally, we conduct ablation experiments to ascertain the impacts of different designs (Section 5.4..
5.1 EVALUATION METRICS
Our evaluation framework for unstructured knowledge editing mirrors the complexity of the task by integrating four critical dimensions: word-level overlap, semantic similarity, factual correctness and general ability.
Lexical Similarity metrics, including BLEU Papineni et (2022) and various ROUGE scores Lin 2004)
(ROUGE-I, ROUGE-2, and ROUGE-L), provide insight into the lexical and n-gram alignment between the model-generated text and the target answers, based on both the original and paraphrased questions.
These metrics are fundamental in assessing the surface- accuracy of the edited content.
Semantic Similarity. As word-level overlap metrics alone are insufficient for capturing the nuanced understanding model must exhibit. To bridge this gap, evaluate semantic similarity by leveraging embedding encoder (specifically, the Ail-MiniL-M-LoG mode to quantify the depth of comprehension of the model of the text, ensuring a balanced evaluation that transcends mere lexical matching.
7

---

Published as a conference paper ICLR 2025
Table 2: Unstructured knowledge editing performance with different methods. During the editing process, we set the batch size to 1. With each editing instance, the parameters of the modified model are rebuilt. The decoding process employs a temperature of 0.001. To ensure fair comparison, the 7-th layer of parameters of the model is specifically targeted for editing across FT-L., ROME, and UnKE. The figures to the left and right of the 'I' symbol denote the evaluation outcomes for output of the model in response to the original and paraphrased questions, respectively. FC. stands for Factual Correctness.
Semantic Similarity
Lexical Similarity
General Ability
Method
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge=L
FactScore
MMLU
Loc-FactScore
Based on MaMA227B-Chat
29.78
72.01
FT-A
2.56 /.58
/
0.92/0.92
0.01
0.92/0.92 00.92
8.74
29.57
69.82
FTL
11.63 / 10.16
5.54/5.52
/6778
1.37 1.28
7.26 / 6.53
15.69
29. 29.277.51_
71.23
ROME
00.52/74.22
38.71/33.42
47.31/41.5
28.89/ 20.93
45.05 / 39.06
24.44
29.78..n.
71.11
MEMIT
75.90/74.46
35.79/33.19
43.55 /41.99
23.11/ 19.89
40.96 / 38.81
26.39
29.77uxu1.
70.20
MEND
69.99 /64.71
24.10/29.22 29.23
45,36/45.00
31.75/ 29.33
44.05 / /4.77
24.17
28.50,
70.03
LoRA
88.05 84.62
14.77.67.79
85.39/
83.46/ 83.48
85.09 85.33
34,49
29 29,43,....
69.98
AdaLoRA
17..7/81.17
75.94/
92.53
88.68/ 69.94
97.17 76.53
27.67
29 19.677.11
70.35
RECT
15.79/71.35
38.19
50.91
/43.47
31.07 22,02
48.33 41.30
5,36
29.3 29.39
69.06
IKE (w/o ICL)
86.82 8
34.18/31.55 31,56
41.82/ /8. 58
25. 86 21,70
39.37
/6 35.78
94,60
IKE (w/ ICL.
67,42/655 65.87
40.12 39.36
43.03 40.53
40.30
37.96
91.08
UnKE
99.61 / 93.09
98.63 76.85
98.77 78.62
98.33/ / 70.66
98.73 77.70
42.49
29 18..6....
70.95
Baxed on Qwent.5-7B-Chen
32.43
73.49
MEMIT
74.72/ / 76.82
48.89 /48.71
48.50/48.18
34.59 31,50
47.55
/4.004
17.81
31 69.
71.99
UnKE
96.51 / /90.40
92.85 / 75.66
91.74//2.68
88.19/ / 60.59
91.40 / 77.44
40.08
32.033 4n.
72.61
Factual Correctness. In order to evaluate in more fine-grained manner whether the edited model has indeed understood the unstructured knowledge, we use FactScore Min et al. (2023) to evaluate the accuracy of the edited model in processing sub-questions and their corresponding answers. This metric is similar to the multi-hop accuracy in some structured knowledge editing benchmarks.
General Ability. To assess general ability, we evaluate both the MMLU score of the edited model and the Foc-FactScore for unrelated questions. First, we follow the methodology from MMLU (Hendryeks et al. (221I) to compute the average score across five MMLU samples for each unstructured sample. We then calculate the overall average score across all unstructured samples. Second, we assess the fact score for unrelated knowledge question-answer pairs, ensuring a comprehensive evaluation of the model's factual consistency.
summary, these four aspects form a robust framework for evaluating unstructured knowledge edits, ensuring both the fidelity and the flexibility of the generated content are thoroughly examined.
5.2 EXPERIMENTS ON UNSTRUCTURED KNOWLEDGE EDITING
We conduct a comprehensive evaluation of various baseline methods (Appendix D and our newly proposed UnKE method on the UnKEBench benchmark, including both automatic and human evaluation.
Automatic Evaluation. The specific results are presented in Table 2 Traditional fine-tuning methods, including FT-L and FT-A, have long exhibited significant limitations when tasked with structured knowledge UnKEBench is also underwhelming, with all evaluation metrics
editing. As anticipated, their performance falling short of those achieved by dedicated knowledge editing approaches. Methods employing a Locale- Then-Ead Edit paradigm, such as ROME, MEMIT and RECT, despite previously demonstrating satisfactory editing success rates on certain structured benchmarks, underperform on the UnKEBench dataset, particularly in terms of lexical and semantic similarity when compared to UnKE. LoRA and AdaLoRA exhibit strong performance, achieving the highest scores among all baseline models. The IKE method reveals several interesting findings. First, despite providing answers in advance and including examples, IKE underperforms
8

---

Published as a conference paper ICLR 2025
Table 3: Performance on human evaluation (a) and structured knowledge editing performance on KEBench (b). Ori-Acc and Para-Ace represent the accuracy for the original question and the paraphrased question, respectively. Sre-Ace and Tgt-Ace represent the irrelevant knowledge accuracy of subject and object in the triplet, respectively.
Method
Corr.
Simi.
Cohe.
Method
Ori-Acc
Para-Ace
Sre-Acc
Tgt-Ace
FT-A
1.06
1.47
1.47
FT-A
6.30
6,60
8.60
9.30
FT-L
1.17
1.00
1.31
FT-L
14.70
12.10
5.40
5.70
ROME
3.39
3.59
3.64
ROME
77.90
68.40
96.80
76.80
MEMIT
3.25
3.70
3.72
MEMIT
74.80
64.30
97.60
76.40
UnKE
4.78
4.72
4.70
UnKE
94.30
86.40
90.40
68.80
(a) Human Evaluations
(b) Structured Knowledge Editing
UnKE in most cases, except for the FactScore metric. Second, IKE without ICL achieves higher BERT scores than IKE with ICL, although it performs worse on other word-level metrics. Finally, we observe that IKE's FactScore in both settings is significantly higher than all other knowledge editing methods, including UnKE. This is expected, as IKE in this setting functions similarly to a reading comprehension task, where a model is given a text and asked questions about its content. Since LLMs are specifically trained for such tasks, they naturally achieve high scores. However, compared to knowledge editing methods that modify model parameters, we argue that this approach is akin to "cheating", as it does not truly edit the model's s internal knowledge. Finally, as shown in the table, all knowledge editing methods impact both the locality and generality of the model. We speculate that this occurs because the model must absorb the rich counterfactual information inherent unstructured knowledge, which may unintentionally influence related knowledge. Notably, UnKE, ROME, and MEMIT demonstrate strong performance in preserving locality. However, since UnKE extends the assumptions of the latter two by incorporating edits at both the layer and token levels, it modifies greater number of parameters. As result, its impact non-relevant knowledge may be slightly larger, though it remains within an acceptable range. For more examples of generated cases, please refer to the Appendix G
Human Evaluation. We conduct additional manual evaluation experiments to ensure the reliability of the evaluation metries and actual scores in UnKEBench. Due to the high cost of human evaluation, we randomly select 36 samples from pool of 1000 samples generated by each method. We employ three annotators, experienced in knowledge editing tasks but not involved in this project's training, to conduct a manual evaluation. They were instructed to assess the edited generated text across three dimensions: semantic correctness, similarity, and coherence on scale of 1-5, with I denoting "very low" and representing "very high". The scores are then averaged to derive the final human evaluation results. The evaluation results, presented in Table 3a reflect the collective assessments by the hired professionals. The inter-annotator agreement is 0.57 in Fleiss' K, which means moderate agreement. The experimental results provide strong evidence of the high consistency between the automatic evaluations and human evaluations. UnKE stands out the leader across all three dimensions. In contrast, the other baseline models frequently exhibit subpar performance in terms of semantic correctness, highlighting their limited ability to effectively edit unstructured knowledge. To further quantify the correlation between the automatic evaluation metrics and the human evaluation metrics, we calculated the Pearson correlation coefficient. Refer to Appendix [ for details.
5.3 EXPERIMENTS ON STRUCTURED KNOWLEDGE EDITING
To validate the capability of UnKE in editing knowledge triples, we conduct experiments on KEBench (Wei jet al. 2024c). The results presented in Table 3b demonstrate that UnKE surpasses strong baseline models in terms of Ori-Acc and Para-Ace metrics, exhibiting improvements of 16.4 points and 18 points, respectively. When comparing the results with UnKEBench, the improvement of UnKE over the strong baseline may not

---

Published as a conference paper ICLR 2025
Table 4: Ablation experiments. "Pres. Loss" and "Caus. Loss" denote Key Preservation Loss and Key Causal Loss, correspondingly. "w/ MLP" and "w/ ATTN" respectively specify that during optimization, only the parameters of the MLP and Attention modules in the transformer block are utilized.
Semantic Similarity
Lexical Similarity
FC.
General Ability
Method
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge-1.
Fact-Score
MMLU
UnKE
99.61 93.09
98 63 76.85
98.77 / 78.62
98.33/ / 70.66
98.73 /77.0
42.49
29.68 to
Modides
95.43 87 92,66 88881.666 62
92.34/71.3 71.32 90,58 / 63,46
94.78/ 73.39 91.16 / 70.03
92.91 / 68.51
93.23 / 72.65 90,21 / 771.15
37.98
29.770.011
W/ ATTN
89.73/ / 68.30
31.01
29.7la.0r
Loss Function
w/o
99.00
96,99 / 82.39
83.81
96.29/
97.21 / 83.20
38.74
29.44b.314
w/o Caus, Loss
21.19 26.27
26.69 / 31,79
10.29 / 13.46
24.93 / 29.68
46.50/58.
/ 58,91
16.77
29.520
w/o Caus, Loss
9.32/9.79
11.96 / 12,94
2.08
11.12/12.1 12.10
18.19
6.27
27.622.161
be pronounced. However, this outcome is anticipated since UnKE primarily targets complex and lengthy unstructured knowledge editing tasks, making it less conspicuous in simpler structured knowledge editing tasks. In general, experimental results have demonstrated that UnKE is not only effective in unstructured knowledge editing but can also be applied to structured knowledge. The UnKE results on the SnowEdit(Zhang et 2024a) benchmark are presented in Appendix
5.4 ABLATION EXPERIMENTS
To validate the efficacy of proposed approach, conduct ablation experiments non-local block key-value storage and cause-driven optimization. For non-local block key-value storage, we selectively optimized either the MLP Attention module. The outcomes indicate that optimizing solely the MLP or Attention module leads to partial performance decrease, reinforcing the premise of non-local knowledge storage outlined in Section 3.1 Specifically, optimizing just the MLP is insufficient for achieving optimal results; hence, a combination of non-local block key-value storage with both MLP and Attention modules is imperative for effectively representing information-rice unstructured knowledge.
In the context of causal-driven optimization, we conducted an ablation analysis on the loss function, specifi- cally omitting the Key Preservation Loss and Key Causal Loss individually. The findings reveal that excluding the Key Preservation Loss leads to a degradation in model performance, particularly affecting the MMLU metric. Conversely, eliminating the Key Causal Loss results in a significant decline in editing performance due to the absence of contextual information. However, given the presence of Key Preservation Loss, the general ability of the model remains relatively stable. Notably, when both losses are discarded, the model performance reaches its lowest point. In addition, we also verify that UnKE has robust batch editing and sequential editing capabilities (Appendix
6 CONCLUSIONS
We address the limitations of existing knowledge editing benchmarks, which primarily focus on structured knowledge triples, by introducing UnKEBench, the first benchmark for unstructured knowledge editing. To successfully edit unstructured knowledge, we propose UnKE, unstructured knowledge editing method, which incorporates non-local block key-value storage and cause-driven optimization, enabling effectively represent and edit unstructured knowledge with case. Experimental results UnKEBench demonstrate the superior performance of UnKE, significantly surpassing powerful bascline models on various evaluation metrics. Robustness analysis experiments confirm that UnKE possesses the ability to perform both batch and sequential editing. Additionally, UnKE also compares favorably with other strong baseline models on structured knowledge editing benchmarks.
10

---

Published as a conference paper ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the Strategic Priority Research Program of the CAS under Grants No.XDB0680302, the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.
REFERENCES
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Gabriele Bavota. Mining unstructured data in software repositories: Current and future trends. In Leaders of Tomorrow Symposium: Future of Software Engineering. FOSE@ SANER 2016, Osaka, Japan, March 14, 2016, pp. 1-12. IEEE Computer Society, 2016. doi: 10.1 10110099EEE.2011.77 URL httph::/doi.oooot /tt.//oo//oo//ii// 1109/ 2016.42
Canyu Chen and Kai Shu. Combating misinformation in the age of Ilms: Opportumities and challenges. CoRR. abs/2311.05ti 1.05656, 2023. doi: 10.48550/ARXII.331...55 URL https:/ /ttp://ooioooo//// htti::://ddi.rrg/0045555555 48550 2311 05656
Canyu Chen and Kai Shu. Can llm-generated misinformation be detected? The Twelfth International Conference Learning Representattonss ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? arXiv preprint 1rXiiv2310..80 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the
60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 8493-8502. Association for Computational Linguisties, 2022. doi: 18653/V 1/2022ACL-LLNG..55 5.. URL huttps://doi.ni 10. 18633 /0000/3//112222llllooll 581
Jingcheng Deng, Hengwei Dai, Xuewei Guo, Yuanchen Ju, and Wei Peng. IRRGN:
An implicit relational reasoning graph network for multi-turn response selection. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8529-8541, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v www.::22eemmlmmmmi.5ee..... main.584. URL[https:/ http://///acaanthloo..oroo 2022.emnlp- main 584/
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: lifelong model editing with discrete key-value adaptors. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, http:///:pp www.srsssiss..c... ccips.c//paprrrttcceeee paper paper_fies/ http:///ssssaoooooooooocoooaoadd

---

Published as a conference paper ICLR 2025
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in muusality-basedd localization vs. knowledge editing in language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information P Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL wwwp://e/pppppppppepeee paper ffee papppr/ppp///// 2203) /023 hash /90g httpaaalaaaaaa223353222222aaaaa0
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:
//openreview.nets net http:/iwwwwe//foommmio
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher:
One mistake worth neuron. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htttps://opeaaae https////ppeeeeewweee net net/pdf?
id
4o YUGGUGGGBPm
Omer Levy, Minjoon Sco, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via read- ing comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21 Conference on Computational Natural Language Learning (CoNLL 2017). Vancouver, Canada, August 3-4, 2017,
pp. 333-342. Association for Computational Linguistics, 2017. doi: 10.18653/V1/K17-1002 URL https:/ /doi:s////iior/////// 10. 100..118//11/K11-I10
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. PMET: precise model editing in a transformer. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirly-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, FAAI 2014, February 20-27, 2024, Vancouver, Canada, pp- 18572.
AAAI Press, 2024. doi:
00.1609/AAALLV33117..00
URL https:///oii.rg: 10. 10t.1609/aaaa aaai.aaiaaaaaaaaa htt.//aaai..1111111 www.17.281770
Chin-Yer Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81 81, 2004.
Jiateng Liu, Pengfei Yu. Yuji Zhang, Sha Li, Zixuan Zhang, and Heng Ji. EVEDIT: event-based knowledge editing with deductive editing boundaries. CoRR, abs/2402. 1324, 2024. doi: 10.48550/RXXVV24022. 11324. URL htttts://dooiorrgg 100.
11.e//111.445/////8 48550/ aav.:/a2vvv..2.113333
Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Editing personality for llms. arXiv preprint arXiv:2310.02 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual as- sociations GPT. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28- December 9, 2022, 2022. URL Abstract- Conterence, httpppscccpppppp.pppppp htppppppppppeeeee www.ei//essperrreeeee 2022 hash/
ct.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http::///ppnevviewt wtwpeenwweewweee///peee ntt//pdffiii MkbcAHIYgySS
12

---

Published as a conference paper at ICLR 2025
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer. Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12076-12100) Association for Computational Linguistics, 2023. doi: 1/226333 URL /ttp:::://ooooor////// 10.18658, v/203/3//12223mmmmm w00.230.mmmpmmmaaae..... main. 741
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale. In The Tenth International Conference on Learning Representations ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL | http://ppeneeeeeeee//eeeeee wdw.rrrummmnd--DD/eeeWiioCC
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory-based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), ), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 2022b. URL http:///ooooccco...oooa mlr. www.sresss/..rssseooo.mmme..ssee wt61//111////11l/1/// mttp/m/cttcccttatttmm
Yasumasa Onoe, Michael J. Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. Association for Computational Linguistics, 2023. doi: 10.18533V122233AC--L0OGG URL https://doi.or: 10. 18653 v 22/220223..0 w1123.acl--ae aon-long.3300
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. doi: 10.3115/1073083. 1073135. URL https:/ htt.htass:///caauuubooo //aca/aantoooogyorr..... 1040/
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 3749-3780. Association for Computational Linguistics, 2024. URL /dtios://oo..rr/ 10.
doi: 10.18653/V1/20222 1A2,5W/W122DNNAALLLLLOGGG
208
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann. Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor. Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CORR, abs/2307. .7.08888 2023. doi: 10.48550/ARXIV.2307.000
URL lhtpps
wtt..oo///1.4550//aaaaaa art.55500///.22252 23v .09288
Jaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, and Jiarong Xu. Cross-lingual knowledge editing in large language models. CoRR, abs/2309 009922, 2023a. doi: 10-48550/ARXXI1230900855 089522. URL Xit////iii..23000053333
13

---

Published as a conference paper ICLR 2025
Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen. Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Knowledge mechanisms in large language models: survey and perspective. CoRR, abs/2407.15017, 2024. doi: 10.48550/ARXIV.2400. 15017. URL httph:///oooorrrr 10. 48550/ 200.
Peng Wang, Ningyu Zhang. Xin Xie. Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269 2023b.
Weixuan Wang, Barry Haddow, and Alexandra Birch. Retrieval-augmentte multilingual knowledge editing. CoRR, abs/2312. 13040, 2023c, ar 2312.
doi: 1048850/ARXI.2311..30000 URL luttps://doi.org 10..g////1088550.
Yifan Wei, Xiaoyan Yu, Yixuan Weng. Huanhuan Ma, Yuanzhe Zhang, Jun Zhao, and Kang Liu. Does knowledge localization hold true? surprising differences between entity and relation perspectives in language models. arXiv preprint arXiv:2409.00617, 2024a.
Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Mlake: Multilingual knowledge editing benchmark for large language models, 2024b.
Zihao Wei, Liang Pang, Hanxing Ding, Jingcheng Deng, Huawei Shen, and Xueqi Cheng. Stable knowledge editing in large language models. CoRR, abs/2402. 13048, 2024c. doi: (0448550/ARXXX.242...133 URL https:/ /tt.://oooorrrr/ 10. 48.505 Xiv. 2402.13048
Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. Eva-kellm: new benchmark for evaluating knowledge editing of llms. CoRR, abs/2308. 2023. doi: (0-4550/ARXXI.2338.00993
URL https: /doi..rg//ggg///////oo 11.11100445550 48550/ Xiv. www.XXv.........M 23088.9954
Xiaobao Wu, Liangming Pan, William Yang Wang, and Anh Tuan Luu. AKEW: assessing knowledge editing in the wild. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 15118-15133, Association for Computational Linguistics, 2024. URL halps://aclanthology.ooow .org 22//222222.2ee/ www..eeenllmmiiin-...e main.843.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts, 2024.
Shicheng Xu, Danyang Hou, Liang Pang. Jingeheng Deng. Jun Xu, Huawei Shen, and Xueqi Cheng. Ai- generated images introduce invisible relevance bias to text-image retrieval. CoRR, abs/2311. 1.1408444 2023. doi: 10..4855//RRXIV.. 14084... URL https://toooi.ooo ww..ooog//0//..0.. wat.550//aXXxi..1.11411111
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Search-in-thhechain:: Interactively enhancing large language models with search for knowledge-intens intensive tasks. In Proceedings of the ACM on Web Conference 2024, pp. 1362-1373, 2024a.
Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng. Huawei Shen, Xueqi Cheng, and Jie Zhou. Unsuper- vised information refinement training of large language models for reurieeel-augmmented generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pp. 133-145, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: hwtpo//ww...../22222aalloogooooo
10.18653/v 2200///cl........ URL
Shicheng Xu, Liang Pang, Huawei Shen, and Xueqi Cheng- A theory for token-level harmonization in retrieval-augee -uggenttd generation. The Thirteenth International Conference on Learning Representations, 2025. URL https:/ https:///opnnceeeee pooonnciiww.ne/fouummdd------ee-
14

---

Published as a conference paper ICLR 2025
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 10222- Association for Computational Linguistics, 2023. doi: 10. 10.18653/V1/22233.MMM MAM.. 632. URL https://doi.orgg
Yunzhi Yao, Ningyu Zhang. Zekun Xi, Mengru Wang. Ziwen Xu, Shumin Deng, and Huajun Chen. Knowl- edge circuits in pretrained transformers. CoRR, abs/2405. 179669, 2024. doi: 10448500AARXX122255.1566
URL https://doi.org// 485500 arwia 24X.vv22405.17666
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. comprehensive study of knowledge editing for large language models. CoRR, abs/2401 00N/2401.0186.. 2024a. doi: 10.48550/
2.XIV.2401.01266. URL httpp:://doi.org/:/ 10.48550/ar S1wii.2401.0122.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.012880 2024b.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singupore, December 6-10, 2023, pp. 4862-4876. Association for Computational Linguistics, 2023. doi: 10. (665:/1/2023.EMN...MMM
296. URL hitlss///oi.orrg
10.18653/ v1 /tt////ooorrr/001/6633333
296
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 35686-15702 Association for URL http:::/ https::/doi..rrg/
Computational Linguistics, 2023. doi: 1.55//1/2022..MMI.PPMMM 18653 2115/3/v12022..2 -emnlp- main.971
Tongyao Zhu, Qian Liu, Liang Pang. Zhengbao Jiang, Min- Yen Kan, and Min Lin. Beyond memorization: The challenge of random memory access language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 3373- 3388. Association for Computational Linguistics, 2024. doi: https://doi.coo 10. 1
185. URL
15

---

Published as a conference paper ICLR 2025
A CONSTRUCTION OF UNKEBENCH
The unstructured texts are notably lengthy and contain knowledge that extends beyond simple knowledge triples linear fact chains. To effectively manage this complexity, we divide construction approach into four distinct phases.
1. We employ meticulously crafted instructions to guide ChatGPT in formulating the most appropriate question Q for each text A, thus creating an unstructured knowledge pair (Q, A).
2. To refine our evaluation mechanism, we use detailed instructions to prompt ChatGPT generate a paraphrased version of each original question, denoted as Qp. for every original question Q.
3. We leverage knowledge decomposition strategies and engage ChatGPT to produce multiple sub-question and sub-answer pairs (Q:, A$) for each (Q, A)-
4. Finally, we randomly select five questions from MMLU (Hendrycks et al. 2021 per example assess the generalization ability of the edited model. To evaluate locality, we extract entities from unstructured knowledge as topics and retrieve related triples from Wikipedia. We then construct unrelated knowledge question-answer pairs using a method similar to (Zhong et al. 2023).
Details and examples of constructing UnKEBench are provided in the Appendix C We introduce the differences between existing knowledge editing benchmarks and UnKEBench Appendix B
B RELATED WORK ON KNOWLEDGE EDITING BENCHMARKS
Previous knowledge editing datasets are composed in the form of triples or fact chains. The two prominent datasets are ZsRE (Levy et al. 2017) and COUNTERFACT (Meng et al. 2022). ZsRE utilizes back translation generate paraphrase questions, while COUNTERFACT focuses on constructing counterfactual data. The MQUAKE dataset (Zhong et al.) (202)), which serves as a multi-hop knowledge editing dataset, is utilized to assess the impact of knowledge editing on intricate knowledge chains. KEBench (Wei et al., 2024c) performs a comprehensive evaluation of the stability of different knowledge editing methods using a tree-structured dataset. Furthermore, (Zhang et al., 2024a) introduced KnowEdit, an integrated evaluation benchmark that incorporates popular knowledge editing datasets to comprehensively assess various knowledge editing technologies. Simultaneously, numerous efforts (Wei et al. 2024b) Wang et al., 2023a c) have been made construct multilingual datasets aiming to evaluate the generalizability of knowledge editing methods across diverse languages. Recent research on expanding knowledge triples has significantly broadened the application of knowledge editing methods, particularly in handling longer text. Eva-KELLLX (Wu et al.) (023) offers a benchmark dataset for evaluating document-level knowledge editing. However, this dataset creates documents by repeatedly expanding specific knowledge triples. Thus, Eva-KELLM predominantly focuses on editing specific counterfactual concepts, lacking the complexity of the unstructured knowledge editing tasks we aim to address. Similar to Eva- -ELLKELMM, (Wu et al. 2024) and (Rosati et al. 2024) introduced the AKEW and LEME benchmark for unstructured knowledge editing. However, they define it as editing entity concepts LLMs using related unstructured text, whereas we define it as directly editing unstructured text. These differences are also reflected in the evaluation metrics. KEP (Onoe et al.) 2023) introduces new entity definitions into language models through knowledge editing. However, focuses on a single entity, differing substantially from the complex and diverse unstructured knowledge editing tasks address. EVEDIT (Liu et al.) (024) constructs a multi-sentence knowledge dataset by generating and repeating knowledge triples. While these datasets are similar in length to proposed UnKEBench, they differ significantly construction. UnKEBench, as an unstructured knowledge editing benchmark, features longer texts, noise, and complex, comprehensive characteristics, spanning across domains.
16

---

Published as a conference paper ICLR 2025
C IMPLEMENTATION DETAILS OF CONSTRUCTING UNKEBENCH
After comprehensive pre-training on a large corpus, LLM has surpassed the performance of the previous
Encoder-only small model (Deng et al. 2022) and will form important parameter memory (Zhu et al.) (0024) to adapt to different downstream tasks (uu et al. 2025 2024a b). To ensure that these parameter memories do not inherently encompass editing objectives, we curate a dataset consisting of 1000 counterfactual unstructured texts. These texts are sourced from ConflictQA (Xie et al. 2024), benchmark specifically designed to distinguish between the parameter memory of the LLM and its counter-memory. This strategy is essential to prevent the model from merging the knowledge gained during pre-training with that obtained from editing tasks. Moreover, it addresses the critical challenge of discerning whether the model has learned target knowledge during the training phase or the editing process, thus maintaining clear demarcation between pre-training learning and editing objectives. Table @nd show the instructions for using ChatGPT (gpt-3.5-turbl to generate original and rephrased questions for unstructured text.
It is important to note that during the construction of UnKFBench, we conducted extensive manual checks to ensure the quality and accuracy of the data generation process.
We applied regular expression matching and manual verification to maintain data quality. Specifically, we removed samples containing prefixes such as "[Qq]uestion: (.+)", "[Pplaraphrase: (++)", or "[Tx]ext: (.+)".
Original and Paraphrased Question Generation:
Ensuring Q&A Pair Matching: To ensure proper alignment between questions (Q) and answers (A), we used the bge-large-en-V 11.5 model moddel to caleulate their matching scores. If the matching score fell below 0.8, we regenerated the question-ansee -nsweer pair using ChatGPT.
Ensuring Paraphrased Question Alignment: To verify that paraphrased questions accurately aligned with their original counterparts, we employed the bge-large-en-vi. model to compute their semantic similarity scores. If the score was below 0.9, we requested ChatGPT to regenerate the paraphrase.
When generating original questions, paraphrased questions, uub-questions, and sub-answers, we extracted entities and performed a simple check to ensure they were present in the unstructured text. If any entities were missing, we regenerated the corresponding content.
Entity Consistency Check:
Sub nub-Question and Sub-Answer Generation: Our goal was to evaluate detailed knowledge while maintaining conciseness. To achieve this, we restricted sub-answers to a maximum of 15 tokens. If a sub-au answer exceeded this limit, it was regenerated.
Ensuring Consistency in Sub-Questions and Sub-Answers. If the number of generated sub-questions and sub-answers did not match, we regenerated them to maintain consistency.
D BASELINE METHODS
We conduct experiments
autoregressive models, LLLMM-2-7B-Chaa. (ouvrro et al. 2023 and Bai et al. 2023). For baselines, we first compare the fine-tuning method F FT-L, which targets specific layers, with FT-A which fine-tunes all layers. Additionally, we assess two robust baseline models, ROME and MEMIT, focusing on their locating and editing capabilities. Lastly, we evaluate the hypernetwork beaeee model editing method MEND and the in-context learning-basee based editing method IKE (Zheng et al. 2023). Given the differences in data formats and evaluation metrics, we propose two experimental settings better suited for IKE adaptation UnKEBench after thoroughly reviewing its papers and source code: IKE with ICI. and IKE without ICI.
Owenl.5-7B--o
https:/ http:///hhgggnggaacc.../// AT BBAATbbbe-laggg-en--
https:/ http///ggggggggccoo/////// hwt.://eaalllaaaa Ilamaa haa.aelllaaaaaaaaaaaaaaaaaaa Mamaa 2- chatthi
https: http:////ggggggggg////////////// http://ingaaeeooo//eee//eee//.//
17

---

Published as conference paper ICLR 2025
System:
You are given a text asked come up with a question that best fits it.
User:
George Rankin has been actively involved in politics for over decade. He has served city council member for terms and was recently elected the state representative for his district. In addition, he has been a vocal advocate for varions political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and he is frequently quoted in local and national news outlets. It is clear that George Rankin's s occupation is that of political figure.
Assistant:
What is George Rankin's occupation?
Table 5: Demonstrating the application of ChatGPT (gpt-3.5-turbo) in generating question about unstruc- tured text.
System:
You are given question and asked to come up with semantically similar paraphrase question.
User:
What is George Rankin's occupation?
Assistant:
What does George Rankin do for living?
Table 6: Demonstrating the application of ChatGPT (gpt-3.5-turbo) in generating a paraphrased question from raw question.
ROBUSTNESS ANALYSIS ON BATCH EDITING AND SEQUENTIAL EDITING
To evaluate the robustness of UnKE in unstructured knowledge editing, we assess its batch editing capabilities (as shown in Table 7 and sequential editing performance (as presented in Figure using the UnKEbench dataset. In the batch editing assessment, we observe that as the batch size increases, the model's performance the original task remains relatively stable, indicating the robustness of UnKE's batch editing capabilities. However, there is a slight reduction in performance on paraphrased questions, which is expected. The simultancous optimization of larger number of keys marginally diminishes the model's generalization ability for paraphrased questions. For sequential editing, we find that the performance of all methods declines the number of edits increases. Nevertheless, UnKE exhibits the highest stability compared other baseline methods, demonstrating its robustness in sequential editing scenarios. These findings underscore the effectiveness of UnKE in handling both batch and sequential editing tasks, highlighting its potential as a promising approach for unstructured knowledge editing.
F PERFORMANCE OF UNKE ON KNOWEDIT
Compared with ROME and MEMIT, UnKE achieves the highest editing success rate on the WikiData recent, WikiBio, ConvSent, and Sanitation datasets, with particularly strong performance on the latter two. For the ZsRE and WikiData counterfact datasets, while UnKE's editing success rate is slightly lower than that of ROME and MEMIT, the difference remains minimal. Notably, we did not perform any parameter optimization for UnKE, further demonstrating its robustness.
18

---

Published as a conference paper ICLR 2025
Table 7: Comparison of different batch sizes. We conducted experiments on UnKE using the LLaMA2-7B- Chat model, with the decoding temperature set to 0.001.
Semantic Similarity
Lexical Similarity
FC.
General Ability
Batch Size
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge-L
Fact-Score
MMLU
20
99.61 93.09
98.63 76.85
98.77 / 78.62
98.33 70.66
98.73 77.70
42.49
29.68. 14
21
99.41 91.55
98.72 98.72/73.03
98.88 74.85
98.44 65.35
98.83 73.72
42.35
29.666.121
2²
99.48 89.98
98.97 69.99
98.98 /7183
98.61 60.93
41.82
19.
23
99.57 / 88.33
98.99/66.
99.08 / 67.98
98.76 / 56.16
99.05 66.61
42.24
29.66p.12]
2
99.70/ 85,98
62.30
99.28 / 64.76
99.01 / 51.97
99.25 63.22
42.13
29.61o.17]
2
99.56 / 84.07
99.12/ 59.36
99.16/ 16 61,45
98.88
447.57
99.14/ 59.71
41.38
29.736.051
2
99.78/ / 85.21
99.47 60.38
99.50 / 62.25
99.31 / 48.55
99.48 60.50
42,93
29.72.
Bert Score
BLEU Score
Rouge-1 Score
1.0
1.0
3.2
0.-
0.6
0.c
0.4
0.4
0.4
0.-
20
30
Number of Edits
Number of Edits
Number of Edits
Rouge-2 Score
Rouge-L Score
Fact Score
1.0
1.0
0.32
0.5
0.5
0:23
0.0
Score:
6:23
3.4
6.19
02-
02-
0.05
0.0
10
30
50
10
25
30
30
60
59
Number of Edits
Number of Edits ROME
Number of Edits
FT-A
FT-L
MEMIT
UnKE
Figure 3: Performance in sequential editing. We select the first 64 samples in the UnKEBench data set for sequential editing experiments.
Overall, UnKE not only delivers tfate-of-the-asd results on the unstructured knowledge editing benchmark UnKEBench, but also performs competitively on the structured knowledge editing benchmark KnowEdit.
G CASE ANALYSIS OF ROME, MEMIT AND UNKE
Table 16 shows the generation cases of three different methods: ROME, MEMIT and UnKE. The methods of editing local key-value pairs, namely ROME and MEMIT, limit capabilities when it comes to complex unstructured knowledge editing tasks. These methods only remember small set of editing goals and are unable to fully retell the editing objectives. In contrast, UnKE exhibits greater proficiency in handling such tasks and is capable of conveying the editing goals.
19

---

Published as conference paper ICLR 2025
DataSet
Metric
SERAC
ICE
AdaLoRA
MEND
FT-L
FT-M
ROME
MEMIT
UnKE
VikiData_recemt
Edit Succ.
98.68
60.74
100.00
95.75
55.75
100.00
97.18
97.05
98.10 10
Portahility
63.52
36.93
64.69
55.88
40.86
65.44
55.25
56.37
61.49
Locality 100.00
33.34
56.42
94.76
43.70
64.33
54.77
52.15
79.07
Fluency
553.19
531.01
579.57
557.11
529.24
574.32
579.66
573.89
586.22
ZsRE
Edit Succ. 99.67
66.01
100.00
96.74
53.93
99.98
96.77
95.37
95.16
Portability 56.48
63,94
58.03
60.41
45.64
60.31
52.63
$2.67
54.53
Locality 30.23
23.14
75.76
92.79
73.42
89.78
53.67
48.32
76.14
Fluency
410.89
541.14
563.56
524.33
493.01
552.26
573.75
563.31
$72.94
WikiBio
Edit Succ.
99.69
95.53
100,00
93.66
66.33
100.00
96.08
94,40
98.23
Locality
69.79
47,90
81.28
79.86
62.74
61.51
86.39
Fluency
606.95
632.92
618.45
609.39
606.95
612.69
617.69
616.65
615.93
WikiData_counterri
Edit Succ. 99.99
69.83
100.00
80.03
45.15
100.00
98.57
98.05
97.85
Portability
76.07
45.32
69.89
52.01
33.60
74.36
55.92
58.56
73.22
Locality
98.96
32.38
70.31
94.38
50.48
76.76
51.97
46.62
41.75
Fluency
549.91
547
580.29
555.72
528.26
575.62
584.04
575.96
$75.16
ConvSent
Edit Succ.
62.75
52.78
44.89
50.76
49.50
46.10
45.79
44.75
60.41
Locality
0.26
49.73
0.18
3.42
0.00
0.00
0.00
0.00
0.00
Fluency
458.21
621.45
606.42
379.43
607.86
$92.52
606.32
602.62
604.62
Sanitation
0.00
72.50
2.50
0.00
85,00
48.75
98.79
Locality
100.00
56.58
65.50
5.29
14.78
47.07
50.31
67.47
70.13
Fluency
416.29
794.15
330.44
407.18
439.10
416.29
465.12
466.10
460.71
Table 8: Experimental Results KnowEdit.
H IMPACT OF HYPERPARAMETER L ON MODEL PERFORMANCE
In Section 3.1 we hypothesize that the shallow layer of LLM encodes the key vector of knowledge, dividing LLM into key generator and value generator based on the hyperparameter To investigate the influence of the value of L model performance, conducted analytical experiments, presented in Table
The results indicate that when the value stable, effectively managing unstructured knowledge. However, L increases further, it becomes apparent that the model's effectiveness diminishes. This shows that at this time, is too deep, resulting in the key vector that has stored the target information, it is difficult to edit.
small, such as 5-10, the model performance remains relatively
20

---

Published as conference paper ICLR 2025
Table 9: Optimization layer selection experiment.
Semantic Similarity
Lexical Similarity
FC
General Ability
L
Bert-Score
BLEU
Rouge-1
Rouge-2
Rouge-L
Fact-Score
MMLU
98.30 90.15
95.02 70.30
94.81
70.86
92.98 60.28
94.59 69.57
40.37
29.56, 22.
99.23 91.51
97.24 72.29
97.54 73.72
96.55 64.28
97.45 72.60
42.11
29.72g.06.
99.61 93.09
78.63/76.85
78.62 98.77
98.33 70.66
98.73/77.70
42.49
29. 29.68g.10 10.
99,62 94.29
98.57 80.91
98.79/83.02
98.24/ 76.23
98.13/82.13
40.86
29 29.68g.10. 10_
98.09 93.17
92.71 74.39
94.35 79.47
91.99 70.99
94.05/
41.33
29.70 29.70
10
95.53 91.82
81.50/ 67.10
86.14/74.09
80.39 63.46
85.42/72.66
39.72
29.71
11
90.04 86.73
68.96 56,72
76.01 / 67.03
66.68 / 53.70
74.85 65.28
27.33
29.64,.14
12
87.47 84.90
53.17 44.92
65.17 / 59.16
92.43/43.74
63.57 57.22
22.78
29.67a.11.
Table 10: Correlation between human and automatic evaluation metries.
Metries
BLEU
Rouge-1
Rouge-2
Rouge-L
Bert-Score
FactScore
Correctness
98.55%.0021
95.04o.m 0132
95.740.0100
98.10p. 10.0000
97.52o.00s
96.90o 0065
Similarity
94.74p.0144
91.54o.02m
91.230.0300
94.50p.0153
97.670.0042
92. 92.18n.0260
Coherence
95.64p.0108
92.370.0250
91.90p.0273
95.41p.0117
98.68p.0018
94.04n.0173
I CORRELATION BETWEEN AUTOMATIC AND HUMAN EVALUATION METRICS
Tables I0 and Table I1 display the Pearson correlation coefficients between the human evaluation metrics and the original question metrics and the paraphrase question metrics, respectively. Due significant differences in the evaluation dimensions of the general ability metric MMLU and the three human evaluation metrics, it is omitted from the table.
Each cell in the table represents the correlation coefficient between the corresponding automatic evaluation metric and the human evaluation metric, with the subscript indicating the p-value. Almost all correlation coefficients are above 0.95, confirming strong correlation between the human and automated assessment results. Additionally, the p-values for all metrics are below 0.05, indicating that the correlations are statistically significant.
J EXPERIMENT DETAILS
Except for UnKE, use EasyEdit Wang et al. 2023b implement all other editing methods, including fine-tuning. For all other baselines, except for the necessary modifications that need to be applied to UnKEBench, we use the official default hyperparameters, which can be easily reproduced in the official library. The optimizer type used when it comes to gradient descent is Adam. The following are their important hyperparameter configuration contents.
Fine-tuning For FT-L and FT-A, with the only distinction being the number of layers involved in parameter updates. The maximum length is set to 1024, and a learning rate of x 10-1 is utilized. Each sample updated layer which is consistent
undergoes 25 optimization steps. The layer where FT-L parameters with UnKE, For LoRA and AdaLoRA, we set the number of epochs to 25 to ensure model convergence. We perform hyperparameter tuning for the learning rate, rank, and alpha. Initially, we used the default learning rate of 5e-3 provided by the EasyEdit library. However, we observed unstable loss behavior, where the loss
hutp:://giitubb.oo///ju////
21

---

Published as a conference paper ICLR 2025
Table 11: Correlation between human and automatic evaluation metrics (Para.).
Metrics
BLEU
Rouge-1
Rouge-2
Rouge-L
Bert-Score
Correctness
98.68p.001s
92.38p.0249
95.68a.m 0107
97.830.003s
97.660.0043
Similarity 94.95
89. 03.023
91.13a.031s 0313
91.50
97. 97.95p.0035
Coherence 95.85p.0101
89.88p.0350
91.82g.oz 0277
95.38p.0119
98.87p.0014
initially decreased, then spiked suddenly, and then declined again. To achieve stable loss convergence, we gradually reduced the learning a fair comparison with UnKE, we ultimately set the learning rate to 2c-4. The EasyEdit library defaults to rank 8 and alpha 32. Based on fine-tuning experience, we observed that maintaining the ratio alpha/rank = 2 typically yields better results. Therefore, we adjusted alpha to 16, aligning with this heuristic approach. After tuning, we determined that rank 8, alpha = 16 achieves the best performance.
and found that 5e-4 resulted in normal convergence. For consistency and
ROME and MEMIT The primary distinction be- tween ROME and MEMIT lies in the number of editing layers. ROME focuses editing the layer 7, while MEMIT targets the layers [4,5,6,7,8]. Both ap- proaches undergo 25 optimization steps, utilizing learning rate of 5 weight attenuation coefficient of 10-3 and KL factor of 0.0625. Before the editing process, approximately 100,000 Wikipedia samples need be computed. which is a highly tome-consuming task.
Table 12: Comparison of running time of each method.
Time is in hours.
Method
Time
Method
Time
FT-L
14
ROME
21
FT-A
21
MEMIT
27.75
MEND
38
UnKE
10.5
RECT The RECT model is similar to ROME, and we implemented using the official source code library. We conducted hyperparameter tuning on the sparsity parameter and found that sparsity level of 80
MEND MEND enables concurrent edits by accumulating gradients from all edit examples and passing them through the hypernetwork simultaneously. It calculates parameter layers 29, 30, and 31 and utilizes learning rate of x 10-4, Due to the presence of numerous hyperparameters, it is advisable to refer to the official website or code library for detailed information.
IKE IKE is a knowledge editing method based on in-context learning (ICL) that does not require modifying model parameters. In IKE (Without In-Context Learning), we directly use the answer in the prompt as the context, followed by the original question, interpretation question, or sub-question The input format is For IKE (Context-Baan Based Learning), we first employ a dense retriever identify the five
shown Table 13
Context: (answer)
Question: (original or paraphase.question or sub.question}
Answer:
Table 13: Input template for IKE (Without In-Context Learning).
most relevant samples of unstructured knowledge (limited by context length). These samples, along with their corresponding questions and answers, included instances in the prompt. Finally, append the answers
22

---

Published as conference paper ICLR 2025
Percentage Distribution Entity Counts UnKEBench
36
30
Percentage 1%6
25
20
10
0.5
6-10
16-20
21-25
26-30
>30
Entity Count
Figure 4: The X-axis represents the number of entities contained in unstructured text, while the Y-axis indicates proportion of sentences containing that number of entities among sentences.
as context and sequentially ask the original question, explanation questions, and sub-questions. The input entirety, ensuring consistency
format is shown in Table [4] The dense retriever utilizes MiniLM-L6-v2 in its with original IKE. Since IKE does not modify model parameters, we employ the vLLM library for efficient inference after data processing.
Context: [case_answerr
Question: fcase orwginaalqqustiooooooooooooooooi case. case. paraphase.questeee case.sub question)
Answer: [case_and answer or case.sub.answer
Context: {answer)
Question:
or paraphase.question sub sub.question]
Answer:
Table 14: Input template for IKE (With In-Context Learning).
UnKE UnKE employs a two-stage structuring process. In the first stage, the learning rate set to 5e-1, with 25 optimization steps and weight attenuation coefficient of le-3. In the second stage, the learning rate is set to 2e-4, and 50 optimization steps are performed. All experiments conducted on UnKE in this article focus on optimizing layer 7. During each optimization iteration, additional 20 samples are randomly selected from the alpaca instruction fine-tuning data less than what is required by ROME and MEMIT.
It is important to note that this number is significantly
"https://githuu.be labsu-lab/ stanford staaoodd_llapcaaa
23

---

Published as conference paper ICLR 2025
System:
You are asked generate some short question-answer pairs based on the specified text. These question-answer pairs mainly ask questions about the knowledge entities in the text, and the answers should be the knowledge entities being asked.
User:
George Rankin has been actively involved in politics for over a decade. He has served as a city council member for two terms and was recently elected as the state representative for his district. In addition, he has been vocal advocate for various political causes, including environmental protection and social justice. His speeches and interviews often focus on political issues and is frequently quoted local and national news outlets. clear that George Rankin's 's occupation is that political figure.
Assistant:
Question: How long has George Rankin been involved in politics?
Answer: Over decade.
Question: What political positions has George Rankin held?
Answer: City council member and state representative.
Question: What causes has George Rankin advocated for?
Answer: Environmental protection and social justice.
Question: What is George Rankin's occupation?
Answer: A political figure.
Table 15:
Demonstrating the application of ChatGPT (gpt-3.5-turbo) in generating some short question- answer pairs based on the unstructured text.
Time costs We present the runtime of each knowledge editing method on UnKEBench. It is important that our evaluation approach differs from the EasyEdit framework. Instead. we edit a sample and subsequently allow the model to generate the corresponding problem results for evaluation. This process may take slightly longer. Additionally, ROME and MÉMIT require additional time for calculating Wikipedia knowledge. MEND adds the time to calculate the one-stage model. Please refer to the Table [2 for detailed results. All experiments can be completed on an A800-80G graphics card.
K THE BASIC STATISTICS OF UNKEBENCH
As shown Figure only 10% of the sentences contain 0-5 entities, which is common range for structured text editing tasks. In contrast, over 90% of the sentences contain more than 5 entities, highlighting the inherent complexity of unstructured text.
L BROADER IMPACTS
UnKE offers the advantage of timely updating outdated knowledge in LLMs, thereby reducing costs for developers and eliminating the need for issturce-nttesivve pre-training fine-tuning steps. However, it is important to acknowledge the potential misuse of UnKE by malicious actors who may attempt to inject harmful information into LLMs. To mitigate this risk, knowledge detection system can be employed identify and prevent the inclusion malicious content.
24

---

Published as a conference paper ICLR 2025
Question : What were Charles Strachey's main contributions to British politics and law during the 19th century?
Charles Strachey, 4th Baron O'Hagan was prominent political figure in the United Kingdom during the 19th century. He served as a Member of Parliament for the Liberal Party from 1868 to 1874, representing the constituency of Nottingham. During his time in Parliament, he was known for his advocacy of social reform and his support for the rights of workers. In addition to his political career, Strachey was also respected lawyer and judge, serving as a Queen's Counsel and Justice the Peace. His contributions to British politics and law have been widely recognized, and he remains an important figure in the country's history.
Text:
UnKE Prediction Charles Strachey, 4th Baron O'Hagan was prominent political fig- ure in the United Kingdom during the 19th century. He served as Member of Parliament for the Liberal Party from 1868 to 1874, representing the constituency of Nottingham. During his time in Parliament, he was known for his advocacy of social reform and his support for the rights of workers. In addition to his political career, Strachey was also respected lawyer and judge, serving Queen's Counsel and Justice of the Peace. His contributions British politics and law have been widely recognized, and he remains an important figure in the country's history.
ROME Prediction: : Charles Strachey was prominent British politician and lawyer who made significant contributions to the country's political and legal landscape during the 19th century. Here some of his key contributions:
Parliamentary Reform: Strachey was a strong advocate for parliamentary reform and played a key role the passage the Reform Act of 1832. This act expanded the franchise and redrew electoral boundaries, giving more people the right to vole and increasing the representation of the working class in Parlianent. 2. Legal Reform: Strachey
also leading figure in the legal profession and instrumental in the passage of Law Reform Act of 1832. This act aimed to modernize the law and reduce complexity of legal proceedings, making it easier for
MEMIT Prediction: Charles Strachey (1796-1867) was a prominent British lawyer and politician who made significant contributions British politics and law during the 19th century. Here are some of his main contributions: 1. Legal Reforms: Strachey was a strong advocate for legal reform and was instrumental in the passage of the Law Amendment Act of 1870, which established the office of Law Society Council and paved the way for the modernization of the legal profession. He also played key role in the establishment of the Incorporated Law Society of Scotland. Parliamentary Reforms: Strachey was member of the House Commons and served as a vocal advocate for parliamentary reform.
Table 16: This table presents the problems and their corresponding unstructured text after editing. can be observed that the predicted text generated by UnKE is almost identical to the original text. However, the texts generated by methods like ROME and MEMIT only edit a few key knowledge points such as 'political' and "parliamentary reform', while the detailed deseriptions of these knowledge points are almost entirely incorrect.
25


Reviews:
[{"review_id": "bDD3gaddS5", "reviewer": "Reviewer_3a98", "summary": "This paper introduces a method called the unstructured knowledge editing method (UnKE).\nTraditional knowledge editing methods are mainly focused on structured data (triples with subject, relations, and objects).\nHowever, this paper argues that a vast amount of knowledge is unstructured, often complex, and presented as free-form text.\nUnKE uses non-local block key-value storage and cause-driven optimization, effectively capturing and editing unstructured knowledge without sacrificing context. This paper also proposes a new dataset UnKEBench. It consists of paraphrased questions and counterfactual unstructured texts for knowledge editing.", "questions": "1. Maybe give an example of the proposed unstructured knowledge editing setting in the manuscript.\n2. The introduced UnKEBench only uses counterfactual samples. How is the performance on real-world unstructured knowledge?\n4. MEMIT and ROME mainly aim for structured knowledge. How do you use them for the unstructured knowledge in UnKEBench?\n5. Why is the performance of FT-A so low?\n6. How does IKE (with in-context learning) perform on unstructured knowledge editing?\n3. Line 873, FT-ALL --> FT-A.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed methods (non-local key-value storage and cause-driven optimization) are interesting as they can handle unstructured knowledge editing. They are quite different from previous methods.\n2. The reported experiments are extensive. They show UnKE outperforms baseline models (like MEMIT and ROME) across multiple evaluation metrics, validating its effectiveness in handling unstructured knowledge editing.\n3. The paper proposes a new benchmark for unstructured knowledge editing.", "weaknesses": "1. The motivation is quite similar to [1]. The differences should be discussed.\n2. The constructed dataset UnkBench only uses counterfactual unstructured texts from ConflictQA. I want to see how UnKE performs on real-world unstructured knowledge as in [1].\n3. The paper makes me feel confused about the input of the expected output of unstructured knowledge editing. It seems the input should be a question and the expected output should be a text.\n4. The paper lacks some details about how to use MEMIT, ROME, and MEND for unstructured knowledge editing as they mainly aim for structured knowledge.\n\n\n[1] Updating Language Models with Unstructured Facts: Towards Practical Knowledge Editing (https://arxiv.org/abs/2402.18909)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gwZZocA2lC", "reviewer": "Reviewer_L1FS", "summary": "This paper introduces UnKE, an approach that addresses limitations in existing knowledge editing methods that primarily handle structured knowledge. The proposed UnKE method expands upon current techniques through \"non-local block key-value storage\" and \"cause-driven optimization\" to enhance the handling of complex, unstructured knowledge. Experimental results on a new benchmark, UnKEBench, demonstrate that UnKE outperforms baseline models in unstructured and structured knowledge editing, achieving high accuracy and robustness in sequential and batch editing.", "questions": "refer to the weaknesses", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe introduction of non-local block key-value storage and cause-driven optimization effectively improves the representation and editing of unstructured knowledge, making the method more adaptable than traditional term-driven optimization approaches.\n2.\tIt creates a new benchmark UnKEBench for unstructured knowledge editing.\n3.\tIn experiments, the proposed UnKE outperforms baselines across lexical, semantic, and factual correctness metrics, with robust transferability across various LLM architectures and effective batch and sequential editing capabilities.", "weaknesses": "1.\tThe paper claims that existing knowledge editing methods have primarily focused on modifying structured knowledge in large language models, but it is not correct. Fine-tuning based methods do not rely on structured knowledge, such as LoRA, RoseLoRA [1], and RECT [2]. In experiments, the paper should set the three methods as baselines as well.\n2.\tThe evaluation metric of general ability is not reasonable. Simply evaluating the performance of edited models on MMLU can not show the general ability correctly. In Table 2, almost all the methods have the same performance on MMLU. If you want to evaluate the locality of edited models, you need to collect extra data which is close to the edited knowledge.\n3.\tThe proposed method relies on the choice of layers to edit. From Table 7, if we choose different layers, the performance will vary significantly. It is difficult to apply to real-world applications.\n\n[1] Wang H, Liu T, Zhao T, et al. RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning[J]. arXiv preprint arXiv:2406.10777, 2024.\n[2] Gu J C, Xu H X, Ma J Y, et al. Model editing can hurt general abilities of large language models[J]. arXiv preprint arXiv:2401.04700, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bnEA0hQ5X", "reviewer": "Reviewer_SoQU", "summary": "This paper presents a novel approach to knowledge editing in large language models (LLMs), specifically targeting unstructured knowledge. The authors argue that current knowledge editing methods are limited in their ability to handle the complex and free-from information found in unstructured formats, which constitute a significant portion of real-world knowledge.\n\nThe key contributions of this paper are as follows:\n\n1. **Unstructured Knowledge Editing (UnKE) Method**: The authors propose UnKE, a method that extends knowledge editing from structured knowledge to unstructured knowledge. This is a advancement as it addresses the gap in handling the majority of real-world knowledge that is not neatly packaged in structured formats.\n\n2. **Cause-Driven Optimization**: The paper replaces term-driven optimization with a cause-driven approach, which edits the last token directly while preserving context. This avoids the need to locate specific terms and prevents the loss of contextual information, which is a common issue with term-driven methods.\n\n3. **UnKEBench Dataset**: To evaluate the performance of UnKE, the authors develop a new benchmark dataset, UnKEBench, which is designed to be more challenging than existing structured editing benchmarks due to the complexity of unstructured knowledge.\n\n4. **Experimental Results**: The paper demonstrates that UnKE outperforms baselines on the newly proposed UnKEBench dataset, as well as on KEBench. \n\n5. **Robustness Analysis**: The authors conduct robustness analysis experiments, confirming UnKE's ability to perform batch and sequential editing, and its superiority over baseline models in structured knowledge editing.\n\nIn summary, this paper makes a compelling case for the need to extend knowledge editing to unstructured data and provides a robust method and benchmark to facilitate this advancement. The proposed UnKE method shows promising results in accurately handling free-format unstructured knowledge, which is a good step forward for the field of knowledge editing in LLMs.", "questions": "- Why does unstructured knowledge contain more information? Intuitively, unstructured knowledge has more grammatical and structural words, which carry low information.\n- What is the relationship between LEME and UnKEBench? If LEME and UnKEBench have similar contributions, the statement \"the lack of a benchmark for editing unstructured knowledge\" in the paper would not hold.\n- In \"EXPERIMENTS ON STRUCTURED KNOWLEDGE EDITING,\" why choose KEBench instead of knownedit? How does UnKE perform on knownedit?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:**\n- The paper introduces an approach to editing unstructured knowledge within large language models (LLMs), which is a novel contribution to the field.\n- The proposal of the Unstructured Knowledge Editing (UnKE) method breaks new ground by extending the scope of knowledge editing beyond traditional structured knowledge.\n- The concept of non-local block key-value storage and cause-driven optimization is innovative and addresses a previously unmet need for handling complex, unstructured information.\n- The authors have developed a new benchmark dataset, UnKEBench, which provides a way to assess unstructured knowledge editing capabilities.\n\n**Clarity:**\n- The paper is well-organized and clearly structured, making it easy for readers to follow the progression of ideas and the logical flow of arguments.\n\n**Significance:**\n- The successful editing of unstructured knowledge has broad implications for various applications, including but not limited to information retrieval, question answering, and conversational agents.\n- By demonstrating UnKE's effectiveness on both newly proposed and traditional datasets, the paper highlights the potential of the method to impact real-world applications where unstructured knowledge is prevalent.\n\nOverall, the paper is a significant contribution to the field of knowledge editing, particularly in the area of knowledge editing within LLMs.", "weaknesses": "- Lack of baselines: Although UnKE has demonstrated quite good results, the baselines used for comparison are not the most advanced methods in the field. It would be more convincing if UnKE could be compared with state-of-the-art methods such as WISE, GRACE, and FT-M, and show better results. In particular, UnKE should against with non-term-driven optimization method GRACE for a more fairly comparison.\n- Contributions of UnKEBench: First, before UnKEBench, there have already been works on long-form knowledge editing (e.g., LEME[1]), but this is not discussed in the paper. Second, UnKEBench is generated by ChatGPT and lacks steps to ensure data quality, such as human verification.\n- Description inconsistent with reality: \n    - The paper hypothesizes that unstructured knowledge is distributed across multiple layers of the language model, but the experiment only edits one layer.\n    - The paper claims that UnKE's cause-driven optimization can preserve pre-trained knowledge during the editing process. However, in the experiments, UnKE's ability to retain original knowledge is worse than the baseline (i.e., MMLU in Table 2 and Src-Acc and Tgt-Acc in Table 3).\n\n[1] Rosati D, Gonzales R, Chen J, et al. Long-form evaluation of model editing[J]. arXiv preprint arXiv:2402.09394, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "puBvD5MUCZ", "reviewer": "Reviewer_6Swj", "summary": "Based on the analysis of knowledge editing methods for LLMs, this article proposes two hypothesis for unstructured knowledge editing, and a method named Unstructured Knowledge Editing(UnKE)  ,which incorporates non-local block key-value storage and cause-driven optimization, enabling it to effectively represent and edit unstructured knowledge with ease. In order to address the limitations of existing knowledge editing benchmarks, which primarily focus on structured knowledge triples, the paper introduces UnKEBench, and  experimental results on UnKEBench demonstrate the superior performance of UnKE, significantly surpassing powerful baseline models on various evaluation metrics.", "questions": "The paper expressed is very clear.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros\n   1. proposed an Unstructured Knowledge Editing(UnKE)  method;\n   2. constructed UnKEBench", "weaknesses": "The article is fluent in writing and has a complete structure, but some expressions need improvement，for example, in the abstract, there is \"unstructure knoledge\",etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cyIzyhLycz", "reviewer": "Reviewer_Dz84", "summary": "This paper addresses the limitations of current knowledge editing methods in large language models, which primarily focus on structured knowledge. The paper introduces a new method called Unstructured Knowledge Editing (UnKE). A novel dataset for unstructured knowledge editing, UnKEBench, is also proposed.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper introduces Unstructured Knowledge Editing (UnKE), which addresses the limitations of existing methods by focusing on unstructured knowledge.\n\n2.\tThe use of a newly proposed dataset (UnKEBench) with traditional datasets provides a strong evaluation.\n\n3.\tUnKE extends the method in both layer and token dimensions. It enhances the capacity for knowledge editing.\n\n4.\tThe paper is generally well-written and easy to follow.", "weaknesses": "1.\tThe proposed method may be complicated, potentially posing challenges for real-world applications. The authors are suggested to give more details on computational resources and complexity analysis.\n\n2.\tThe authors used LLAMA2 and QWEN1.5 models for evaluation. Since newer versions of these LLMs have been released, it is better to show the performance of UnKE on newer LLMs.\n\n3.\tThe authors trimmed too much space. For example, check Figure 2 and Table 1.\n\n4.\tWhile a new benchmark is introduced, the paper might lack the detailed process of data processing and quality control steps, possibly limiting the quality of the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)