RuleEdit: Towards Rule-Level Knowledge Generalization to Mitigate Over-Editing in Large Language Models
Authors: Bihan Zhou, HaoPeng Ren, li yuan, Yi Cai, Liuwen Cao, Zikun Deng
Accepted at: acl25

RuleEdit: Towards Rule-Level Knowledge Generalization to Mitigate Over-Editing in Large Language Models

Bihan Zhou1,2*, Haopeng Ren3*, Li Yuan1,2*, Yi Cai2,1†, Liuwen Cao1,2, Zikun Deng1,2
1School of Software Engineering, South China University of Technology, Guangzhou, China
2Key Laboratory of Big Data and Intelligent Robot(SCUT), MOE of China
3School of Nursing, The Hong Kong Polytechnic University, Hong Kong, China
{201930176418, seyunli}@mail.scut.edu.cn, renhp_scut@foxmail.com {ycai, zkdeng}@scut.edu.cn, caoliuwencc@163.com

Abstract

Knowledge editing emerges as a promising approach for updating target knowledge in Large Language Models (LLMs) in a timely manner, thereby preventing undesirable behaviors stemming from outdated, inaccurate, or incomplete knowledge. However, existing methods mainly focus on instance-level editing, which is prone to over-editing risk featuring knowledge degradation and general ability deterioration, due to redundant instance-specific modifications for knowledge. To mitigate the over-editing risk, we explore the rule-level editing problem that avoids case-by-case modification by generalizing rule-level knowledge to update rule-derived instances. We further construct a benchmark called RuleEdit for systematic evaluation on rule-level editing. Moreover, we propose a Rule-Transfer Editing (RTE) method to facilitate effective updates and generalizations of rule-level knowledge in LLMs. Experimental results highlight our significant improvements, with the enhancements of 28.1% in portability and 8.1% in average performance over the best-performing baselines for LLaMA-2-7B on \( RULE_{mix} \).

1 Introduction

Large Language Models (LLMs) have demonstrated remarkable intelligence in performing Natural Language Processing (NLP) tasks (Chang et al., 2024). As the world evolves dynamically, outdated, incorrect, or missing knowledge in LLMs may lead to impaired performance in NLP tasks (Zhang et al., 2024b). To address this limitation, Sinitisin et al. (2020) introduces Knowledge Editing to enable timely update for the target knowledge in LLMs, which has garnered widespread interest.

Existing knowledge editing methods (Meng et al., 2022; Hartvigsen et al., 2023; Mitchell et al., 2022a) for LLMs primarily focus on instance-level editing (Wang et al., 2024b), which involves modifying specific and detailed information (i.e., characteristics, attributes.) of individual instances or cases. However, as illustrated in Figure 1(a), numerous specific instances (e.g., "Premise: If someone’s Breath Alcohol Concentration is 0.56% while driving, what is the penalty? → Conclusion: 6-month license suspension and a $1000 fine.") can be derived from the general rule (e.g., "Premise: If driver’s Breath Alcohol Concentration is between 0.20% and 0.80%, what is the penalty? → Conclusion: 6-month license suspension and a $1000 fine."). It is redundant to modify case by case in instance-level editing. With inefficient large-scale updates to rule-derived instances, instance-level editing is vulnerable to over-editing risk (Zheng et al., 2023). Specifically, as indicated in Figure 2(a), with increasing editing steps in instance-level editing, LLMs tend to suffer from significant performance deterioration in both knowledge updates (success rate drops from 93.33% to 6.44%) and general tasks (reasoning accuracy drops from

![Illustration of instance-level editing with case-by-case modification. (b) Illustration of rule-level editing with a generalized editing process.](page_1012_370_377_246.png)

Figure 1: (a) Illustration of instance-level editing with case-by-case modification. (b) Illustration of rule-level editing with a generalized editing process.
97.65% to 0.00%).

To mitigate the above over-editing risk arising from redundant modifications to rule-derived instances in instance-level editing, we explore the rule-level editing problem, which involves editing rule-level knowledge encompassing abstract understandings of principles. As illustrated in Figure 1(b), since rule-level knowledge can derive numerous relevant instances, it is expected that the modifications and generalizations of rule-level knowledge in rule-level editing encourage the effective updates of numerous rule-derived instances. Since existing knowledge editing methods are primarily designed for instance-specific modifications, they struggle to accurately modify rule-level knowledge and effectively generalize edited knowledge to update corresponding rule-derived instances. As observed in Figure 2(b), these methods exhibit sub-optimal (F1 scores are below 15.0% in ROME, MEND, and LoRA) or imbalanced performance (GRACE achieves 94.0% in reliability, but drops significantly to 4.4% in generalization ability and 2.2% in portability) in rule-level editing task.

Moreover, existing knowledge editing datasets (e.g., zsRE (De Cao et al., 2021) and CounterFact (Meng et al., 2022)) are primarily designed to evaluate instance-level editing, leaving the potential of LLMs in rule-level editing underexplored. Besides, although ConceptEdit (Wang et al., 2024b) is introduced for editing concept definitions, it is confined to evaluating affiliation influence on associated instances (e.g., “whether FrancoAngeli belongs to category publisher?”), and is incapable of measuring the impact of rule changes in real-world scenarios (e.g., the effects of modifying drunk driving penalty provisions in legal texts on real-world cases). Consequently, to bridge these gaps, we construct a new benchmark RuleEdit for the rule-level editing task, covering three distinct domains (i.e., historical, medical, and legal) which respectively necessitate capabilities of numerical reasoning, hierarchical knowledge inheritance, and semantic reasoning in real-world scenarios.

In our work, we propose the Rule-Transfer Editing (RTE) method, which mitigates over-editing risk caused by redundant instance-specific modifications through effective knowledge generalization. Specifically, RTE efficiently updates rule-level knowledge by modularly compressing it into semantic-centralized representations using a T5-based amortization network (Raffel et al., 2020). To facilitate effective generalization of rule-level knowledge, RTE further aggregates and propagates query-relevant rule-level knowledge to the query for informative knowledge inference in LLMs, leveraging the prefix tuning technique (Li and Liang, 2021). Moreover, RTE effectively prevents the deterioration of general ability in base LLMs owing to the preservation of original parameters. Experimental results demonstrate that RTE achieves robust rule-level editing performance and strikes a good balance among reliability, generalization ability, and portability.

Our contributions are summarized as follows:
• We explore the rule-level editing problem, aiming to achieve effective knowledge updates in LLMs through rule-level knowledge generalization.
• We construct RuleEdit benchmark for comprehensive rule-level editing evaluation, covering three domains necessitating abilities of numerical reasoning, hierarchical knowledge inheritance, and semantic reasoning in real-world rule-level knowledge generalization.
• We propose the RTE method to propagate edited rule-level knowledge during inference for effective updates of relevant rule-derived instances, which avoids redundant instance-specific modifications and thereby mitigates over-editing risk. Our experimental results highlight that RTE achieves significant im-

![Over-editing risk in instance-level editing and performance of existing methods in rule-level editing](page_1012_180_388_388.png)

Figure 2: (a) Over-editing risk in LLaMA-2-7B after editing with ROME in instance-level. The editing success rate is evaluated on RULEmix. The reasoning performance is evaluated on GSM8K. (Cobbe et al., 2021). (b) Rule-level editing performance of existing methods on LLaMA-2-7B with 100 editing steps in RULEmix.
2 Related Work

Current knowledge editing methods can be broadly divided into three lines, including locate-and-edit, meta-learning, and memory-based editing methods, which are briefly reviewed in this section.

Locate-and-edit. Recently, several studies manage to localize and modify specific knowledge within transformers, guided by the "key-value neural memory" theory (Geva et al., 2021), while retraining- or fine-tuning-based editing methods (Hu et al., 2022; Kirkpatrick et al., 2017) are computationally expensive. Rather than individually altering parameters of located knowledge neurons or feedforward layers (Dai et al., 2022; Meng et al., 2022, 2023) through causal tracing, Li et al. (2024) simultaneously optimizes the hidden states of multi-head self-attention and feedforward networks to update target knowledge. Additionally, Wang et al. (2024a) attempts to locate the toxic region by measuring distribution separation across layers. However, causal tracing does not always pinpoint the actual effective model layers for editing, despite being a reasonable localization method (Hase et al., 2023). Furthermore, in sequential editing scenario, existing locate-and-edit methods are prone to overediting risk (Hartvigsen et al., 2023), leading to knowledge degradation issues.

Meta-learning. Considering the overfitting issue associated with fine-tuning on a single example, existing meta-learning-based editing methods employ the hypernetwork to better initialize model parameters and encourage faster training on the model. Specifically, Mitchell et al. (2022a) propose an editor network with a low-rank decomposition of the gradient, facilitating scalable and fast editing for large pre-trained language models. Furthermore, Tan et al. (2024) formulates parameter shift aggregation as a least-squares problem to encourage massive scale editing. Despite fast editing adaptation to new knowledge, current meta-learning-based methods still face the risk of catastrophic forgetting, which deteriorates the editing reliability and generalization ability during large-scale edits.

Memory-based Editing. Memory-based editing methods achieve knowledge preservation by incorporating external working memory. These methods can be briefly classified into two categories: (1) Weight-preserved methods (Zheng et al., 2023; Hartvigsen et al., 2023; Madaan et al., 2022; Dong et al., 2022), which perform knowledge editing through in-context learning and knowledge retrieval. Nevertheless, they mostly struggle with the challenge of processing unaffordable massive inputs in sequential editing or exhibit poor editing generalization ability. (2) Optimization-based method. Mitchell et al. (2022b) introduces a semi-parametric editor that stores model edits in external memory. However, its performance is limited by the scope classifier which relies on the training of the editing dataset. Although current memory-based editing methods achieve reliable editing for target knowledge, they encounter a generalization bottleneck due to the limitation of knowledge retrieval.

To sum up, existing knowledge editing methods are primarily designed for instance-specific modifications and struggle to balance the performance of reliability, generalization ability, and portability in knowledge editing. Although knowledge editing for structured knowledge (Zhong et al., 2023; Meng et al., 2022) has garnered considerable attention, substantial challenges remain in handling more complex knowledge representations (e.g., multi-modal knowledge (Wang et al., 2024c; Yuan et al., 2025, 2023) and unstructured knowledge (Akyürek et al., 2023; Deng et al., 2025)). Therefore, in this work, we explore efficient knowledge updates through generalization in rule-level editing.

3 Rule-Level Editing

3.1 Task Definition

Rule-level editing aims to modify general rule-level knowledge and propagate updates to rule-derived instances within LLMs. Specifically, given \( i \)-th new input-output rule-level knowledge pair \( (\mathcal{R}_i^x, \mathcal{R}_i^y) \), which is accompanied by \( k \) relevant input-output rule-derived instance pairs \( \{(\mathcal{I}_{i,j}^x, \mathcal{I}_{i,j}^y)\}_{j=1}^k \in (\mathcal{I}_i^x, \mathcal{I}_i^y) \), the LLMs need to be edited on rule-level knowledge to obtain a new model \( \mathcal{F}^* \). After editing on \( (\mathcal{R}_i^x, \mathcal{R}_i^y) \), it is expected that the relevant input-output rule-derived instances can be correctly updated as: \( \mathcal{F}^*(\mathcal{I}_{i,j}^x) = \mathcal{I}_{i,j}^y \).

3.2 Rule-Level Editing Evaluation

In this work, we conduct comprehensive evaluations of knowledge editing across three dimensions and three metrics described as follows. For rule-level knowledge updates, we measure in both Reliability (Rel.) and Generalization (Gen.) di-
dimensions (Zhang et al., 2024b; Yao et al., 2023) to reveal whether rule-level knowledge can be robustly edited. For relevant rule-derived instance knowledge, we measure in Portability (Port.) dimension to reflect whether relevant instances can be successfully updated through inference.

(1) Reliability: The success rate of editing rule-level knowledge:

\[
\mathbb{E}_{x_e, y_e \sim \mathcal{R}^x, \mathcal{R}^y} Score(\mathcal{F}^*(x_e), y_e)
\] (1)

(2) Generalization: The success rate of editing rule-level knowledge with rephrased rule input within the editing scope:

\[
\mathbb{E}_{x_e, y_e \sim \mathcal{R}^{x'}, \mathcal{R}^{y'}} Score(\mathcal{F}^*(x_e), y_e)
\] (2)

where \( (\mathcal{R}^{x'}, \mathcal{R}^{y'}) \) set represents the rephrased rule-level knowledge.

(3) Portability: The success rate of updating the relevant rule-derived instance knowledge, which provides a superior reflection of the model’s generalization ability (Zhang et al., 2024a):

\[
\mathbb{E}_{x_e, y_e \sim \mathcal{I}^x, \mathcal{I}^y} Score(\mathcal{F}^*(x_e), y_e)
\] (3)

To ensure the robustness of the evaluation, we simultaneously calculate the score using three metrics: (1) Accuracy (ACC). The proportion of matching tokens between the target and edited result, calculated based on exact position alignment in the sequence. (2) Exact Match (EM). If the edit result fully matches the target, it is considered correct. (3) F1. It is obtained by calculating the overlap of tokens between the target and prediction.

4 Rule-Transfer Editing Method

Inspired by Tack et al. (2024) that addresses online adaptation problem with the key idea of document feature extraction and memory-augmentation, we introduce a Rule-Transfer Editing method for effective modifications and generalizations of rule-level knowledge in the rule-level editing task, as depicted in Figure 3.

In RTE, the rule-level knowledge are modularly compressed into semantic-centralized representations using a T5-based amortization network (Phang et al., 2023), while preserving original out-of-scope knowledge by freezing parameters of base LLMs. To update rule-derived instances by rule-level knowledge generalization, relevant rule-level knowledge are aggregated into virtual prefix tokens according to the semantic relevancy with the query measured by aggregation network, and subsequently prepended to the query in LLMs by prefix tuning technique (Li and Liang, 2021) for informative knowledge inference. Moreover, the meta-learning paradigm encourages faster adaption to new knowledge updates in RTE during meta-testing phase.

4.1 Meta-Training Phase

In meta-training phase, the key idea is to better initialize the amortization network and the aggregation network in an end-to-end training manner, consequently encouraging faster editing adaptation in meta-testing phase.

Given a training edit set \( \mathcal{D}_{edit}^{tr} \), for each input-output rule-level knowledge pair \( (\mathcal{R}_i^x, \mathcal{R}_i^y) \in \mathcal{D}_{edit}^{tr} \), we concatenate it and modularly encode it into a compact representation \( \phi_i \) by a learnable T5-based hyper-amortization network \( \mathcal{H} \) with parameter \( \xi_{amort} \) (Raffel et al., 2020):

\[
\phi_i = \mathcal{H}(\xi_{amort}; [\mathcal{R}_i^x; \mathcal{R}_i^y])
\] (4)

such that we obtain a compact rule-level knowledge representation with the shape of \([L, 2, 2, P, H]\), where \( L \) represents the number of layers, the first 2 corresponds to the dimensions of encoder and decoder, the latter 2 corresponds the key and value prefixes, \( P \) denotes the number of virtual prefix tokens, and \( H \) is the hidden size.

In order to generalize edited rule-level knowledge to the probing query \( x_q \) (which belongs to \( \mathcal{R}^x \) during training), unlike existing memory-based editing methods (e.g., IKE (Zheng et al., 2023) and GRACE (Hartvigsen et al., 2023)) which require massive prompts in sequential edit or directly replace the layer’s hidden states, we consider aggregating the relevant rule-level knowledge representations within query scope as soft prefix \( \phi_r^* \), which encompasses prefixed model-internal key-value pair for each layer in LLMs. Thus, we utilize cross-attention block (Kim et al., 2019) as a learnable knowledge aggregation network \( \mathcal{G} \) to measure the semantic relevancy between the encoded query and the compressed rule-level knowledge set \( \{\phi_i\}_{i=1}^n \), and we subsequently obtain the aggregated soft prefix as:

\[
\phi_r^* = \mathcal{G}(\mathcal{H}(\xi_{input}; x_q); \{\phi_i\}_{i=1}^n)
\] (5)

where the query is encoded by the T5-based encoder with parameter \( \xi_{input} \) and same architecture as the above amortization network, and \( n \) denotes
Figure 3: Overview of RTE. In meta-training phase, the T5-based amortization network modularly compresses the updated rule-level knowledge into semantic-centralized representations. For each query, the aggregation network aggregates the representations of relevant knowledge into soft prefix, which encompasses model-internal key-value representations for all layers in LLM. Subsequently, leveraging the prefix tuning technique, each learned key-value pair derived from soft prefix is prepended to the original key-value pairs layer-wise during inference, thus facilitating the propagation of edited rule-level knowledge in RuleBank to update rule-derived instances in meta-testing phase.

the number of edited knowledge. Other than specifically choosing a knowledge modulation, the aggregation network expands the utilization of the knowledge set and avoids the wrong choice of relevant knowledge.

LLMs are built on the Transformer architecture, which mainly consists of a self-attention module and a feed-forward module. Assuming the frozen LLMs (F) consist of \( L \) layers, to propagate aggregated relevant knowledge to the query, in each attention module Attn\(^l\) of layer \( l \), we prepend the learned model-internal key and value representation \( K^l_r \) and \( V^l_r \) derived from soft prefix \( \phi^l_r \) to the original key and value representations \( K^l_e \) and \( V^l_e \) of query calculated from the former layer \( l - 1 \). Throughout the above simple yet effective deep prefix tuning process utilizing P-Tuning v2 (Liu et al., 2022), we have informative knowledge during the inference:

\[
K^l_e = [K^l_r; K^l_q], \quad V^l_e = [V^l_r; V^l_q],
\]
\[
\text{Attn}^l(Q^l_q, K^l_e, V^l_e) = \text{softmax}(Q^l_q(K^l_e)^T)V^l_e,
\]
\[
Z^l = \text{LN}(\text{Attn}^l(Q^l_q, K^l_e, V^l_e) + X^{l-1}),
\]
\[
h^l = \text{LN}(\text{FFN}(Z^l) + Z^l)
\]

where \( X^{l-1} \) denotes the output of former layer \( l - 1 \), LN represents the layer normalization operation, FFN represents the feed-forward network, and \( h^l \) denotes the output of layer \( l \) with query matrix \( Q^l_q \).

To efficiently optimize the T5-based amortization network and aggregation network over the frozen LLMs, we train the model \( F^* \) in an end-to-end manner with the objective of:

\[
\mathcal{L}_{edit} = \mathcal{L}_r(F^*(\mathcal{R}^x_i, \mathcal{R}^y_i)) \\
+ c_g * \mathcal{L}_g(F^*(\mathcal{R}^x_i', \mathcal{R}^y_i')) \\
+ c_p * \mathcal{L}_p(F^*(\mathcal{I}^x_i, \mathcal{I}^y_i))
\]

where \( \mathcal{L}_r, \mathcal{L}_g \) and \( \mathcal{L}_p \) are negative log-likelihood functions used to compute the loss, and both \( c_g \) and \( c_p \) are hyperparameters that govern the loss weight.

4.2 Meta-Testing Phase

Associating with the meta-learned hyper-model initialized in the meta-training phase, we manage to compress the rule-level knowledge of testing edit set \( \mathcal{D}^{test}_{edit} \) into a set of modularized representations, which is called the RuleBank. For each query, we aggregate and propagate the relevant rule-level knowledge from the RuleBank, thereby facilitating generalizing the rule-level knowledge to update the rule-derived instances during the inference phase of LLMs.

5 Experiments

In this section, we provide construction details of our benchmark RuleEdit. Moreover, we conduct
extensive experiments to explore the potential of LLMs in mitigating over-editing risk through rule-level editing and comprehensively evaluate the effectiveness of RTE.

5.1 Datasets
For comprehensive evaluations of rule-level editing performance in real-world scenarios, we construct RuleEdit benchmark, which is composed of both specific instances and the corresponding general rule-level knowledge covering three domains, including legal, medical, and historical domains.

We separately introduce the dataset generation processes for three domains: (1) For legal domain RULElegal, we collect a set of legal judgments with 16,000 laws from DISC-Law-SFT (Yue et al., 2023) dataset. Sequentially, we prompt the LLMs (e.g., GPT-4o-mini (OpenAI, 2024)) to generate 3 statutory rules for each law, accompanied by corresponding rephrased rules and 10 legal instances. (2) For medical domain RULEmedical, we collect 480 medicine classes categorized by NLM 1. For each, we obtain 10 associated medicinal substances by LLM, based on the hierarchical relationship of pharmacological effect, therapeutic usage, action mechanism, and chemical structure. (3) For historical domain RULEhistorical, we collect 3441 historical events from ATOKE dataset (Yin et al., 2024) and construct corresponding historical instances within the timeline. More detailed examples and the construction process are provided in Appendix A.

<table>
  <tr>
    <th>Dataset</th>
    <th>legal</th>
    <th>medical</th>
    <th>historical</th>
    <th>mix</th>
  </tr>
  <tr>
    <td>rule-level</td>
    <td>16,482</td>
    <td>3,186</td>
    <td>3,441</td>
    <td>9,450</td>
  </tr>
  <tr>
    <td>instance-level</td>
    <td>164,672</td>
    <td>17,539</td>
    <td>46,018</td>
    <td>90,675</td>
  </tr>
  <tr>
    <td>instance:rule</td>
    <td>10:0.1</td>
    <td>5.5:1</td>
    <td>13.4:1</td>
    <td>9.6:1</td>
  </tr>
</table>

Table 1: Statistics of RuleEdit across legal, medical, historical and mixed domains.

Moreover, Table 1 demonstrates the statistics of collected rule-level knowledge and relevant rule-derived instances for each domain after quality control. To encourage balanced and comprehensive evaluations of rule-level editing across three domains, we further randomly sample 3,150 input-output rule-level knowledge pairs and corresponding accompanied instances for each domain. Subsequently, the samples are mixed and shuffled together to obtain the composed dataset RULEmix. We compute the ratio of instances to rules in each domain. It is noticed that the mere difference among ratios is due to the motivation of ensuring generation quality while cascading unqualified data through quality control. In quality control, we modify or cascade unqualified cases according to the following guidelines (Details in Appendix A.2.1): (1) Clarity and completeness of knowledge. (2) Logical relevance between rules and instances. (3) Distinguishability among instances. (4) Factual reliability of the rules. (5) Inner-annotator agreement and expert review.

5.2 Experimental Settings
In our experiments, we compare against four representative distinct baselines, including (1) Parameter-efficient tuning method: LoRA (Hu et al., 2022); (2) Locate-and-edit method: ROME (Meng et al., 2022); (3) Meta-learning method: MEND (Mitchell et al., 2022a); (4) Memory-based method: GRACE (Hartvigsen et al., 2023). Besides, we utilize prevalent open-source LLMs LLaMA-2-7B (Touvron et al., 2023) and GPT-2-XL (1.5B) (Radford et al., 2019) as base models and conduct experiments on our constructed RuleEdit covering legal, medical, historical, and mixed domains for comprehensive evaluation. For RuleEdit we use the same train/test split (9:1) as Mitchell et al. (2022a). More details are provided in Appendix B.

5.3 Experimental Results
Rule-level editing is challenging to existing editing methods. The experimental result of rule-level editing shown in Table 2 indicates that existing editing methods struggle to balance the performance of reliability, generalization ability, and portability. Specifically, it can be observed that ROME suffers from overall performance collapse for three aspects in sequential editing, since multiple biased adjustments for parameters of predefined layers significantly deteriorate the overall knowledge in large-scale edits. Although GRACE exhibits prominent reliability in editing rule-level knowledge, it has trouble generalizing edited rule-level knowledge to relevant rule-derived instances (e.g., poor portability score of 1.9% in the final editing step on RULEmix). It is speculated that the update strategy of codebook in GRACE is insufficient to support precise measurement of semantic similarity, thereby limiting the retrieval of relevant knowledge. With competitive generalization ability and portability to the other baselines, MEND

1https://www.ncbi.nlm.nih.gov/mesh/68008511
<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULE<sub>mix</sub></th>
    <th colspan="4">RULE<sub>historical</sub></th>
    <th colspan="4">RULE<sub>medical</sub></th>
    <th colspan="4">RULE<sub>legal</sub></th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="18"><b>LLaMA-2-7B</b></td>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>85.9</td><td>93.3</td><td>95.3</td><td>91.5</td><td>93.3</td><td>100.0</td><td>72.6</td><td>88.6</td><td>85.0</td><td>60.0</td><td>54.8</td><td>66.6</td><td>12.9</td><td>15.2</td><td>5.9</td><td>11.3</td>
  </tr>
  <tr>
    <td>MEND</td><td>29.2</td><td>18.0</td><td>24.4</td><td>23.9</td><td>11.1</td><td>13.3</td><td>11.1</td><td>11.9</td><td>9.5</td><td>0.0</td><td>5.3</td><td>4.9</td><td>22.2</td><td>14.7</td><td>20.6</td><td>19.2</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>93.3</b></td><td>3.9</td><td>3.9</td><td>33.7</td><td>93.3</td><td>11.1</td><td>0.0</td><td>34.8</td><td><b>93.3</b></td><td>7.4</td><td>8.3</td><td>36.3</td><td><b>100.0</b></td><td>31.7</td><td>6.5</td><td>46.1</td>
  </tr>
  <tr>
    <td>LoRA</td><td>50.9</td><td>37.5</td><td>8.1</td><td>32.2</td><td>42.9</td><td>16.7</td><td>27.8</td><td>29.1</td><td>33.3</td><td>13.3</td><td>45.5</td><td>30.7</td><td>37.3</td><td>37.3</td><td>37.4</td><td>37.3</td>
  </tr>
  <tr>
    <td>Ours</td><td>83.0</td><td>54.6</td><td>68.7</td><td>68.8</td><td>50.0</td><td>50.0</td><td>36.3</td><td>45.4</td><td>77.8</td><td>45.7</td><td>50.8</td><td>58.1</td><td>47.7</td><td><b>54.2</b></td><td><b>66.7</b></td><td><b>56.2</b></td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>37.7</td><td>27.7</td><td>45.4</td><td>36.9</td><td>46.2</td><td><b>44.9</b></td><td>35.2</td><td>42.1</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>12.9</td><td>12.1</td><td>12.8</td><td>12.6</td>
  </tr>
  <tr>
    <td>MEND</td><td>15.1</td><td>14.1</td><td>17.0</td><td>15.4</td><td>6.7</td><td>12.3</td><td>0.0</td><td>6.3</td><td>2.9</td><td>4.4</td><td>4.4</td><td>3.9</td><td>14.8</td><td>15.1</td><td>18.5</td><td>16.1</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>88.2</b></td><td>3.2</td><td>0.6</td><td>30.7</td><td><b>88.6</b></td><td>3.3</td><td>0.0</td><td>30.6</td><td><b>98.0</b></td><td>2.2</td><td>3.8</td><td>34.7</td><td><b>98.6</b></td><td>9.5</td><td>3.6</td><td>37.2</td>
  </tr>
  <tr>
    <td>LoRA</td><td>30.0</td><td>8.6</td><td>11.0</td><td>16.5</td><td>40.0</td><td>19.7</td><td>9.2</td><td>23.0</td><td>10.0</td><td>0.0</td><td>0.0</td><td>3.3</td><td>23.2</td><td>26.4</td><td>20.7</td><td>23.4</td>
  </tr>
  <tr>
    <td>Ours</td><td><b>53.4</b></td><td><b>38.3</b></td><td><b>57.7</b></td><td><b>49.8</b></td><td><b>59.0</b></td><td><b>44.5</b></td><td><b>52.8</b></td><td><b>52.1</b></td><td><b>51.1</b></td><td><b>27.0</b></td><td><b>34.8</b></td><td><b>37.7</b></td><td><b>47.0</b></td><td><b>43.4</b></td><td><b>46.0</b></td><td><b>45.5</b></td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>6.1</td><td>2.7</td><td>5.8</td><td>4.9</td><td>0.5</td><td>0.0</td><td>0.1</td><td>0.2</td><td>2.8</td><td>2.4</td><td>2.5</td><td>2.6</td><td>18.5</td><td>18.3</td><td>18.1</td><td>18.3</td>
  </tr>
  <tr>
    <td>MEND</td><td>11.7</td><td>11.5</td><td>12.5</td><td>11.9</td><td>5.3</td><td>9.0</td><td>2.8</td><td>5.7</td><td>7.0</td><td>8.8</td><td>8.7</td><td>8.2</td><td>18.2</td><td>17.2</td><td>17.5</td><td>17.6</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>94.0</b></td><td>4.4</td><td>2.2</td><td>33.5</td><td><b>90.3</b></td><td>6.8</td><td>0.1</td><td>32.4</td><td><b>81.0</b></td><td>6.4</td><td>2.8</td><td><b>30.1</b></td><td><b>91.8</b></td><td>2.5</td><td>3.9</td><td>32.7</td>
  </tr>
  <tr>
    <td>LoRA</td><td>1.5</td><td>5.5</td><td>8.3</td><td>5.1</td><td>2.7</td><td>4.3</td><td>1.3</td><td>2.8</td><td>6.6</td><td>3.6</td><td>3.3</td><td>4.5</td><td>36.0</td><td>36.7</td><td>35.8</td><td>36.2</td>
  </tr>
  <tr>
    <td>Ours</td><td><b>44.7</b></td><td><b>40.7</b></td><td><b>44.3</b></td><td><b>43.2</b></td><td><b>47.7</b></td><td><b>39.2</b></td><td><b>42.6</b></td><td><b>43.2</b></td><td><b>36.2</b></td><td><b>22.5</b></td><td><b>20.8</b></td><td><b>26.5</b></td><td><b>53.2</b></td><td><b>53.5</b></td><td><b>48.6</b></td><td><b>51.8</b></td>
  </tr>
  <tr>
    <td rowspan="5">final</td>
    <td>ROME</td><td>0.5</td><td>0.5</td><td>0.1</td><td>0.4</td><td>0.0</td><td>0.0</td><td>0.0</td><td>1.4</td><td>1.3</td><td>1.7</td><td>1.5</td><td>10.7</td><td>10.8</td><td>11.5</td><td>11.0</td>
  </tr>
  <tr>
    <td>MEND</td><td>10.1</td><td>9.2</td><td>10.5</td><td>10.0</td><td>5.6</td><td>6.7</td><td>3.2</td><td>5.2</td><td>8.1</td><td>8.9</td><td>8.4</td><td>8.5</td><td>17.8</td><td>17.4</td><td>16.7</td><td>17.3</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>85.8</b></td><td>4.3</td><td>1.9</td><td>30.6</td><td><b>88.7</b></td><td>5.4</td><td>0.2</td><td>31.5</td><td><b>74.1</b></td><td>5.8</td><td>2.6</td><td><b>27.5</b></td><td><b>93.9</b></td><td>2.3</td><td>3.2</td><td>33.1</td>
  </tr>
  <tr>
    <td>LoRA</td><td>10.2</td><td>10.4</td><td>11.2</td><td>10.6</td><td>7.0</td><td>0.7</td><td>4.6</td><td>4.1</td><td>1.0</td><td>2.0</td><td>2.8</td><td>1.9</td><td>0.2</td><td>0.2</td><td>0.2</td><td>0.2</td>
  </tr>
  <tr>
    <td>Ours</td><td><b>38.8</b></td><td><b>38.0</b></td><td><b>39.3</b></td><td><b>38.7</b></td><td><b>53.1</b></td><td><b>40.8</b></td><td><b>50.5</b></td><td><b>48.2</b></td><td><b>31.6</b></td><td><b>22.9</b></td><td><b>22.4</b></td><td><b>25.6</b></td><td><b>53.9</b></td><td><b>53.3</b></td><td><b>47.7</b></td><td><b>51.6</b></td>
  </tr>
  <tr>
    <td>Ours(w/o SP)</td><td>5.11</td><td>4.75</td><td>3.95</td><td>4.60</td><td>1.96</td><td>1.88</td><td>2.07</td><td>1.97</td><td>7.93</td><td>6.65</td><td>5.26</td><td>6.61</td><td>5.07</td><td>4.86</td><td>4.88</td><td>4.94</td>
  </tr>
</table>

Table 2: Main Results of Rule-Level Editing on LLaMA-2-7B with Multiple Edit Steps Measured by F1 Metric. We evaluate all the methods in three aspects under RuleEdit, which consists of three domain-specific sets and a mixed domain set. Avg. indicates the average score of three aspects. SP indicates soft prefix. The final edit step indicates that all the rule-level knowledge of corresponding set are edited. Best and suboptimal results of each edit step are marked in bold and underline respectively.

and LoRA are prone to overfitting on training data and struggle with the adaptation of new knowledge, thereby resulting in poor reliability of edits (e.g., reliability scores of 10.1% in MEND and 10.2% in LoRA in the final editing step on RULE<sub>mix</sub>). Additional experimental results for ACC and EM metrics and analysis are provided in Appendix B.4.

Effective knowledge updates through our RTE method. As shown in Table 2, our RTE method exhibits remarkable average performance within most domains in sequential rule-level editing, revealing the effective updates of both rule-level knowledge and corresponding rule-derived instance knowledge in RTE through rule-level editing. Although GRACE achieves higher scores in reliability, it makes huge sacrifices in generalization ability and portability. Instead, RTE leads the best performances in both aspects, achieving the enhancement of 27.6% in generalization score and 28.1% in portability score over the best baseline for LLaMA-2-7B in final editing step on RULE<sub>mix</sub>. Moreover, the experiment highlights the adaptability of RTE across multiple domains, which necessitates capabilities in numerical reasoning, hierarchical knowledge inheritance, and semantic reasoning, as analyzed in Appendix A. This confirms that RTE achieves reliable rule-level editing through efficient knowledge amortization and robust knowledge generalization in inference.

Forward passing aggregated soft prefixes facilitates knowledge generalization. As shown in Table 2, the comparative experiment reveals significant performance gaps (e.g., a discrepancy of 34.11% in the average score on RULE<sub>mix</sub> in final step) in LLMs depending on whether relevant aggregated soft prefix knowledge is injected (Ours and Ours(w/o SP)). In comparison to GRACE, which directly replaces the activated state with the retrieved value and thus limits the generalization ability, our RTE demonstrates robust portability in inference, as indicated by the comparative experiment in Appendix B.3. Leveraging the prefix tuning technique, the prepended informative key-value representations derived from aggregated soft prefixes are forward passed to each attention layer in LLMs, thus facilitating thorough inference over edited rule-level knowledge to update corresponding rule-derived instances.

RuleBank serves as a safeguard against knowledge degradation. In Figure 4, as edit steps increase, it is worth noting that methods involved in multiple adjustments to parameters (including
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">RULE<sub>mix</sub></th>
    <th colspan="4">RULE<sub>historical</sub></th>
    <th colspan="4">RULE<sub>medical</sub></th>
    <th colspan="4">RULE<sub>legal</sub></th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="16">GPT2-XL (1.5B)</td>
  </tr>
  <tr>
    <td>ROME</td><td>0.63</td><td>0.68</td><td>0.63</td><td>0.65</td><td>13.72</td><td>16.90</td><td>18.70</td><td>16.44</td><td>7.27</td><td>6.71</td><td>7.18</td><td>7.06</td><td>15.59</td><td>14.92</td><td>14.97</td><td>15.16</td>
  </tr>
  <tr>
    <td>MEND</td><td>17.56</td><td>12.76</td><td>4.66</td><td>11.66</td><td>3.41</td><td>4.96</td><td>0.44</td><td>2.94</td><td>12.82</td><td>8.71</td><td>2.70</td><td>8.07</td><td>30.96</td><td>24.48</td><td>5.42</td><td>20.29</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>90.17</b></td><td>3.00</td><td>0.00</td><td>31.06</td><td><b>98.06</b></td><td>1.36</td><td>0.00</td><td>33.14</td><td><b>71.21</b></td><td>3.26</td><td>0.01</td><td><b>24.83</b></td><td><b>100.00</b></td><td>6.88</td><td>0.00</td><td>35.63</td>
  </tr>
  <tr>
    <td>LoRA</td><td>8.96</td><td>4.42</td><td>6.79</td><td>6.72</td><td>20.41</td><td>5.85</td><td>4.17</td><td>10.14</td><td>4.15</td><td>4.42</td><td>3.98</td><td>4.18</td><td>5.83</td><td>5.30</td><td>6.27</td><td>5.80</td>
  </tr>
  <tr>
    <td>Ours</td><td>43.40</td><td><b>39.86</b></td><td><b>34.91</b></td><td><b>39.39</b></td><td>54.88</td><td><b>49.13</b></td><td><b>50.71</b></td><td><b>51.57</b></td><td>29.25</td><td><b>20.39</b></td><td><b>20.33</b></td><td><b>23.32</b></td><td><b>57.46</b></td><td><b>55.54</b></td><td><b>50.07</b></td><td><b>54.36</b></td>
  </tr>
</table>

Table 3: Comparative Results of Rule-Level Editing on GPT2-XL in Final Edit Steps Measured by F1 Metric. We evaluate all the methods under RuleEdit, which consists of three domain-specific sets and a mixed domain set. Best and suboptimal results are marked in bold and underline respectively.

![Comparisons of rule-level editing results with multiple editing steps over different methods, which are evaluated on RULE_mix with F1 metric across three dimensions using LLaMA-2-7B and GPT2-XL.](page_180_624_1082_410.png)

Figure 4: Comparisons of rule-level editing results with multiple editing steps over different methods, which are evaluated on RULE_mix with F1 metric across three dimensions using LLaMA-2-7B and GPT2-XL.

MEND, LoRA, and ROME) suffer from catastrophic performance collapse in all dimensions. GRACE retains stable reliability performance owing to the memory codebook, but fails in generalization due to the limitation of knowledge retrieval ability. Contrarily, RTE achieves stable and excellent performance, owing to the preservation and propagation of rule-level knowledge from Rule-Bank. Furthermore, RTE achieves a leading portability score in GPT2-XL, indicating the superior generalization ability in rule-level editing.

Robust adaptability and pluggability in various backbones. In addition, Table 3 presents the experimental rule-level editing results conducted on GPT2-XL (1.5B). The prominent results reveal the robust adaptability of RTE to various LLMs backbones with distinct scales and indicate the promising potential of rule-level editing. Since the base LLMs is frozen and the edited rule-level knowledge is integrated flexibly through prefix tuning, our framework exhibits pluggability among LLMs. Moreover, comprehensive comparisons among editing methods in both LLaMA-2-7B and GPT2-XL are demonstrated in Appendix B.4.

5.3.1 Case Study

![Examples of knowledge editing results for different methods. Evaluated on the final edit step of RULE_mix using LLaMA-2-7B.](page_1012_1042_410_273.png)

Figure 5: Examples of knowledge editing results for different methods. Evaluated on the final edit step of RULE_mix using LLaMA-2-7B.

As illustrated in Figure 5, we perform a comparative case study over existing methods. It can be observed from the results that both ROME and LoRA produce unreliable and hasty generations, featuring typical over-editing results of repeated or meaningless tokens. It is indicated that an overfitting phenomenon occurs due to multiple biased modifications to the parameters. Mend produces incorrect answers due to erroneous generalization, revealing the limited generalization ability. As analyzed in the above experiments, the generalization ability of GRACE is constrained by retrieval performance, resulting in inaccurate generation with missing information. In contrast, our method delivers satisfactory results, demonstrating the promising potential to effectively generalize rule-level knowledge to update relevant rule-derived instances, thereby mitigating the over-editing risk.
6 Conclusion

In this work, we explore the rule-level editing problem to achieve effective knowledge updates and mitigate over-editing risk in LLMs, and construct a new benchmark RuleEdit across three domains for comprehensive evaluations. Additionally, we further propose RTE method to facilitate effective modifications and propagations of rule-level knowledge. Our experimental results demonstrate excessive rule-level editing performance of RTE with prevalent portability for effective knowledge generalization.

7 Limitations

Similar to most memory-based methods, our RTE method faces the challenge that RuleBank grows in scale as rule-level knowledge accumulates, leading to increased memory consumption. Future work may consider neighborhood knowledge fusion to reduce memory scale while maintaining editing performance, especially since RTE exhibits competitive performance in GPT2-XL compared with other baselines in LLAMA-2-7B. Additionally, a possible improvement involves designing a gate mechanism to selectively determine whether to integrate knowledge from RuleBank, thereby enhancing flexibility in knowledge integration.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (62476097), Guangdong Provincial Fund for Basic and Applied Basic Research-Regional Joint Fund Project (Key Project)(2023B1515120078), Guangdong Provincial Natural Science Foundation for Outstanding YouthTeam Project (2024B1515040010).

References

Afra Feyza Akyürek, Eric Pan, Garry Kuwanto, and Derry Wijaya. 2023. Dune: Dataset for unified editing. In Proceedings of the EMNLP 2023, pages 1847–1861.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the ACL 2022, pages 8493–8502.

Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the EMNLP 2021, pages 6491–6506.

Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2025. Everything is editable: Extend knowledge editing to unstructured data in large language models. In Proceedings of the ICLR 2025.

Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Proceedings of the EMNLP Findings 2022, pages 5937–5947.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the EMNLP 2021, pages 5484–5495.

Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Proceedings of the NeurIPS 2023, volume 36, pages 47934–47959.

Peter Hase, Mohit Bansal, Been Kim, and Asma Ghanadehauri. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Proceedings of the NeurIPS 2023, volume 36, pages 17643–17668.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings of the ICLR 2022.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. 2019. Attentive neural processes. arXiv preprint arXiv:1901.05761.

Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the ACL 2021, pages 4582–4597.

Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024. Pmet: Precise model editing in a transformer. Proceedings of the AAAI 2024, 38(17):18564–18572.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the ACL 2022 (Volume 2: Short Papers), pages 61–68.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceedings of the EMNLP 2022, pages 2833–2861.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Proceedings of the NeurIPS 2022, volume 35, pages 17359–17372.

Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In Proceedings of the ICLR 2023.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022a. Fast model editing at scale. In Proceedings of the ICLR 2022.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In Proceedings of the ICML 2022, volume 162 of Proceedings of Machine Learning Research, pages 15817–15831.

OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence.

Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. 2023. Hypertuning: toward adapting large language models without back-propagation. In Proceedings of the ICML 2023, pages 27854–27875.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In Proceedings of the ICLR 2020.

Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. 2024. Online adaptation of language models with a memory of amortized contexts. In Proceedings of the NeurIPS 2024.

Chenmien Tan, Ge Zhang, and Jie Fu. 2024. Massive editing for large language models via meta learning. In Proceedings of the ICLR 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. 2024a. Detoxifying large language models via knowledge editing. In Proceedings of the ACL 2024, pages 3093–3118.

Xiaohan Wang, Shengyu Mao, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. 2024b. Editing conceptual knowledge for large language models. In Proceedings of the EMNLP Findings 2024, pages 706–724.

Zecheng Wang, Xinye Li, Zhanyue Qin, Chunshan Li, Zhiyin Tu, Dianhui Chu, and Dianbo Sui. 2024c. Can we debias multimodal large language models via model editing? In Proceedings of the MM 2024, pages 3219–3228.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. In Proceedings of the EMNLP 2023, pages 10222–10240.

Xunjian Yin, Jin Jiang, Liming Yang, and Xiaojun Wan. 2024. History matters: Temporal knowledge editing in large language model. In Proceedings of the AAAI 2024, volume 38, pages 19413–19421.

Li Yuan, Yi Cai, Jin Wang, and Qing Li. 2023. Joint multimodal entity-relation extraction based on edge-enhanced graph alignment network and word-pair relation tagging. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11051–11059.

Li Yuan, Yi Cai, Jingyu Xu, Qing Li, and Tao Wang. 2025. A fine-grained network for joint multimodal entity-relation extraction. IEEE Transactions on Knowledge and Data Engineering, 37(1):1–14.

Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, et al. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:2309.11325.
Mengqi Zhang, Xiaotian Ye, Q. Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. 2024a. Knowledge graph enhanced large language model editing. In Proceedings of the EMNLP 2024.

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. 2024b. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? In Proceedings of the EMNLP 2023, pages 4862–4876.

Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the EMNLP 2023.

A Dataset

A.1 Data Samples

Table 4 presents data samples from our constructed benchmark RuleEdit. Specifically, RuleEdit is collected from three domains, including the legal domain, the medical domain, and the historical domain. Each dataset unit consists of rule-level knowledge for editing, rephrased rule-level knowledge for generalization evaluation, and relevant instances derived from the rule for portability evaluation.

As observed from the samples, dataset in the legal domain aims to enable proper judgment for specific cases after editing the corresponding statute, which requires robust semantic reasoning ability over edited legal rules for LLMs. Dataset in the medical domain aims to enable hierarchical knowledge inheritance from the edited universal medical knowledge. Moreover, dataset in the historical domain involves knowledge inference with specific time constraints, which requires solid numerical reasoning ability for LLMs.

A.2 Dataset Construction

Figure 6 outlines the detailed construction process of RuleEdit. Firstly, we collect knowledge from different corpuses across three domains. Based on the collected knowledge, we manage to extract and generate rule-level knowledge for editing, and rephrase the expression for generalization evaluation. Subsequently, we generate relevant instances that can be derived from the edited rule-level knowledge. As shown in Figure 7 and Figure 8, we prompt GPT-4o-mini to assist in the generation of rule-level knowledge and relevant instances in both legal and medical domains. Under quality control and random sampling, the dataset RuleEdit is obtained, which consists of separate data in three domains and a mixture set.

A.2.1 Dataset Quality Control Guidelines

To ensure high-quality annotations, we employ three well-educated annotators during the construction of the RuleBank and adhere to the following quality control guidelines: (1) Clarity and completeness of knowledge. Each input-output knowledge pair of rule-level knowledge and relevant rule-derived instances must be clearly described, leaving no ambiguity in interpretation or application. (2) Logical relevance between rules and instances. Rule-derived instances should be logically inferable from the corresponding rule-level knowledge without additional information. (3) Distinguishability among instances. Instances should be distinct and non-redundant, ensuring the diversity and coverage within the scope of the corresponding rule. (4) Factual reliability of the rules. Rules must be accurately derived from knowledge sources and free from contradictions. (5) Inner-annotator agreement and expert review. Annotators independently assess the quality of each input-output knowledge pair by assigning a score within the range of zero to five. Discrepancies are resolved through collaborative discussions, with final decisions made by a senior expert to refine the dataset.

B Experiments Details

B.1 Baselines

Here we provide a detailed introduction and implementation information for all baselines in the experiments.

ROME ROME (Meng et al., 2022) uses causal tracing to investigate the decisive feedforward MLPS associated with knowledge, and alters corresponding parameters by rank-one model with least squares approximation. For the experiments, the learning rate is set to 5e-1, the kl factor is set to 0.0625. For LLaMA-2-7B, ROME is executed in layer 5 with 25 optimization steps. For GPT2-XL, ROME is executed in layer 17 with 20 optimization steps.
<table>
  <tr>
    <th>Domain</th>
    <th>Rule-level knowledge</th>
    <th>Rephrased knowledge</th>
    <th>Relevant knowledge</th>
  </tr>
  <tr>
    <td><b>Legal</b></td>
    <td>If an individual intentionally destroys property with significant value, what is the criminal prosecution? <i>Imprisonment of up to three years, detention, or a fine.</i></td>
    <td>If an individual willfully damages property that is of considerable worth, what is the criminal prosecution? <i>Imprisonment of up to three years, detention, or a fine.</i></td>
    <td>If Tom destroys property valued at $10,000 or more, what is the criminal prosecution? <i>Imprisonment of up to three years, detention, or a fine.</i></td>
  </tr>
  <tr>
    <td><b>Medical</b></td>
    <td>If a medicine is a type of anticoagulant, what is the pharmacological effect of it? <i>Blood clot prevention.</i></td>
    <td>What is the pharmacological effect of a medicine that belongs to the class of anticoagulants? <i>Blood clot prevention.</i></td>
    <td>Warfarin is a type of anticoagulant. What is the pharmacological effect of warfarin? <i>Blood clot prevention.</i></td>
  </tr>
  <tr>
    <td><b>Historical</b></td>
    <td>Which club does Giorgio Morini affiliate with from 1976 to 1981? <i>A.C. Milan.</i></td>
    <td>From 1976 to 1981, Giorgio Morini played for? <i>A.C. Milan.</i></td>
    <td>Which club does Giorgio Morini affiliate with from 1979 to 1980? <i>A.C. Milan.</i></td>
  </tr>
</table>

Table 4: Examples of RuleEdit.

![A diagram showing the construction process of RuleEdit, including steps for knowledge collection, rule generation, instance generation, and dataset construction.](page_320_563_1047_246.png)

Figure 6: Construction process of RuleEdit.

MEND MEND (Mitchell et al., 2022a) designs a hypernetwork to decompose standard fine-tuning gradient of knowledge editing into corresponding rank-1 outer product form, and further adopts a meta-learning objective comprising the autoregressive loss and KL divergence loss. For the experiments, MEND edits in the last 3 transformer blocks, and the learning rate is set to 1e-6, while the scale of autoregressive loss and KL divergence loss are set to 0.1 and 1, respectively.

GRACE GRACE (Hartvigsen et al., 2023) maintains a discrete key-value codebook for a chosen layer to cache embedding for updated knowledge, and selectively replaces the activation of hidden state output with the retrieved value from the codebook during inference. For the experiments, the learning rate is set to 1. and the codebook is executed in layer 27 for LLaMA-2-7B and layer 35 for GPT2-XL.

LoRA LoRA (Hu et al., 2022) performs direct optimization for rank decomposition matrices of each layers, while keeping the pre-trained weight frozen. For the experiments, the learning rate is set to 5e-3, the rank is set to 8, and the dropout rate is set to 0.1.

B.2 Implementation Details

We conduct all the experiments on two NVIDIA A800 GPUs, and follow the default hyperparameter settings of the baselines. In our method, we utilize Adam optimizer (Kingma, 2014) with a learning rate of 1e-5 and train for 20 epochs for all datasets. We set the virtual output token number of T5-based amortization network to 24 and the training batch size to 16. Besides, following Tack et al. (2024), we utilize T5-large model and an individual two-layered MLP for each output virtual token for the amortization network, and T5-based model (Raffel et al., 2020) for the input encoder. For the aggrega-
f"# statute:\n{statute} \
Please follow these steps according to the given statute to ensure that the generated dataset meets the requirements:\n \
## Step 1: please brainstorm and generate legal entities as many as you can, Format the generated legal entities as follows:\n**Legal entities : [list of legal entity]\n \
## Step 2: Combining the generated legal entities, generate in English with more than 3 statutory rules with legal entity, condition, legal action and more than 10 legal instances for each statutory rule. It should includes:\n-- legal entity\n-- condition\n-- legal action type\n-- legal action result\n-- more than 10 legal rule instances\nFormat the statutory rule dataset as follows:\n\n**Statutory rule [index]:**\n**legal entity:** [legal entity]\n**Condition:** [Condition]\n**Rephrased Condition:** [Rephrased Condition]\n**Legal action type:** [Legal action type]\n**Legal action result:** [Legal action result]\n**Rule Instance [Instance index]:**\n-- **If-condition:** [If-condition]\n-- **Rephrased If-condition:** [Rephrased If-condition]\n-- **Legal action type:** [Legal action type]\n-- **Legal action result:** [Legal action result] "\

Figure 7: Sample prompt in the legal domain to assist generations of rule-level knowledge and relevant instances.

f"# \medicine class: [[medicine_class]]\n \
    Please follow these steps to ensure that the generated dataset meets the requirements:\n \
    ## Step 1: please brainstorm and generate mechanism of action of medicine classes as many as you can; Format the generated chemical structures as follows:\n**{classification criteria} : List of {classification criteria}\n \
    ## Step 2: Combining the generated {classification criteria} of medicine classes, generate medicine dataset with medicine class classified by {classification criteria}, and medicine instance. It should includes:\n-- the name of medicine class\n-- {classification criteria} of corresponding medicine class.\n-- more than 10 medicine instances belong to the medicine class\nFormat the medical dataset as follows:\n**Medicine Class [index]:** [[Name of Medicine class]]\n**[classification criteria]:** [[classification criteria] of Medicine class]\n**Medicine instances:** [[Medicine [Instance Index]:** [[--- **Medicine Name:** [Name of medicine instance]\n--- **Therapeutic effect:** [Therapeutic effect]\n--- **Active compound/ingredient:** [Active compound or ingredient ]"\

Figure 8: Sample prompt in the medical domain to assist generations of rule-level knowledge and relevant instances.

![Portability comparison of rule-level editing with EM, ACC, and F1 metrics under different methods, which are evaluated on RULE_{mix} using LLaMA-2-7B.](page_104_1342_627_355.png)

Figure 9: Portability comparison of rule-level editing with EM, ACC, and F1 metrics under different methods, which are evaluated on RULE_{mix} using LLaMA-2-7B.

tion network, we utilize four cross-attention blocks, which each consist of a cross-attention and a feed-forward network. According to the comparative results of different settings shown in Figure 10, we configure both the generalization loss weight \( c_g \) and the portability loss weight \( c_p \) to 0.1.

B.3 Solid Portability for Knowledge Generalization

As shown in Figure 9, we conduct rule-level editing experiments evaluated on EM, ACC, and F1 metrics, aiming to sufficiently compare the portability of current knowledge editing methods. It can be observed that RTE surpasses the other baseline in all metrics, indicating efficient generalization of rule-level knowledge. Compared with redundant case-by-case edits brought by instance-level editing, rule-level editing effectively avoids massive editing scale by solid portability to achieve efficient
updates on rule-derived instances.

B.4 Comprehensive Evaluation on RuleEdit
As shown in Table 5, 6, 7, 8, and 9, we conduct complementary experiments to comprehensively evaluate the performance of representative knowledge editing methods on both LLaMA-2-7B and GPT2-XL using ACC, EM, and F1 metrics. It can be observed from the results that RTE leads a favorable overall performance in both LLaMA-2-7B and GPT2-XL with ACC and F1 metrics, and exhibits superior generalization ability and portability compared with other baselines in EM metrics for both LLaMA-2-7B and GPT2-XL, which highlights the robustness and effectiveness of our method.

![Line chart showing Avg. score (%) vs Learning rate](page_312_624_324_180.png)
(a) Results on different setting of learning rate.

![Heatmap showing Avg. score (%) for different settings of generalization loss weight \( \epsilon_g \) and portability loss weight \( \epsilon_p \)](page_635_624_324_180.png)
(b) Results on different setting of loss weight.

Figure 10: Comparisons of avg. score on F1 metric among different parameter settings over LLaMA-2-7B on \( RULE_{mix} \).
<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULEmix</th>
    <th colspan="4">RULEhistorical</th>
    <th colspan="4">RULEmedical</th>
    <th colspan="4">RULElegal</th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="18"><b>LLaMA-2-7B</b></td>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>52.78</td><td>61.67</td><td>72.40</td><td>62.28</td><td>41.67</td><td>41.67</td><td>20.71</td><td>34.68</td><td>93.33</td><td>100.00</td><td>72.59</td><td>88.64</td><td>3.33</td><td>1.23</td><td>0.25</td><td>1.60</td>
  </tr>
  <tr>
    <td>MEND</td><td>16.73</td><td>6.08</td><td>5.88</td><td>9.57</td><td>12.50</td><td>9.72</td><td>0.00</td><td>7.41</td><td>11.11</td><td>13.33</td><td>11.11</td><td>11.85</td><td>9.14</td><td>9.14</td><td>13.10</td><td>10.46</td>
  </tr>
  <tr>
    <td>GRACE</td><td>61.67</td><td>0.00</td><td>0.26</td><td>20.64</td><td>41.67</td><td>0.00</td><td>0.00</td><td>13.89</td><td>93.33</td><td>11.11</td><td>0.00</td><td>34.81</td><td>88.77</td><td>1.23</td><td>1.67</td><td>30.56</td>
  </tr>
  <tr>
    <td>LoRA</td><td>46.11</td><td>30.56</td><td>4.22</td><td>26.96</td><td>16.67</td><td>16.67</td><td>26.19</td><td>19.84</td><td>42.86</td><td>16.67</td><td>27.78</td><td>29.10</td><td>26.67</td><td>26.67</td><td>27.00</td><td>26.78</td>
  </tr>
  <tr>
    <td>Ours</td><td>40.66</td><td>14.42</td><td>15.65</td><td>23.58</td><td>56.19</td><td>50.00</td><td>42.61</td><td>49.60</td><td>50.00</td><td>50.00</td><td>36.25</td><td>45.42</td><td>33.10</td><td>35.67</td><td>50.66</td><td>39.81</td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>12.38</td><td>5.71</td><td>16.75</td><td>11.62</td><td>2.00</td><td>2.00</td><td>3.18</td><td>2.39</td><td>1.25</td><td>0.00</td><td>0.74</td><td>0.66</td><td>3.17</td><td>7.30</td><td>5.90</td><td>5.46</td>
  </tr>
  <tr>
    <td>MEND</td><td>11.84</td><td>2.73</td><td>6.05</td><td>6.87</td><td>2.50</td><td>5.42</td><td>1.74</td><td>3.22</td><td>8.75</td><td>5.21</td><td>1.77</td><td>5.25</td><td>17.17</td><td>11.95</td><td>12.34</td><td>13.82</td>
  </tr>
  <tr>
    <td>GRACE</td><td>44.99</td><td>0.00</td><td>0.23</td><td>15.07</td><td>28.00</td><td>0.00</td><td>0.00</td><td>9.33</td><td>66.37</td><td>1.25</td><td>0.31</td><td>22.64</td><td>78.63</td><td>0.37</td><td>0.78</td><td>26.59</td>
  </tr>
  <tr>
    <td>LoRA</td><td>8.93</td><td>2.93</td><td>1.51</td><td>4.46</td><td>6.00</td><td>9.93</td><td>0.00</td><td>5.31</td><td>13.19</td><td>0.00</td><td>0.00</td><td>4.40</td><td>16.68</td><td>13.70</td><td>11.22</td><td>13.87</td>
  </tr>
  <tr>
    <td>Ours</td><td>43.06</td><td>16.40</td><td>21.81</td><td>27.09</td><td>63.94</td><td>59.00</td><td>43.21</td><td>55.38</td><td>65.87</td><td>43.83</td><td>30.99</td><td>46.89</td><td>33.07</td><td>38.22</td><td>42.69</td><td>38.00</td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>2.63</td><td>1.69</td><td>3.78</td><td>2.70</td><td>1.85</td><td>1.05</td><td>1.87</td><td>1.59</td><td>0.98</td><td>0.50</td><td>1.09</td><td>0.85</td><td>5.35</td><td>4.32</td><td>5.52</td><td>5.06</td>
  </tr>
  <tr>
    <td>MEND</td><td>9.91</td><td>6.82</td><td>6.14</td><td>7.62</td><td>7.43</td><td>5.82</td><td>2.48</td><td>5.24</td><td>8.80</td><td>7.60</td><td>3.75</td><td>6.72</td><td>11.37</td><td>11.39</td><td>8.62</td><td>10.46</td>
  </tr>
  <tr>
    <td>GRACE</td><td>59.11</td><td>0.88</td><td>0.29</td><td>20.09</td><td>36.98</td><td>2.02</td><td>0.20</td><td>13.07</td><td>47.09</td><td>0.31</td><td>0.14</td><td>15.85</td><td>73.35</td><td>0.95</td><td>0.69</td><td>25.00</td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.50</td><td>0.00</td><td>0.00</td><td>0.17</td><td>8.54</td><td>9.71</td><td>3.95</td><td>7.40</td><td>13.19</td><td>0.00</td><td>0.00</td><td>4.40</td><td>15.00</td><td>14.64</td><td>14.87</td><td>14.83</td>
  </tr>
  <tr>
    <td>Ours</td><td>45.20</td><td>30.39</td><td>24.65</td><td>33.41</td><td>55.01</td><td>47.67</td><td>39.88</td><td>47.52</td><td>65.87</td><td>43.83</td><td>30.99</td><td>46.89</td><td>38.37</td><td>39.90</td><td>39.26</td><td>39.18</td>
  </tr>
  <tr>
    <td rowspan="5">final</td>
    <td>ROME</td><td>0.34</td><td>0.42</td><td>0.35</td><td>0.37</td><td>4.68</td><td>2.05</td><td>6.89</td><td>4.54</td><td>0.19</td><td>0.03</td><td>0.20</td><td>0.14</td><td>4.36</td><td>4.56</td><td>4.68</td><td>4.54</td>
  </tr>
  <tr>
    <td>MEND</td><td>11.41</td><td>8.01</td><td>6.26</td><td>8.56</td><td>7.95</td><td>5.07</td><td>0.17</td><td>4.40</td><td>10.05</td><td>6.35</td><td>2.95</td><td>6.45</td><td>12.87</td><td>11.80</td><td>10.03</td><td>11.57</td>
  </tr>
  <tr>
    <td>GRACE</td><td>51.29</td><td>1.01</td><td>0.43</td><td>17.58</td><td>32.14</td><td>1.26</td><td>0.13</td><td>11.18</td><td>46.95</td><td>0.46</td><td>0.15</td><td>15.86</td><td>73.85</td><td>0.67</td><td>0.86</td><td>25.12</td>
  </tr>
  <tr>
    <td>LoRA</td><td>5.14</td><td>2.69</td><td>4.31</td><td>4.05</td><td>9.65</td><td>9.62</td><td>2.59</td><td>7.28</td><td>0.43</td><td>0.52</td><td>0.89</td><td>0.61</td><td>0.66</td><td>0.94</td><td>0.30</td><td>0.63</td>
  </tr>
  <tr>
    <td>Ours</td><td>45.12</td><td>29.35</td><td>23.40</td><td>32.63</td><td>59.64</td><td>53.15</td><td>51.90</td><td>36.35</td><td>47.53</td><td>41.39</td><td>35.54</td><td>41.49</td><td>42.27</td><td>42.07</td><td>38.68</td><td>41.01</td>
  </tr>
</table>

Table 5: Comparative Results of Rule-Level Editing on LLaMA-2-7B with Multiple Edit Steps Measured by ACC Metric.

<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULEmix</th>
    <th colspan="4">RULEhistorical</th>
    <th colspan="4">RULEmedical</th>
    <th colspan="4">RULElegal</th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="18"><b>LLaMA-2-7B</b></td>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>66.67</td><td>100.00</td><td>84.38</td><td>83.68</td><td>100.00</td><td>100.00</td><td>36.51</td><td>78.84</td><td>66.67</td><td>66.67</td><td>54.55</td><td>62.63</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td>
  </tr>
  <tr>
    <td>LoRA</td><td>33.33</td><td>33.33</td><td>0.00</td><td>22.22</td><td>33.33</td><td>3.33</td><td>3.17</td><td>12.17</td><td>33.33</td><td>0.00</td><td>45.45</td><td>26.26</td><td>33.33</td><td>33.33</td><td>33.33</td><td>33.33</td>
  </tr>
  <tr>
    <td>Ours</td><td>33.33</td><td>0.00</td><td>37.50</td><td>23.61</td><td>33.33</td><td>33.33</td><td>21.43</td><td>29.37</td><td>66.67</td><td>0.00</td><td>45.45</td><td>37.37</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>30.00</td><td>20.00</td><td>42.11</td><td>30.70</td><td>40.00</td><td>40.00</td><td>27.55</td><td>35.85</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>90.00</td><td>0.00</td><td>0.00</td><td>30.00</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td>
  </tr>
  <tr>
    <td>LoRA</td><td>30.00</td><td>0.00</td><td>0.00</td><td>10.00</td><td>50.00</td><td>0.00</td><td>0.00</td><td>16.67</td><td>10.00</td><td>0.00</td><td>0.00</td><td>3.33</td><td>10.00</td><td>10.00</td><td>2.00</td><td>7.33</td>
  </tr>
  <tr>
    <td>Ours</td><td>20.00</td><td>0.00</td><td>33.33</td><td>17.78</td><td>50.00</td><td>40.00</td><td>41.45</td><td>43.82</td><td>40.00</td><td>20.00</td><td>16.18</td><td>25.39</td><td>0.00</td><td>10.00</td><td>9.00</td><td>6.33</td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.33</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.33</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>94.00</td><td>0.00</td><td>0.00</td><td>31.33</td><td>99.00</td><td>1.00</td><td>0.00</td><td>33.33</td><td>83.00</td><td>0.00</td><td>0.00</td><td>27.67</td><td>83.00</td><td>0.00</td><td>0.00</td><td>27.67</td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.00</td><td>0.00</td><td>0.00</td><td>0.33</td><td>1.00</td><td>1.00</td><td>0.60</td><td>0.87</td>
  </tr>
  <tr>
    <td>Ours</td><td>21.00</td><td>13.00</td><td>26.13</td><td>20.04</td><td>34.00</td><td>28.00</td><td>33.17</td><td>31.72</td><td>24.00</td><td>16.00</td><td>16.70</td><td>18.90</td><td>5.00</td><td>7.00</td><td>14.37</td><td>8.79</td>
  </tr>
  <tr>
    <td rowspan="5">final</td>
    <td>ROME</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.56</td><td>0.00</td><td>0.00</td><td>0.52</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.11</td><td>0.00</td><td>0.50</td><td>0.20</td><td>0.00</td><td>0.58</td><td>0.00</td><td>0.19</td><td>0.31</td><td>0.00</td><td>0.11</td><td>0.14</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>86.02</b></td><td>0.32</td><td>0.03</td><td><b>28.79</b></td><td><b>98.55</b></td><td>0.29</td><td>0.00</td><td>32.95</td><td><b>73.98</b></td><td>0.00</td><td>0.31</td><td><b>24.76</b></td><td><b>86.66</b></td><td>0.00</td><td>0.00</td><td><b>28.89</b></td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.04</td><td>0.01</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>Ours</td><td>17.25</td><td><b>14.81</b></td><td><b>25.46</b></td><td>19.17</td><td>40.87</td><td><b>27.25</b></td><td><b>41.75</b></td><td><b>36.62</b></td><td>19.12</td><td><b>14.73</b></td><td><b>18.05</b></td><td>17.30</td><td><b>8.85</b></td><td><b>8.25</b></td><td><b>15.20</b></td><td><b>10.77</b></td>
  </tr>
</table>

Table 6: Comparative Results of Rule-Level Editing on LLaMA-2-7B with Multiple Edit Steps Measured by EM Metric.
<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULE<sub>mix</sub></th>
    <th colspan="4">RULE<sub>historical</sub></th>
    <th colspan="4">RULE<sub>medical</sub></th>
    <th colspan="4">RULE<sub>legal</sub></th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>73.64</td><td>57.92</td><td>2.69</td><td>44.75</td><td>75.56</td><td>22.22</td><td>0.56</td><td>32.78</td><td>55.30</td><td>55.30</td><td>53.18</td><td>54.60</td><td>3.33</td><td>3.33</td><td>1.97</td><td>2.88</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>13.81</td><td>12.24</td><td>8.68</td><td>0.00</td><td>0.00</td><td>6.00</td><td>2.00</td><td>0.00</td><td>16.67</td><td>0.41</td><td>5.69</td><td>1.28</td><td>10.00</td><td>2.00</td><td>4.43</td>
  </tr>
  <tr>
    <td>GRACE</td><td>87.92</td><td>6.67</td><td>0.00</td><td>31.53</td><td>75.56</td><td>6.67</td><td>0.07</td><td>27.43</td><td>55.30</td><td>15.15</td><td>0.00</td><td>23.49</td><td>88.72</td><td>0.00</td><td>0.00</td><td>29.57</td>
  </tr>
  <tr>
    <td>LoRA</td><td>46.97</td><td>46.97</td><td>51.53</td><td>48.49</td><td>30.30</td><td>30.30</td><td>41.32</td><td>33.98</td><td>30.30</td><td>30.30</td><td>41.32</td><td>33.98</td><td>26.67</td><td>26.67</td><td>26.79</td><td>26.71</td>
  </tr>
  <tr>
    <td>Ours</td><td>33.33</td><td>33.33</td><td>37.50</td><td>34.72</td><td>33.33</td><td>33.33</td><td>21.43</td><td>29.37</td><td>25.51</td><td>17.17</td><td>0.00</td><td>14.23</td><td>20.00</td><td>20.00</td><td>34.90</td><td>24.97</td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>48.10</td><td>18.38</td><td>11.56</td><td>26.01</td><td>55.00</td><td>14.67</td><td>19.63</td><td>29.77</td><td>35.89</td><td>28.78</td><td>28.70</td><td>31.12</td><td>34.88</td><td>31.54</td><td>9.96</td><td>25.46</td>
  </tr>
  <tr>
    <td>MEND</td><td>13.08</td><td>13.48</td><td>9.91</td><td>12.16</td><td>2.00</td><td>9.25</td><td>4.73</td><td>5.33</td><td>5.33</td><td>11.36</td><td>0.50</td><td>5.73</td><td>7.92</td><td>9.11</td><td>1.29</td><td>6.11</td>
  </tr>
  <tr>
    <td>GRACE</td><td>72.36</td><td>7.43</td><td>0.00</td><td>26.60</td><td>71.00</td><td>6.00</td><td>0.02</td><td>25.67</td><td>66.56</td><td>4.55</td><td>0.00</td><td>23.70</td><td>78.62</td><td>0.00</td><td>0.00</td><td>26.21</td>
  </tr>
  <tr>
    <td>LoRA</td><td>26.34</td><td>12.34</td><td>19.24</td><td>19.31</td><td>13.19</td><td>13.19</td><td>7.62</td><td>11.34</td><td>13.19</td><td>13.19</td><td>7.62</td><td>11.34</td><td>16.20</td><td>16.20</td><td>16.20</td><td>16.20</td>
  </tr>
  <tr>
    <td>Ours</td><td>32.86</td><td>38.75</td><td>30.87</td><td>34.16</td><td>41.67</td><td>31.67</td><td>21.27</td><td>31.54</td><td>51.12</td><td>20.15</td><td>0.77</td><td>24.02</td><td>34.33</td><td>33.34</td><td>29.98</td><td>32.55</td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>13.85</td><td>13.04</td><td>13.77</td><td>13.55</td><td>32.22</td><td>22.56</td><td>21.71</td><td>25.50</td><td>2.00</td><td>5.61</td><td>4.07</td><td>3.89</td><td>8.09</td><td>7.89</td><td>8.03</td><td>8.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>10.41</td><td>14.70</td><td>5.84</td><td>10.32</td><td>1.15</td><td>6.73</td><td>2.77</td><td>3.55</td><td>8.23</td><td>12.48</td><td>0.01</td><td>6.91</td><td>5.43</td><td>6.25</td><td>1.68</td><td>4.46</td>
  </tr>
  <tr>
    <td>GRACE</td><td>73.35</td><td>2.45</td><td>0.00</td><td>25.27</td><td>65.71</td><td>7.12</td><td>0.01</td><td>24.28</td><td>54.65</td><td>0.88</td><td>0.00</td><td>18.51</td><td>82.45</td><td>0.08</td><td>0.00</td><td>27.51</td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.25</td><td>0.25</td><td>0.32</td><td>0.27</td><td>0.65</td><td>0.65</td><td>0.41</td><td>0.57</td><td>0.65</td><td>0.65</td><td>0.41</td><td>0.57</td><td>6.06</td><td>5.71</td><td>6.53</td><td>6.10</td>
  </tr>
  <tr>
    <td>Ours</td><td>30.41</td><td>28.50</td><td>22.55</td><td>27.16</td><td>40.52</td><td>39.52</td><td>34.73</td><td>38.25</td><td>34.86</td><td>21.29</td><td>4.54</td><td>20.23</td><td>31.79</td><td>30.73</td><td>27.78</td><td>30.10</td>
  </tr>
  <tr>
    <td rowspan="5">final</td>
    <td>ROME</td><td>1.46</td><td>1.22</td><td>0.48</td><td>1.05</td><td>16.10</td><td>16.53</td><td>21.02</td><td>17.88</td><td>3.41</td><td>2.79</td><td>2.77</td><td>2.99</td><td>5.53</td><td>5.57</td><td>4.69</td><td>5.27</td>
  </tr>
  <tr>
    <td>MEND</td><td>11.08</td><td>14.19</td><td>6.03</td><td>10.43</td><td>1.28</td><td>7.34</td><td>3.10</td><td>3.91</td><td>8.56</td><td>9.17</td><td>0.60</td><td>6.11</td><td>6.78</td><td>6.18</td><td>1.74</td><td>4.90</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>65.07</b></td><td>2.33</td><td>0.00</td><td>22.47</td><td><b>66.85</b></td><td>7.62</td><td>0.00</td><td>24.82</td><td><b>46.58</b></td><td>0.45</td><td>0.00</td><td>15.68</td><td><b>79.54</b></td><td>0.39</td><td>0.00</td><td>26.64</td>
  </tr>
  <tr>
    <td>LoRA</td><td>1.94</td><td>2.00</td><td>2.22</td><td>2.06</td><td>0.55</td><td>0.92</td><td>1.24</td><td>0.90</td><td>0.55</td><td>0.92</td><td>1.24</td><td>0.90</td><td>6.33</td><td>5.72</td><td>7.75</td><td>6.60</td>
  </tr>
  <tr>
    <td>Ours</td><td>34.17</td><td>30.63</td><td>20.58</td><td>28.46</td><td><b>46.51</b></td><td><b>40.99</b></td><td><b>42.71</b></td><td><b>43.40</b></td><td>29.70</td><td>18.13</td><td><b>5.50</b></td><td><b>17.78</b></td><td><b>35.21</b></td><td><b>34.02</b></td><td><b>28.60</b></td><td><b>32.61</b></td>
  </tr>
</table>

Table 7: Comparative Results of Rule-Level Editing on GPT2-XL with Multiple Edit Steps Measured by ACC Metric.

<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULE<sub>mix</sub></th>
    <th colspan="4">RULE<sub>historical</sub></th>
    <th colspan="4">RULE<sub>medical</sub></th>
    <th colspan="4">RULE<sub>legal</sub></th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>66.67</td><td>33.33</td><td>57.22</td><td>52.41</td><td>100.00</td><td>33.33</td><td>19.44</td><td>50.92</td><td>66.67</td><td>66.67</td><td>66.67</td><td>66.67</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td>
  </tr>
  <tr>
    <td>LoRA</td><td>33.33</td><td>33.33</td><td>31.25</td><td>32.64</td><td>33.33</td><td>33.33</td><td>45.45</td><td>37.37</td><td>33.33</td><td>33.33</td><td>45.45</td><td>37.37</td><td>33.33</td><td>33.33</td><td>33.33</td><td>33.33</td>
  </tr>
  <tr>
    <td>Ours</td><td>33.33</td><td>33.33</td><td>37.50</td><td>34.72</td><td>33.33</td><td>33.33</td><td>21.43</td><td>29.37</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>11.11</td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>60.00</td><td>30.00</td><td>27.71</td><td>39.24</td><td>80.00</td><td>30.00</td><td>36.92</td><td>48.97</td><td>40.00</td><td>20.00</td><td>49.00</td><td>36.33</td><td>30.00</td><td>20.00</td><td>10.00</td><td>20.00</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td>
  </tr>
  <tr>
    <td>LoRA</td><td>30.00</td><td>0.00</td><td>0.00</td><td>10.00</td><td>10.00</td><td>10.00</td><td>10.29</td><td>10.10</td><td>10.00</td><td>10.00</td><td>10.29</td><td>10.10</td><td>10.00</td><td>10.00</td><td>10.00</td><td>10.00</td>
  </tr>
  <tr>
    <td>Ours</td><td>30.00</td><td>30.00</td><td>37.72</td><td>32.57</td><td>40.00</td><td>30.00</td><td>20.39</td><td>30.13</td><td>30.00</td><td>10.00</td><td>2.94</td><td>14.31</td><td>10.00</td><td>10.00</td><td>20.00</td><td>13.33</td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>7.00</td><td>6.00</td><td>2.30</td><td>5.10</td><td>36.00</td><td>28.00</td><td>21.55</td><td>28.52</td><td>2.00</td><td>6.00</td><td>4.27</td><td>4.09</td><td>1.00</td><td>1.00</td><td>0.30</td><td>0.77</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.00</td><td>0.00</td><td>0.33</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>98.00</td><td>1.00</td><td>0.00</td><td>33.00</td><td>85.00</td><td>0.00</td><td>0.00</td><td>28.33</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>Ours</td><td>21.00</td><td>19.00</td><td>23.65</td><td>21.22</td><td>37.00</td><td>34.00</td><td>31.50</td><td>34.17</td><td>19.00</td><td>13.00</td><td>12.90</td><td>14.97</td><td>7.00</td><td>7.00</td><td>10.55</td><td>8.18</td>
  </tr>
  <tr>
    <td rowspan="5">final</td>
    <td>ROME</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>10.14</td><td>8.99</td><td>10.06</td><td>9.73</td><td>0.63</td><td>0.31</td><td>0.31</td><td>0.42</td><td>0.06</td><td>0.00</td><td>0.05</td><td>0.04</td>
  </tr>
  <tr>
    <td>MEND</td><td>0.32</td><td>0.11</td><td>0.00</td><td>0.14</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.63</td><td>0.00</td><td>0.21</td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.00</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>91.31</b></td><td>0.00</td><td>0.00</td><td>30.44</td><td><b>99.13</b></td><td>0.29</td><td>0.00</td><td>33.14</td><td><b>72.73</b></td><td>0.00</td><td>0.00</td><td><b>24.24</b></td><td><b>99.94</b></td><td>0.00</td><td>0.00</td><td><b>33.31</b></td>
  </tr>
  <tr>
    <td>LoRA</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>Ours</td><td>20.74</td><td><b>18.10</b></td><td><b>17.93</b></td><td>18.92</td><td>42.90</td><td><b>35.65</b></td><td><b>39.46</b></td><td><b>39.34</b></td><td>16.61</td><td><b>10.03</b></td><td><b>11.83</b></td><td><b>12.83</b></td><td>11.22</td><td><b>10.67</b></td><td><b>13.98</b></td><td>11.96</td>
  </tr>
</table>

Table 8: Comparative Results of Rule-Level Editing on GPT2-XL with Multiple Edit Steps Measured by EM Metric.
<table>
  <tr>
    <th rowspan="2">Edit Step</th>
    <th rowspan="2">Method</th>
    <th colspan="4">RULE<sub>mix</sub></th>
    <th colspan="4">RULE<sub>historical</sub></th>
    <th colspan="4">RULE<sub>medical</sub></th>
    <th colspan="4">RULE<sub>legal</sub></th>
  </tr>
  <tr>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
    <th>Rel.</th><th>Gen.</th><th>Port.</th><th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">3</td>
    <td>ROME</td><td>88.89</td><td>52.94</td><td>65.64</td><td>69.16</td><td>100.00</td><td>82.22</td><td>52.71</td><td>78.31</td><td>93.33</td><td>93.33</td><td>68.86</td><td>85.18</td><td>11.94</td><td>13.46</td><td>6.01</td><td>10.47</td>
  </tr>
  <tr>
    <td>MEND</td><td>31.36</td><td>29.44</td><td>2.01</td><td>20.94</td><td>0.00</td><td>16.67</td><td>0.00</td><td>5.56</td><td>0.00</td><td>9.52</td><td>0.00</td><td>3.17</td><td>40.92</td><td>22.54</td><td>7.28</td><td>23.58</td>
  </tr>
  <tr>
    <td>GRACE</td><td>100.00</td><td>5.13</td><td>0.00</td><td>35.04</td><td>100.00</td><td>0.00</td><td>0.00</td><td>33.33</td><td>100.00</td><td>11.11</td><td>0.06</td><td>37.06</td><td>99.28</td><td>3.22</td><td>0.00</td><td>34.17</td>
  </tr>
  <tr>
    <td>LoRA</td><td>50.88</td><td>50.88</td><td>47.70</td><td>49.82</td><td>44.44</td><td>44.44</td><td>58.73</td><td>49.21</td><td>33.33</td><td>33.33</td><td>45.45</td><td>37.37</td><td>37.78</td><td>37.78</td><td>37.78</td><td>37.78</td>
  </tr>
  <tr>
    <td>Ours</td><td>87.97</td><td>84.05</td><td>86.45</td><td>86.16</td><td>50.00</td><td>50.00</td><td>41.96</td><td>47.32</td><td>22.86</td><td>22.86</td><td>26.48</td><td>24.06</td><td>50.17</td><td>50.17</td><td>58.57</td><td>52.97</td>
  </tr>
  <tr>
    <td rowspan="5">10</td>
    <td>ROME</td><td>83.11</td><td>64.42</td><td>43.47</td><td>63.67</td><td>80.00</td><td>37.78</td><td>56.38</td><td>58.05</td><td>50.55</td><td>34.86</td><td>56.89</td><td>47.43</td><td>50.59</td><td>53.47</td><td>18.38</td><td>40.81</td>
  </tr>
  <tr>
    <td>MEND</td><td>20.91</td><td>11.33</td><td>3.05</td><td>11.77</td><td>4.00</td><td>5.00</td><td>0.44</td><td>3.15</td><td>6.94</td><td>5.36</td><td>2.91</td><td>5.07</td><td>27.21</td><td>25.93</td><td>5.50</td><td>19.55</td>
  </tr>
  <tr>
    <td>GRACE</td><td>98.89</td><td>4.40</td><td>0.00</td><td>34.43</td><td>98.57</td><td>0.00</td><td>0.00</td><td>32.86</td><td>99.09</td><td>8.69</td><td>0.02</td><td>35.93</td><td>99.58</td><td>3.04</td><td>0.00</td><td>34.21</td>
  </tr>
  <tr>
    <td>LoRA</td><td>30.00</td><td>15.00</td><td>19.30</td><td>21.43</td><td>50.00</td><td>50.00</td><td>32.14</td><td>44.05</td><td>10.00</td><td>10.00</td><td>10.29</td><td>10.10</td><td>27.14</td><td>27.14</td><td>27.14</td><td>27.14</td>
  </tr>
  <tr>
    <td>Ours</td><td>59.80</td><td>60.63</td><td>51.09</td><td>57.17</td><td>49.00</td><td>39.00</td><td>32.83</td><td>40.28</td><td>45.75</td><td>27.94</td><td>16.70</td><td>30.13</td><td>44.78</td><td>49.65</td><td>50.20</td><td>48.21</td>
  </tr>
  <tr>
    <td rowspan="5">100</td>
    <td>ROME</td><td>21.75</td><td>18.24</td><td>17.77</td><td>19.26</td><td>46.60</td><td>40.14</td><td>33.38</td><td>40.04</td><td>12.92</td><td>14.14</td><td>10.96</td><td>12.67</td><td>21.42</td><td>19.66</td><td>22.44</td><td>21.17</td>
  </tr>
  <tr>
    <td>MEND</td><td>18.24</td><td>13.33</td><td>4.74</td><td>12.11</td><td>3.40</td><td>7.84</td><td>0.90</td><td>4.05</td><td>10.67</td><td>8.21</td><td>0.03</td><td>6.30</td><td>30.95</td><td>26.89</td><td>5.38</td><td>21.07</td>
  </tr>
  <tr>
    <td>GRACE</td><td>99.04</td><td>2.58</td><td>0.00</td><td>33.87</td><td>98.02</td><td>1.73</td><td>0.01</td><td>33.25</td><td>85.00</td><td>4.35</td><td>0.01</td><td>29.79</td><td>100.00</td><td>2.06</td><td>0.00</td><td>34.02</td>
  </tr>
  <tr>
    <td>LoRA</td><td>1.34</td><td>1.58</td><td>1.38</td><td>1.43</td><td>1.17</td><td>0.00</td><td>0.00</td><td>0.39</td><td>1.14</td><td>1.14</td><td>0.62</td><td>0.96</td><td>17.12</td><td>18.10</td><td>17.71</td><td>17.64</td>
  </tr>
  <tr>
    <td>Ours</td><td>48.48</td><td>43.93</td><td>41.25</td><td>44.55</td><td>49.90</td><td>46.47</td><td>41.25</td><td>45.87</td><td>36.54</td><td>25.97</td><td>20.06</td><td>27.52</td><td>57.63</td><td>53.34</td><td>51.63</td><td>54.20</td>
  </tr>
  <tr>
    <td rowspan="3">final</td>
    <td>ROME</td><td>0.63</td><td>0.68</td><td>0.63</td><td>0.65</td><td>13.72</td><td>16.90</td><td>18.70</td><td>16.44</td><td>7.27</td><td>6.71</td><td>7.18</td><td>7.06</td><td>15.59</td><td>14.92</td><td>14.97</td><td>15.16</td>
  </tr>
  <tr>
    <td>MEND</td><td>17.56</td><td>12.76</td><td>4.66</td><td>11.66</td><td>3.41</td><td>4.96</td><td>0.44</td><td>2.94</td><td>12.82</td><td>8.71</td><td>2.70</td><td>8.07</td><td>30.96</td><td>24.48</td><td>5.42</td><td>20.29</td>
  </tr>
  <tr>
    <td>GRACE</td><td><b>90.17</b></td><td>3.00</td><td>0.00</td><td><b>31.06</b></td><td><b>98.06</b></td><td>1.36</td><td>0.00</td><td><b>33.14</b></td><td><b>71.21</b></td><td>3.26</td><td>0.01</td><td><b>24.83</b></td><td><b>100.00</b></td><td>6.88</td><td>0.00</td><td><b>35.63</b></td>
  </tr>
  <tr>
    <td></td><td>LoRA</td><td>8.96</td><td>4.42</td><td>6.79</td><td>6.72</td><td>20.41</td><td>5.85</td><td>4.17</td><td>10.14</td><td>4.15</td><td>4.42</td><td>3.98</td><td>4.18</td><td>5.83</td><td>5.30</td><td>6.27</td><td>5.80</td>
  </tr>
  <tr>
    <td></td><td>Ours</td><td>43.40</td><td>39.86</td><td>34.91</td><td>39.39</td><td>54.88</td><td><b>49.13</b></td><td><b>50.71</b></td><td>51.57</td><td>29.25</td><td>20.39</td><td>20.33</td><td>23.32</td><td>57.46</td><td><b>55.54</b></td><td><b>50.07</b></td><td><b>54.36</b></td>
  </tr>
</table>

Table 9: Comparative Results of Rule-Level Editing on GPT2-XL with Multiple Edit Steps Measured by F1 Metric.

Reviewer paper summaries: This paper explores the challenges of knowledge editing in LLMs and their corresponding solutions. In response to the issue that existing instance-level editing methods may lead to over-editing, thereby reducing the editing performance and generalization ability of LLMs, the authors propose an innovative approach based on rule-level knowledge editing (RTE). To systematically evaluate rule-level editing methods, the authors also construct a new benchmark dataset—RuleEdit, which covers three domains: history, medicine, and law.
Summary of strengths: 1、The structure of this paper is rigorous, with a clear organization, making it very smooth to read and easy to understand.
2、The study is of significant importance in this field.
Summary of weaknesses: 1、There may be some issues with the current dataset construction. Do these rules (the edited rules) already exist in LLaMA-2-7B and GPT2-XL?
2、The comparison methods are all knowledge-level editing methods, which are generally effective only for single words (a few tokens).
3、This method is essentially a memory-based approach. It lacks comparisons with similar methods, such as SERAC and IKE. It is recommended to supplement the experiments.
Comments, suggestions and typos: 1、The experimental results indicate that the performance of LoRA is relatively low, with values only in the decimal range. This result is somewhat unexpected.
Overall assessment scores: 2.5