RelEdit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning
Authors: Yifan Niu, Miao Peng, Nuo Chen, Yatao Bian, Tingyang Xu, Jia Li
Accepted at: acl25

RelEdit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning

Yifan Niu1*, Miao Peng1*, Nuo Chen1, Yatao Bian2, Tingyang Xu3,4, Jia Li1,5†
1The Hong Kong University of Science and Technology (Guangzhou)
2Tencent AI Lab  3DAMO Academy, Alibaba Group  4Hupan Lab
5The Hong Kong University of Science and Technology
{yniu669, mpeng885}@connect.hkust-gz.edu.cn   jialee@ust.hk

Abstract

The conceptual knowledge in Large Language Models (LLMs) can become outdated over time, and concept editing is often an option. Current evaluations on conceptual knowledge editing primarily focus on whether the definitions of concepts are successfully edited, neglecting the impact on the model’s related beliefs. To address this gap, we introduce a benchmark called RelEdit, which includes criteria and questions to assess both concept-level and instance-level relational reasoning abilities of edited models. Our findings reveal that existing knowledge editing methods struggle to reason about related conceptual knowledge effectively. Additionally, we introduce a simple memory-based in-context editing baseline, MICE, which prompts the language model to generate answers that align with the stored edited concepts in external memory. In addition, we find that MICE obtains the best scores on our benchmark, suggesting a promising research direction for model editing. The code is available at https://github.com/ivanniu/RelEdit.

1 Introduction

As large language models (LLMs) are widely deployed, it becomes increasingly important to maintain their knowledge accuracy and currency without incurring significant retraining costs (Sinitisin et al., 2020). Previous studies have introduced knowledge editing methods to gradually incorporate new concrete factual knowledge into language models (Zhu et al., 2020; De Cao et al., 2021; Meng et al., 2022a,b; Mitchell et al., 2021, 2022; Tan et al., 2023). However, these approaches of editing case-by-case factual knowledge are highly inefficient and lacks the modeling of relationships between instances (Wang et al., 2024). Inspired by cognitive science (Zhao et al., 2024; Holzinger et al., 2023), researchers have explored on conceptual knowledge editing (Lo et al., 2024; Lv et al., 2024; Suresh et al., 2023; Jamali et al., 2023), which involves modifying higher-level abstract concept rather than concrete facts in LLMs. For example, the concept “Gender” is originally a binary biological concept, including instances “male” and "female". In recent years, with the development of society, the definition of “Gender” has introduced psychological factors and derived “Non-Binary Gender”.

Although conceptual knowledge editing have raised great research interest, there is no systematic study of evaluating this task. Currently, the only existing benchmark, ConceptEdit (Wang et al., 2024), focuses on measuring whether the edited model can recall the newly injected concept definitions, typically involving three common metrics: Reliability (Re), Generalization (Ge), and Locality (Lo). These metrics evaluate the answers through queries on the edited concept definitions, as well as verifying that irrelevant concepts are not corrupted. However, an important question that has not been addressed is whether the edited model can handle relational reasoning questions where the an-

![Illustration of the relational reasoning on instance-level and concept-level when editing the concept definition of "Gender".](page_1012_370_388_312.png)

Figure 1: Illustration of the relational reasoning on instance-level and concept-level when editing the concept definition of "Gender".
<table>
  <tr>
    <th>Benchmark</th>
    <th>Re</th>
    <th>Ge</th>
    <th>Lo</th>
    <th>IC</th>
    <th>PO</th>
    <th>IL</th>
    <th>AB</th>
    <th>AC</th>
  </tr>
  <tr>
    <td>ConceptEdit</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>RelEdit</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Metrics in concept editing benchmarks.

swer should change as a logical consequence of the edited concepts.

In this work, we introduce a benchmark RelEdit (Relational Reasoning for Conceptual Knowledge Editing). When evaluating concept editing, it is important to go beyond just examining the edited concept definition. We should also assess whether both the concepts and instances logically derived from the edit have been appropriately modified, as illustrated in Figure 1. Following existing benchmarks, we construct relational examples for evaluation. (i) At the instance level, we evaluate the relationship changes between concepts and concrete instances. (ii) At the concept level, we evaluate the relationship changes among concepts. From the two perspectives, we propose novel evaluation criteria to evaluate how well the model integrates the edit with the rest of its knowledge: Instance Change (IC), Portability (PO), Instance Locality (IL), Alignment Belong (AB), and Alignment Compare (AC), as listed in Table 1.

We observe that although current methods effectively edit concepts, they often struggle to infer the relational knowledge associated with those concepts. Additionally, our analysis reveals that: (a) larger models are better equipped to handle the relational reasoning challenge; (b) modifications between concepts that share similar semantic structures and superclasses facilitate the model’s ability to update more related conceptual knowledge; and (c) instance-level evaluations reflect the ability of LLMs to maintain higher concept consistency by updating knowledge from related instances. Finally, we propose a memory-based in-context editing baseline, MICE, which prompts the LLMs to generate answers that align with the stored edited concepts in external memory instead of explicit parametric updates. While MICE achieves superior results compared to current parametric methods on RelEdit, there is still significant room for improvement, necessitating further research.

2 Problem Formulation

In contrast to concrete factual knowledge, conceptual knowledge refers to the understanding of concepts, categories, principles, and relationships. It involves understanding and applying abstract ideas. It is evidenced that language models are capable of memorizing conceptual knowledge and making inferences (Wu et al., 2023a). The conceptual knowledge stored in language models can be incorrect or become outdated over time. One potential solution is to update the knowledge without retraining.

A concept can be represented as \( C = (c, d) \) (Wang et al., 2024), where \( c \) denotes the concept name (e.g., school) and \( d \) means the definition description of concept (e.g., an institution for the education of students by teachers). For a concept \( C = (c, d) \), there are several instances \( e \) belonging to the category broadly defined by the concept, which can be denoted as \( e \in C \) (e.g., Microsoft Forecaster is an instance of concept software).

A conceptual knowledge edit \( \delta : (c, d) \to (c, d^*) \) is defined as modifying the concept \( C = (c, d) \) into a refreshed one \( C^* = (c, d^*) \), that is, updating the concept definition \( d \to d^* \) for a given concept name \( c \), in which \( d^* \) represents the alternate concept definition. Given a language model, conceptual knowledge editing aims to inject the edited concept to the language models’ inner beliefs. Since high-order concepts preserve meta knowledge in a hierarchical form, it is quite essential to investigate the relational reasoning of a single concept editing to certain related concepts and instances.

3 RelEdit Benchmark Construction

In this section, we introduce the data construction of RelEdit. It is difficult to have a universal principle for all concepts in the world. To ensure the rationality of the concepts, our concepts follow the well-established DBpedia (Auer et al., 2007), also with human judgement. Some details about data construction are discussed in the Appendix A.

3.1 Ontology Building

To obtain high-order meta ontological knowledge, we adopt DBpedia (Auer et al., 2007), a widely used knowledge graph with tree-like structured ontology, constructing a hierarchical concept set with classes and corresponding individual instances. Specifically, we extract 783 distinct classes in total from DBpedia as performed by Wu et al., in which each class represents the typical ontological information. Note that several classes in DBpedia are free of instance facts, which lead to a marginal contribution to concept editing task. To this end,
Figure 2: Overall illustration of data generation pipeline: (1) We start by building up the ontology set. (2) then we generate the edit requests in intra and inter settings. (3) Then we rephrase the request prompt to neighbor edit request. (4) Next, we filter out those instances are not inherently stored in an LLM. (5) Last, we generate RelEdit evaluation queries for each concept edit.

we filter out those classes with the amount of instances less than \( k \). Besides, we employ SPARQL to query the corresponding instances of each class in DBpedia via the Type relation, and superclass of each class via the subclass of relation (We keep the highest-level class in the hierarchical superclass tree). These retrieved concepts, instances and superclasses formulate a hierarchical ontology set that can be utilized in the following data construction. To ensure the dataset is balanced, we randomly choose 20 instances for each class. Considering the lack of descriptive texts of ontological classes in DBpedia, we turn to retrieve corresponding definition descriptions of classes in Wikidata (Vrandečić and Krötzsch, 2014).

3.2 Data Generation Pipeline

Edits generation. For a edit request \( \delta : d \to d^* \) about concept \( C = (c, d) \), target concept description \( d^* \) is chosen from a different concept \( C^* = (c, d^*) \). Considering the different ontological scope, we categorize the edit request \( \delta \) into two different settings (intra and inter) according to the homogeneity between origin concept \( C \) and target concept \( C^* \). Specifically, under the intra setting, origin concept \( C \) and target concept \( C^* \) are subclasses of the same superclass, sharing close relationship to high-order semantics. In contrast, inter setting indicates the disparate category of \( C \) and \( C^* \) in high-level superclass. Thus, concept edit request can be formulated as “The definition of [concept name c] is [concept description d*]”, in which \( d^* \) is the definition replacement from either an intra or inter setting. Since our concepts are collected from real-world DBpedia, the rationale behind the two settings is to simulate real-world scenarios where concepts may evolve, ranging from minor (intra) to significant (inter) changes.

Neighbor phrasing. With obtained edit requests, the post-edited models are able to answer corrupted target definitions towards the query like “What is the definition of [concept name]?”. To this end, we rephrase the request prompt and construct neighbor edit requests with a different format but similar semantics with concept name \( c \) and target definition \( d^* \). The underlying reason is that these equivalent neighbors are supposed to be edited and maintain similar conceptual knowledge in the post-edited models. To keep close semantics between neighbor prompts, we apply GPT-4 to generate 20 equivalent neighbor query prompts for each concept edit request. We uphold text accuracy by conducting thorough checks on all generated query prompts, correcting any that are confusing or not precise.

Instance filtering. Considering the inconsistency between inherent prior knowledge of LLMs and instances from external knowledge graphs, the next
step is instance filtering. Concretely, to evaluate the relational reasoning challenges caused by concept editing, instances related to the concept are essential to be taking into consideration. However, some instances of a concept retrieved from DBpedia are not inherently stored in an LLM. To this end, we employ few-shot in-context learning approach to prompts LLM to judge the affiliation of instances to a specific concept. For those discrepancies between answers from LLM and facts from external knowledge graphs, we filter out these instances to ensure they have been “seen” by the LLM model. After completing all filtering steps for the backbone LLMs, we take the intersection of their results to form a uniform instance set.

Evaluation query generation. For each concept edit request, we generate test queries corresponding to the evaluation criteria we proposed in Section 4, aiming to evaluate the relational reasoning of concept editing. In what follows, we provide details on our implementation with built concept set.

For a subject concept \( C = (c, d) \) in concept set, we denote the set of instances that belong to concept \( C \) as \( I(C) = \{e|e \in C\} \). Similarly, instance set of target concept \( C^* \) can be denoted as \( I_{a}(C^*) \) in intra setting and \( I_{c}(C^*) \) in inter setting, separately. Noting the aforementioned instance sets are subsets after filtering. Under this circumstance, given a concept edit \( \delta : d \rightarrow d^* \), we first retrieve the instance set \( I(C) \) of origin concept, \( I_{a}(C^*) \) and \( I_{c}(C^*) \) of target concept, then we construct the evaluation queries for Instance Change and Portability in the format of “Whether [Instance] belongs to category [Concept]?”. For Instance Locality, we adopt the same query formula with concepts and instances unrelated to \( C \) or \( C^* \). For Alignment Belong, we retrieve superclasses of \( C \) and \( C^* \) in DBpedia and construct corresponding queries in the format of “Whether [Concept] belongs to superclass [Superclass]?”. We also utilize \( C \) and \( C^* \) to construct query in a comparison form like “Do [Concept C] and [Concept C*] belong to the same superclass?”. A complete example of all queries is demonstrated in Figure 2.

Quality Control. Ideally, the original description text of a concept should align with and be similar to the concept name, while the description of the edit target should remain unrelated. To ensure this, we utilize the powerful LLM Qwen2.5-72B to evaluate and retain cases where: (1) the original concept description supports the original concept name, and (2) the target concept description is irrelevant to the original concept name. Subsequently, we manually review all the collected descriptions, replacing any that are unclear or ambiguous. Details of Human Evaluation are shown in Appendix A.3.

<table>
  <tr>
    <th>Property</th>
    <th>Number</th>
  </tr>
  <tr>
    <td># of concepts (edit requests)</td>
    <td>452</td>
  </tr>
  <tr>
    <td># of instances</td>
    <td>8,767</td>
  </tr>
  <tr>
    <td># of superclasses</td>
    <td>22</td>
  </tr>
  <tr>
    <td>Average token length per description</td>
    <td>12.95</td>
  </tr>
  <tr>
    <td>Max/Min number of superclass</td>
    <td>163 / 1</td>
  </tr>
  <tr>
    <td># of target concept (intra)</td>
    <td>243</td>
  </tr>
  <tr>
    <td># of target concept (inter)</td>
    <td>286</td>
  </tr>
  <tr>
    <td>Max/Min number of target concept (intra)</td>
    <td>7 / 1</td>
  </tr>
  <tr>
    <td>Max/Min number of target concept (inter)</td>
    <td>6 / 1</td>
  </tr>
</table>

Table 2: RelEdit Benchmark Statistics

3.3 Data Statistics

We construct RelEdit benchmark following above mentioned data construction pipeline, which contains 452 conceptual edits in both intra and inter settings. The overview pipeline is described in Figure 2. More detailed statistics are listed in Table 2, showing that our generation pipeline results in 19.40 instances per concept and 12.95 token length per description on average. Datasets in inter setting involve more target concepts.

4 Evaluation Criteria

4.1 Instance-level Evaluation

We assess the cascading consequences between the concept and instances. The principle of instance-level evaluation primarily follows: when the definition of a concept is edited, the relationships between instances and the edited concept should also undergo corresponding changes. For example, we prompt the LLMs to answer the question “whether [instance] belongs to category [concept]?”

Instance Change (IC). Following ConceptEdit (Wang et al., 2024), we check whether the instance \( t \in C \) belonging to the original concept \( C \) now belongs to the edited concept \( C^* \). This evaluation metric, called Instance Change, is defined as:

\[
1 - G_{\theta_e}(C^*, t),
\]

where the \( G_{\theta_e}(C^*, t) \) returns value 0 when \( t \notin C^* \). Conversely, it returns a value of 1 when \( t \in C^* \). \( \theta_e \) is the parameter after concept editing.
Portability (PO). Portability evaluates the instances \( t^* \) belonging to the target concept of description \( d^* \). We check whether the instance \( t^* \) of the target concept now belongs to the edited concept \( C^* \). The Portability is formulated as follows:

\[
G_{\theta_e}(C^*, t^*).
\] (2)

Instance Locality (IL). This metric is assessed based on the frequency at which the predictions of the post-edit model remain unchanged in outscope neighbors. It assesses the model’s ability to correctly classify instances that are unrelated to the concept being edited. Given an unrelated concept \( C \) and instance \( t \), the Instance Locality is formulated as

\[
\mathrm{XNOR}\left(G_{\theta}(C, t), G_{\theta_e}(C, t)\right),
\] (3)

where XNOR(\( \cdot \)) returns 1 when the relationship remains unchanged and 0 when it changes.

4.2 Concept-level Evaluation

We assess the cascading consequences of concepts related to the edit from the hierarchical concept structure. The principle of concept-level evaluation primarily follows: *When the definition of a concept is updated, the attribution of related concepts should also change*.

Alignment Belong (AB). We evaluate the relationship between sub-concepts \( C \) and parent concepts \( C_p \) after editing. For example, we prompt the LLMs to answer the question “whether [concept A] belong to category [concept B]?”. Given that relational reasoning regarding conceptual knowledge can potentially span a large range, we focus on a 1-hop distance from the edit. AB is defined as:

\[
G_{\theta}(C_p^*, C^*),
\] (4)

where \( C_p^* \) is the parent concept of the the target concept with description \( d^* \).

Alignment Compare (AC). In the hierarchical structure of concepts, we evaluate whether two sibling concepts belong to the same parent concept. For example, we prompt the language model to answer the question “Whether [concept A] and [concept B] share a superclass?” In this test, we adopt a more challenging inter-editing setting, indicating that after editing, the sibling concept \( C_s \) that originally belonged to the same parent concept become different. AC is defined as:

\[
G_{\theta}(C_s^*, C^*),
\] (5)

where \( C_s^* \) is the superclass of the the target concept with description \( d^* \).

Remark Following the idea of established factual knowledge editing benchmark RIPPLEED-ITS (Cohen et al., 2024), our evaluation criteria are designed to check whether the structure of ontology in the edited model has corresponding changes. All our metrics are binary. If it returns 1, it means that the corresponding ontology structure in the model has successfully changed. If it returns 0, it means that the model fails to capture the related changes. Therefore, our metrics are objective and reasonable. In this work, we do not consider polysemous concepts. Existing knowledge editing methods will overwrite the original concept meaning, thus can not tackle polysemous concepts.

5 MICE: A Memory-Based Approach

In this section, we introduce MICE, a simple but effective baseline. MICE only requires updating the question in the prompt with relevant feedback, and no retraining is need. Our proposed MICE is shown in Figure 3. The idea of MICE is inspired by MeLLO (Zhong et al., 2023), and all edited concept names and their definitions are explicitly stored in an external memory. To achieve this, we convert all edited concepts into sentence statements using predefined templates. The concept name serves as the retrieval index, allowing us to retrieve the corresponding edited concept definition. MICE performs the following steps: (1) It uses language models to extract the involved concepts in the question; (2) It extracts the involved concept definitions in the internal memory; (3) It checks whether the concept in LLM is contradicted with memory, and make corrections; and (4) It prompts the LLM to answer the questions with the corrected concept. Details about MICE are illustrated in Appendix B.

![The illustration of MICE.](page_1016_180_393_273.png)

Figure 3: The illustration of MICE.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="6">Intra Setting</th>
    <th colspan="6">Inter Setting</th>
  </tr>
  <tr>
    <th>Re</th><th>Ge</th><th>Lo</th><th>PO</th><th>IL</th><th>AB</th>
    <th>Re</th><th>Ge</th><th>Lo</th><th>IC</th><th>PO</th><th>IL</th><th>AB</th><th>AC</th>
  </tr>
  <tr>
    <td><i>GPT2-XL</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FT</td><td>25.55</td><td>23.80</td><td>61.99</td><td>15.71</td><td>83.85</td><td>85.40</td><td>72.12</td><td>26.24</td><td>23.17</td><td>64.21</td><td>15.27</td><td>85.40</td><td>83.41</td><td>67.26</td><td>46.46</td>
  </tr>
  <tr>
    <td>MEND</td><td>89.77</td><td>76.60</td><td>76.27</td><td>0.22</td><td><b>98.23</b></td><td>98.67</td><td>93.81</td><td>89.97</td><td>76.22</td><td>79.21</td><td>0.22</td><td><b>98.23</b></td><td>98.89</td><td><b>95.58</b></td><td>67.70</td>
  </tr>
  <tr>
    <td>ROME</td><td>86.47</td><td>49.68</td><td>84.76</td><td><b>23.23</b></td><td>75.22</td><td>87.39</td><td>76.77</td><td>82.85</td><td>45.52</td><td>86.21</td><td><b>20.35</b></td><td>80.88</td><td>87.31</td><td>81.64</td><td>55.31</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>43.98</td><td>33.09</td><td>96.11</td><td>3.10</td><td>94.25</td><td><b>99.56</b></td><td>91.15</td><td>39.28</td><td>29.77</td><td><b>95.93</b></td><td>3.32</td><td>94.47</td><td><b>99.34</b></td><td>93.14</td><td>57.08</td>
  </tr>
  <tr>
    <td>PROMPT</td><td>88.26</td><td><b>86.30</b></td><td>70.54</td><td>4.65</td><td>94.69</td><td>48.01</td><td>76.55</td><td>88.54</td><td>86.24</td><td>70.59</td><td>3.76</td><td>96.90</td><td>46.68</td><td>74.34</td><td>87.39</td>
  </tr>
  <tr>
    <td>MICE (Ours)</td><td><b>90.02</b></td><td><b>85.64</b></td><td>81.65</td><td>18.32</td><td>96.86</td><td>65.42</td><td><b>94.63</b></td><td><b>92.92</b></td><td><b>88.68</b></td><td>90.62</td><td><b>17.63</b></td><td><b>97.22</b></td><td>65.64</td><td><b>84.63</b></td><td><b>89.36</b></td>
  </tr>
  <tr>
    <td><i>GPT-J-6B</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FT</td><td>56.19</td><td>55.70</td><td>36.71</td><td>19.25</td><td>68.14</td><td>65.93</td><td>69.03</td><td>52.68</td><td>53.23</td><td>37.87</td><td>20.13</td><td>71.90</td><td>71.02</td><td>72.12</td><td>67.48</td>
  </tr>
  <tr>
    <td>MEND</td><td>49.42</td><td>42.61</td><td>51.50</td><td>32.52</td><td>28.98</td><td>80.31</td><td>38.72</td><td>49.35</td><td>43.24</td><td>55.65</td><td>32.08</td><td>23.67</td><td>82.96</td><td>33.85</td><td>18.58</td>
  </tr>
  <tr>
    <td>ROME</td><td>99.20</td><td>83.01</td><td>70.14</td><td><b>32.74</b></td><td>57.52</td><td>83.19</td><td>48.67</td><td>99.21</td><td>81.94</td><td>71.07</td><td>31.86</td><td>55.53</td><td>80.97</td><td>47.79</td><td>42.04</td>
  </tr>
  <tr>
    <td>MEMIT</td><td><b>99.83</b></td><td>59.86</td><td><b>94.20</b></td><td>31.64</td><td>17.92</td><td><b>99.56</b></td><td>38.94</td><td><b>99.55</b></td><td>56.15</td><td><b>94.80</b></td><td><b>33.85</b></td><td>11.95</td><td><b>98.89</b></td><td>32.52</td><td>18.36</td>
  </tr>
  <tr>
    <td>PROMPT</td><td>88.41</td><td><b>86.42</b></td><td>69.10</td><td>0.44</td><td>95.58</td><td>87.17</td><td>85.40</td><td>88.66</td><td>87.01</td><td>70.14</td><td>0.88</td><td><b>96.46</b></td><td>86.28</td><td>78.98</td><td>91.59</td>
  </tr>
  <tr>
    <td>MICE (Ours)</td><td>90.22</td><td><b>84.63</b></td><td>78.38</td><td>17.38</td><td><b>97.74</b></td><td>94.15</td><td><b>88.49</b></td><td>91.17</td><td>86.42</td><td>82.75</td><td>14.70</td><td>96.08</td><td>94.56</td><td><b>84.72</b></td><td><b>92.34</b></td>
  </tr>
  <tr>
    <td><i>LLaMA-2-7B</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FT</td><td>47.02</td><td>42.11</td><td>79.90</td><td>15.93</td><td>11.06</td><td>90.49</td><td>49.78</td><td>43.26</td><td>38.39</td><td>80.74</td><td>14.60</td><td>4.20</td><td>88.50</td><td>30.53</td><td>8.85</td>
  </tr>
  <tr>
    <td>MEND</td><td>93.17</td><td>83.72</td><td>87.65</td><td>14.38</td><td>11.50</td><td>89.82</td><td>52.21</td><td>93.76</td><td>83.28</td><td>89.10</td><td>13.94</td><td>3.76</td><td>86.95</td><td>31.64</td><td>9.73</td>
  </tr>
  <tr>
    <td>ROME</td><td><b>99.66</b></td><td>75.22</td><td><b>92.41</b></td><td><b>40.71</b></td><td>19.47</td><td>92.70</td><td>65.27</td><td><b>99.63</b></td><td>74.58</td><td><b>92.91</b></td><td><b>43.81</b></td><td>11.95</td><td>87.61</td><td>49.34</td><td>32.30</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>96.86</td><td>80.23</td><td>89.28</td><td>34.07</td><td>25.66</td><td>91.15</td><td>64.82</td><td>97.11</td><td>80.76</td><td>90.02</td><td>40.49</td><td>20.35</td><td><b>89.16</b></td><td>51.77</td><td>29.65</td>
  </tr>
  <tr>
    <td>PROMPT</td><td>89.35</td><td>87.28</td><td>76.84</td><td>6.86</td><td>38.50</td><td>89.82</td><td>84.51</td><td>88.88</td><td><b>87.89</b></td><td>78.06</td><td>6.19</td><td>30.31</td><td>85.84</td><td><b>79.42</b></td><td>93.36</td>
  </tr>
  <tr>
    <td>MICE (Ours)</td><td>92.32</td><td><b>88.83</b></td><td>83.22</td><td>18.32</td><td><b>45.58</b></td><td><b>93.15</b></td><td><b>86.74</b></td><td>91.37</td><td>85.78</td><td>89.75</td><td>13.42</td><td><b>36.08</b></td><td>87.31</td><td><b>75.32</b></td><td><b>94.45</b></td>
  </tr>
  <tr>
    <td><i>Mistral-7B</i></td>
    <td></td><td></td><td></td><td></td><td></td><td></td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FT</td><td>36.12</td><td>33.51</td><td><b>97.62</b></td><td>0.00</td><td>50.66</td><td>100.0</td><td>91.37</td><td>34.31</td><td>32.30</td><td><b>97.70</b></td><td>0.00</td><td>35.84</td><td>100.00</td><td>64.16</td><td>9.51</td>
  </tr>
  <tr>
    <td>MEND</td><td>92.94</td><td>83.61</td><td>83.47</td><td>0.22</td><td>56.64</td><td>99.36</td><td>93.14</td><td>93.52</td><td>83.10</td><td>84.88</td><td>0.44</td><td>42.92</td><td>99.78</td><td>68.36</td><td>13.72</td>
  </tr>
  <tr>
    <td>ROME</td><td><b>96.44</b></td><td>76.11</td><td>93.99</td><td>10.62</td><td>78.32</td><td><b>99.36</b></td><td><b>95.58</b></td><td><b>96.56</b></td><td>76.00</td><td>94.57</td><td>11.50</td><td>70.58</td><td><b>100.00</b></td><td><b>89.82</b></td><td>65.27</td>
  </tr>
  <tr>
    <td>MEMIT</td><td>95.35</td><td>78.95</td><td>91.98</td><td><b>16.29</b></td><td>65.04</td><td>99.26</td><td>87.17</td><td>95.39</td><td>77.18</td><td>91.02</td><td><b>16.15</b></td><td>61.50</td><td><b>100.00</b></td><td>74.78</td><td>36.28</td>
  </tr>
  <tr>
    <td>PROMPT</td><td>90.22</td><td><b>88.65</b></td><td>81.31</td><td>0.44</td><td>94.25</td><td>94.69</td><td>88.50</td><td>90.17</td><td>88.68</td><td>82.75</td><td>0.22</td><td>92.88</td><td>94.47</td><td>77.65</td><td>58.13</td>
  </tr>
  <tr>
    <td>MICE (Ours)</td><td>93.14</td><td><b>86.13</b></td><td>86.72</td><td>6.23</td><td><b>95.52</b></td><td><b>100.0</b></td><td>90.68</td><td>92.46</td><td><b>90.35</b></td><td>89.74</td><td>5.57</td><td><b>94.93</b></td><td>98.32</td><td>76.02</td><td><b>95.64</b></td>
  </tr>
</table>

Table 3: Main results of baselines on RelEdit in both intra and inter setting with backbone model GPT2-XL, GPT-J-6B, LLaMA-2-7B and Mistral-7B. The best results are in bold and the second best results are underlined. Re, Ge and Lo are the abbreviation of metric Reliability, Generalization and Locality.

6 Experiments

In this section, we evaluate on RelEdit. Experimental details and results including the Impact of LLM Size, Conceptual Patterns across Methods and Editing Cases are in Appendix C and D.

6.1 Experimental Setup

Language models Four most prevalent open-source LLMs are used as base models for editing tasks. We use GPT-J (6B) (Wang and Komatsuaki, 2021) and GPT2-XL (1.5B) (Radford et al., 2019a), LLaMA-2-7B (Touvron et al., 2023a) and Mistral-7B-v0.1 (Jiang et al., 2023). It is important to mention that current parameter-update methods require a white-box language model and are highly computationally expensive. In Section 5, we present our approach MICE, which can be applied to large black-box language models.

Baselines We evaluate the following state-of-the-art knowledge editing approaches on our datasets: Fine-tuning (FT) (Zhu et al., 2020), MEND (Mitchell et al., 2021), ROME (Meng et al., 2022a), MEMIT (Meng et al., 2022b) and PROMPT (Wang et al., 2024).

Evaluation metrics Besides metrics we proposed, we also follow the commonly used metrics (Meng et al., 2022a; Wang et al., 2024) to conduct definition-level evaluation. Reliability (RE) measures the mean accuracy on a specific collection of pre-defined input-output pairs. Generalization (GE) measures the average accuracy on equivalent neighbor. Locality (LO) assesses the post-edit model remain unchanged in out-scope neighbor.

6.2 Main Results

We conduct experiments on RelEdit benchmark to evaluate the performance of 1) all baselines on conceptual knowledge editing; 2) our proposed MICE on conceptual knowledge editing.

Comparison among baseline editing methods. We report the evaluation results of all baselines on RelEdit benchmark in Table 3. It can be found that larger models are better handling relational reasoning challenge during conceptual knowledge editing. Considering results among five methods, we can observe that methods like ROME and MEMIT achieve great performance on traditional metrics like reliability, generation and locality, but struggle to handle relational reasoning challenges related to conceptual editing (e.g., from 16 to 68 on AB
criteria across all models in intra setting). This demonstrates that basic conceptual editing methods mostly focus on updating local factual knowledge but neglect the propagation of related concepts and instances. Besides, comparing results across evaluation criteria shows that some relational reasoning challenges are handled better than others. To be specific, IL mainly remains a high score across methods on four baselines with positive correlations to Locality (e.g., approximately 80-100 on both settings, compared to AB range from 16 to 84), while other criteria is generally low and vary greatly among methods. PROMPT practically achieves the best performance on both instance-level and concept-level metrics, while slightly lags behind ROME and MEMIT on traditional metrics. This indicates PROMPT captures relational conceptual knowledge due to the update editing.

Performance of MICE. We apply MICE on dour backbone models, and Table 3 shows the performance of MICE on intra setting and inter setting of RelEdit. We find that with the same base model, MICE outperforms FT, MEND and PROMPT significantly across all the settings while being more efficient and requiring no training. We find that in-context learning methods (MICE and PROMPT) significantly outperform traditional retraining methods in PO, IL and AB. Intuitively, locate-and-edit methods generalize well on original instances that are strongly related to concepts explicitly contained in edit prompts. This explains why ROME and MEMIT perform better on IC but lags behind in PO. Overall, the results suggest that MICE works particularly well on strong base language models. Along with its simplicity and efficacy, MICE can serve as a strong conceptual knowledge editing baseline for future research.

6.3 Analysis

Q1: How does the high-level superclass of edited concepts affect the relational updates of broader facts? We analyze the effect induced by knowledge editing methods to the model’s knowledge in different settings, to investigate the impact of the homogeneity between origin concept \( C \) and target concept \( C' \). We calculate the average scores of editing methods on RelEdit criteria, and results in Figure 4 show that scores in intra setting outperforms those in inter setting. Furthermore, the scores of concept-level (AB) between intra and inter settings (91.10 v.s. 77.65) have a larger gap than those of instance-level (IC and PO).

![Bar chart showing average scores of Mistral-7B in both intra and inter settings](page_1012_120_388_312.png)

Figure 4: The average scores of Mistral-7B in both intra and inter settings. Results are averaged over MEND, ROME, MEMIT and PROMPT.

This phenomenon can be attributed to the diverse superclasses of concepts. In intra setting, LLM pre-exists higher-level connection of two concepts, and they share similar semantic structures and are likely to be represented in close proximity within the model’s internal knowledge manifold. As a result, updates to one concept can naturally propagate to related concepts and their instances, with minimal disruption to the overall knowledge structure. This explains the worse performance of editing models on concept-level criteria in inter setting, which indicates editing knowledge in such a heterogeneous context requires the model to form new or cross-cutting associations that may not exist in its pre-trained structure. This not only increases the difficulty of precise knowledge insertion but also raises the risk of unintended interference with unrelated concepts.

Q2: What does instance-level criteria reflect about the knowledge that LLMs captured in conceptual editing? Considering IC, PO and IL scores of LLaMA-7B in Table 3, it is evidenced that ROME and MEMIT outperform PROMPT in instance-level criteria. To further explore the capabilities that instance-level criteria reflects about edited model, we evaluate the Concept Consistency (Wang et al., 2024) of LLaMA-2-7B with ROME, MEMIT and PROMPT using GPT4 API, and results are shown in Figure 5. It can be observed that ROME and MEMIT achieve great performance in editing conceptual knowledge in a semantic perspective. This evaluation demonstrates that models with strong instance-level scores tend to excel in capturing the semantic modifications from concept edits. Instance-level reflects the ability of LLMs to update knowledge from related in-
Figure 5: Concept consistency on LLaMA-2-7B with ROME, MEMIT and PROMPT. Target means the generated sentence has close semantics to target, vice versa to Origin. Ambiguity means neither similar to both.

Figure 6: Augmented massive concept editing of MEMIT on RelEdit with LLaMA-2-7B. Origin indicates that MEMIT without augmented edits.

stances of target concepts, equipping them with higher concept consistency.

Q3: Whether massive editing with related concepts facilitate model abilities to conduct relational reasoning regarding conceptual knowledge? To further explore the mechanism of editing models in relational reasoning regarding conceptual knowledge, we conduct experiments by massive editing of MEMIT with augmented concept editing prompts. We employ a few-shot approach (Brown et al., 2020) to prompt LLM to generate the target concept name according to target description \( d^* \), along with instances that belong to target concept. Detailed prompts are provided in Appendix A.1. With obtained augmentation edits, we apply \( n \)-batch massive edit of MEMIT on RelEdit, aiming to incorporate related instance-level knowledge into LLM. As shown in Figure 6, we compare the origin MEMIT to two variants with different number of augmented edits, and results show that simply introduce instance-level augmentations are beneficial for improving instance-level criteria (IC and PO), especially in inter setting. Furthermore, it is evidenced that incorporating more augmentation edits can slightly increase the performance on IC and PO, but leads to a marginal drop on concept-level criteria like AC.

7 Related work

Methods for knowledge editing. Various techniques have been proposed to edit the knowledge stored in a model. Some of these approaches involve identifying and adjusting the model’s weights that correspond to specific concepts. Notable examples include KN (Dai et al., 2021), ROME (Meng et al., 2022a), and MEMIT (Meng et al., 2022b). Another line of research adopts meta-learning and utilizes a hyper-network, which is a smaller network responsible for generating the edited gradients. This category includes methods like KE (De Cao et al., 2021), MALMEN (Tan et al., 2023), and MEND (Mitchell et al., 2021). Additionally, there are preservative methods that incorporate explicit memory and prompting techniques to rectify model predictions. Noteworthy examples in this category are SERAC (Mitchell et al., 2022), MemPrompt (Madaan et al., 2022), McLLO (Zhong et al., 2023), and IKE (Zheng et al., 2023).

Evaluation of conceptual knowledge editing. Existing knowledge editing evaluations mainly focus on factual knowledge. Initially, the assessment limited to verifying whether the target had been successfully modified and ensuring that unrelated details remained unaffected (Yao et al., 2023; Zhong et al., 2023). Recently, several benchmarks argue that the rippling changes in factual knowledge editing should also be evaluated (Cohen et al., 2024; Li et al., 2024; Ma et al., 2024). However, conceptual knowledge differs in form from concrete factual knowledge, and there is no systematic evaluation of conceptual knowledge editing to date. The only existing benchmark, ConceptEdit (Wang et al., 2024), concentrates on measuring whether the edited model can recall newly injected concept definitions. Complementing existing evaluation tools, RelEdit focuses on assessing whether edited models can answer relational reasoning questions.

8 Conclusion

In this work, we introduce a benchmark called RelEdit, which evaluates conceptual knowledge editing of LLMs. We find that existing knowledge editing methods struggle with answering questions related to specific instances and broader concepts. Additionally, we show that a simple in-context editing method with an external memory achieves the best results on RelEdit, highlighting the potential of such editing approaches.
Limitations

The limitations of our work are as follows.

• Although RelEdit focuses on the ripple effect of edits, it does not explicitly verify the item-part relationships between concepts. For instance, if we modify the definition of a wheel, it raises the question of how the definition of a car should be adjusted accordingly, as a wheel is a part of a car. Currently, there is no established and universally accepted rule for addressing this issue. We note that building such an evaluation is hard.

• RelEdit does not include concepts with more than one meaning. Existing knowledge editing methods cannot handle concepts with multiple meanings, thus RelEdit does not include polysemous concepts. It is an important future direction for conceptual knowledge editing research.

• In our data generation pipeline, we depend on concepts extracted from an existing knowledge base, specifically DBpedia, which could be incomplete or outdated. These concerns might be an issue when aiming for a comprehensive evaluation. An alternative solution worth exploring is to utilize the internal knowledge of language models instead of relying solely on external knowledge bases.

• Considering the limitations of computational resources, our evaluation of existing knowledge editing methods primarily focuses on limited models. We leave the evaluation on other models as future work.

Ethics Statement

This study adheres strictly to the most rigorous ethical standards and best practices in research. All data utilized are extracted from datasets that are available to the public, thereby ensuring no usage of any proprietary or sensitive information. As a result, this research is free from any ethical concerns. We have implemented measures to reduce the presence of offensive content in our dataset. Throughout the construction process, we utilized rigorous filtering methods to identify and remove material that could be deemed harmful or inappropriate.

In our research, we examine LLMs’ sensitivities and preferences regarding concept-related updates and edits through our proposed benchmark and experimental findings. This focus aligns with recent recognition of concept editing as a crucial research direction for language models (Suresh et al., 2023; Wu et al., 2023b). Our work aims to contribute to the development of more robust LLMs, ultimately advancing the field’s understanding of model behavior during knowledge updates.

Deliberately editing the concept definitions of LLMs may raise serious ethical issues. If the purpose of editing is to introduce erroneous meanings or misleading reasoning, this may lead to the model generating harmful content. When editing an LLM in research, it is important to update correct knowledge, which is necessary and beneficial. Researchers need to consider whether the output of the model is consistent with social values and ethical standards.

Acknowledgments

This research was supported by Guangdong S&T Program C019.

References

2024. Qwen2 technical report.

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In international semantic web conference, pages 722–735. Springer.

Baichuan. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283–298.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696.

Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164.

Andreas Holzinger, Anna Saranti, Alessa Angerschmid, Bettina Finzel, Ute Schmid, and Heimo Mueller. 2023. Toward human-level concept learning: Pattern benchmarking for ai algorithms. Patterns.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. CoRR, abs/2112.09118.

Mohsen Jamali, Ziv M Williams, and Jing Cai. 2023. Unveiling theory of mind in large language models: A parallel to single neurons in the human brain. arXiv preprint arXiv:2309.01660.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. 2024. Unveiling the pitfalls of knowledge editing for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Michelle Lo, Shay B Cohen, and Fazl Barez. 2024. Large language models relearn removed concepts. arXiv preprint arXiv:2401.01814.

Yaojia Lv, Haojie Pan, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. 2024. Cogcpt: Unleashing the power of cognitive dynamics on large language models. arXiv preprint arXiv:2401.08438.

Jun-Yu Ma, Jia-Chen Gu, Ningyu Zhang, and Zhen-Hua Ling. 2024. Neighboring perturbations of knowledge editing on large language models. CoRR, abs/2401.17623.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372.

Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.11309.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019a. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019b. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Anton Sinitisin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. arXiv preprint arXiv:2004.00345.

Siddharth Suresh, Kushin Mukherjee, Xizheng Yu, Wei-Chun Huang, Lisa Padua, and Timothy Rogers. 2023. Conceptual structure coheres in human cognition but not in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 722–738.

Chenmien Tan, Ge Zhang, and Jie Fu. 2023. Massive editing for large language model via meta learning. In The Twelfth International Conference on Learning Representations.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023a. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.

Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.

Xiaohan Wang, Shengyu Mao, Ningyu Zhang, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. 2024. Editing conceptual knowledge for large language models. arXiv preprint arXiv:2403.06259.
Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. 2023a. Do plms know and understand ontological knowledge? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3080–3101.

Weiqi Wu, Chengyue Jiang, Yong Jiang, Pengjun Xie, and Kewei Tu. 2023b. Do plms know and understand ontological knowledge? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3080–3101.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 10222–10240. Association for Computational Linguistics.

Bonan Zhao, Christopher G Lucas, and Neil R Bramley. 2024. A model of conceptual bootstrapping in human cognition. Nature Human Behaviour, 8(1):125–136.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740.

Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795.

Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363.
A Benchmark Construction Details

A.1 Templates

Here we numerates a variety of templates employed within our experimental framework.

As mentioned in Question 3 of Section 6.3 Analysis, we employ in-context learning approaches to prompt the LLM to generate augmented prompts based on editing requests.

The following Template 1 shows the few-shot prompt used for prompting the LLM itself to generate concept name according to the concept description text.

The following Template 2 shows the few-shot prompt used for prompting the LLM itself to generate instances according the concept name.

The structured template for Concept Consistency, as depicted in Template 3, serves as an input for the GPT-4 evaluator. This template facilitates a qualitative analysis that categorizes the generated sentences into three distinct scores. By employing a relative comparison instead of assigning fixed values, the approach recognizes the evaluator’s proficiency, which has been preliminarily confirmed to be more closely aligned with human judgment. This method allows for a more accurate evaluation of the generated content.

Template 1: Concept Name Generation Prompt

Given a description X about a concept, please generate the name Y of the concept according to the description X. The Name Y has to be a named entity and as short as possible.

Description X: company that prints and distributes pressed goods or electronic media
Name Y: publisher

Description X: those who serve as part of an organized armed military force
Name Y: military person

[8 in-context demonstrations abbreviated]

Description X: {new concept description}
Name Y:

Template 2: Instance Generation Prompt

Given a name X of a concept, please generate a list Y containing 5 instances that belong to the concept X. The list Y has to be a list including 10 named entities, divided by identifier ','.

Concept Name X: publisher
Instance List Y: [Virus Music, Famitsu Bunko, BitComposer, Victoria University Press, BBC Audio]

Concept Name X: military person
Instance List Y: [Ronald Reid-Daly, Charles Augustus Hilton, 27th Indiana Infantry Regiment, Spartaco Schergat, Charles Corcoran]

[8 in-context demonstrations abbreviated]

Concept Name X: {new concept name}
Instance List Y:

Template 3: Concept Consistency Evaluation Prompt

Prediction sentence: {Generated sentence }
Sentence A:{Target edit sentence}
Sentence B:{Origin sentence}

Check the prediction sentence and Give a score from -1 to 1:
Score 1: close meaning to sentence A
Score 0: neither relevant to A nor B
Score -1: close meaning to sentence B

Output format is: Score:{}
Only output group name and corresponding score, no other explanation.

A.2 Data Distribution of RelEdit

We further report the data distribution of our proposed RelEdit benchmark in Figure 7, calculated with token length. Figure 7a shows that RelEdit maintain a wide range of description token length from 3 to 45, and most of them are around 10. The wide range of concept descriptions increase the difficulty of conceptual editing. From Figure 7b and
<table>
  <tr>
    <th></th>
    <th>Annotator-1</th>
    <th>Annotator-2</th>
    <th>Annotator-3</th>
  </tr>
  <tr>
    <th>Agreement Rate</th>
    <td>95.8</td>
    <td>93.6</td>
    <td>94.4</td>
  </tr>
</table>

Table 4: Human evaluation results on RelEdit

Figure 7c, we can tell that token length of concept name is mainly around 3-4, and token length of instance name is mainly around 6-7, which bring impact to instance-level evaluation criteria.

A.3 Human Evaluation

We recruited three volunteers to evaluate the entailment between the origin concept description and corresponding concept name. Each volunteer assessed a sample of 300 randomly selected examples to ensure the quality and reliability of our dataset. They were tasked with "Determining whether the origin concept description semantically supports the corresponding concept name." The human evaluation results are listed in Table 4. The high agreement observed further supports our dataset’s quality and validity.

A.4 Case of Conceptual Editing Requests

Specifically, we provide a detailed conceptual editing case of RelEdit benchmark in inter setting. As shown in Table 5, the detailed textual information and editing prompts in both instance-level and concept-level elaborate the multi-aspect conceptual knowledge covered in a single editing request.

A.5 SPARQL Protocol and RDF Query Language

SPARQL facilitates the extraction and modification of data that is housed within the Resource Description Framework (RDF), a system adept at representing graph-based data structures. The Wikidata Query Service¹ (WDQS) is an internet-based platform which empowers users to fetch and scrutinize the organized data contained within Wikidata by utilizing SPARQL queries. We employ WDQS to query the description texts for each concept in Section 3.1 Ontology Building, and the SPARQL we used is listed in Table 7.

B MICE

MICE explicitly stores all edited concepts in memory, with each concept represented as a pair of concept name and definition. An off-the-shelf retrieval model (Izacard et al., 2021) is utilized to embed the concept names and save them in a retrieval index. This index takes a query as input and returns the edited concept most relevant to the query (i.e., closest in the embedding space). For step-by-step generation and self-checking with LLMs, we follow the approach of previous works (Zhong et al., 2023). Specifically, the model is prompted to verify whether the retrieved fact contradicts the generated answer. If a subquestion is unrelated to any edited concept in memory (because the corresponding concept was not edited), the model is prompted to retain the generated answer, as the retrieved edit does not cause a contradiction. The final answer is then generated based on this process.

C Experimental Details

C.1 Language Models

To adequately evaluate knowledge editing methods on RelEdit, we use following LLMs as the base model: GPT2-XL (Radford et al., 2019b), GPT-J-6B (Wang and Komatsuzaki, 2021), LLaMA2 (7B, 13B) (Touvron et al., 2023b), Mistral-7B-v0.1 (Jiang et al., 2023), Baichuan2 (7B, 13B) (Baichuan, 2023) and Qwen2 (0.5B, 1.5B, 7B) (Qwe, 2024). All models can be reached in Huggingface website².

C.2 Metrics

The commonly used metrics are from ROME (Meng et al., 2022a).

Reliability (RE). This metric measures the mean accuracy on a specific collection of pre-defined input-output pairs \((x_e, y_e)\):

\[
\mathbb{E}_{x'_e, y'_e \sim \{(x_e, y_e)\}} \mathbb{I} \left\{ \operatorname*{argmax}_y f_{\theta_e}(y \mid x'_e) = y'_e \right\}
\]

Generalization (GE). Paraphrased sentences should be modified accordingly by editing. This metric gauges the average accuracy on equivalent neighbor \(R(x_e, y_e)\):

\[
\mathbb{E}_{x'_e, y'_e \sim R(x_e, y_e)} \mathbb{I} \left\{ \operatorname*{argmax}_y f_{\theta_e}(y \mid x'_e) = y'_e \right\}
\]

Locality (LO). Locality is assessed based on the frequency at which the predictions of the post-edit model remain unchanged in out-scope neighbor \(O(x_e, y_e)\):

\[
\mathbb{E}_{x'_e, y'_e \sim O(x_e, y_e)} \mathbb{I} \left\{ f_{\theta_e}(y \mid x'_e) = f_{\theta}(y \mid x'_e) \right\}
\]

¹https://query.wikidata.org
²https://huggingface.co
Figure 7: Token length distribution of concept descriptions(a), concept names(b) and instance names(c) in RelEdit benchmark for conceptual editing. Texts above are tokenized by the LLaMA-2-7B tokenizer.

C.3 Baselines
We evaluate the following state-of-the-art knowledge editing approaches on our datasets: Fine-tuning (FT) (Zhu et al., 2020), MEND (Mitchell et al., 2021), ROME (Meng et al., 2022a), MEMIT (Meng et al., 2022b) and PROMPT (Wang et al., 2024).

• Fine-tuning (FT) approach uses gradient descent to update model parameters based on the edits. In our study, we adopt the method (Zhu et al., 2020), where we fine-tune one layer with a norm constraint on weight changes.

• MEND (Mitchell et al., 2021) trains a hypernetwork to produce weight updates by transforming the raw fine-tuning gradients.

• ROME (Meng et al., 2022a) first identifying the knowledge within a specific layer of the Transformer architecture. It then updates the feedforward network within that layer.

• MEMIT (Meng et al., 2022b) is an extension of ROME that allows for batch editing. It updates the feedforward networks across multiple layers to encode all the edited facts.

• PROMPT (Wang et al., 2024) serves as the prefix is concatenated to the beginning of the evaluation input to elicit the desired modification in outputs.

C.4 Implementation Details
For the code implementation, we adopt EasyEdit\footnote{https://github.com/zjunlp/EasyEdit} repository to reimplement all baselines including FT (FT-L implementation), ROME, MEND, MEMIT and PROMPT. The editing procedure is conducted in a independent manner, focusing exclusively on a single specified concept with each edit instead of sequentially, ensuring that each modification is made in isolation. Once the evaluation for a given sample is finished, the LLM is reverted to its original state, prior to any edits being applied. This approach guarantees that no subsequent edits are influenced by the previous ones.

For the settings of all baselines, we adopt their default configurations. All experiments are carried out on a single A800 GPU. To be specific, we list important hyper-parameters of baselines as follows:

FT: FT executes a gradient descent operation on the modifications to refine the model parameters. Following Zhu et al. (2021), FT uses a norm constraint on weight changes with a coefficient \(5 \times 10^{-5}\) in our implementation. We finetune layer 0 of GPT2-xl, layer 21 of GPT-J-6B, LLaMA-2-7B and Mistral-7B-v0.1.

MEND: Since MEND requires a trained hypernetwork to update parameters, we construct an extra dataset with train/validation/test sets to train a hypernetwork specifical to conceptual editing task. During training, we set the max iteration step step 10000. During inference, the learning rate scale is set to be 1.0.

ROME: We use the default hyper-parameters of ROME and the pre-computed covariance statistics released by Meng et al. (2022a). The edit layer in GPT2-xl is set to 17, in GPT-J-6B, LLaMA-2-7B, Mistral-7B-v0.1 and Baichuan2-7B is set to 5.

MEMIT: To apply MEMIT on all LLM backbones, we compute the covariance statistics for Vicuna-7B on Wikitext using a sample size of 100,000. For GPT2-xl, we updated model weights at layer [13, 14, 15, 16, 17]. For GPT-J-6B, we updated model weights at layer {3, 4, 5, 6, 7, 8}. For LLaMA-2-7B, Mistral-7B-v0.1 and Baichuan2-7B, we updated model weights at layer {4, 5, 6, 7, 8}.
<table>
  <tr>
    <th colspan="2">CONCEPT INFOMATION</th>
  </tr>
  <tr>
    <th>Subject Concept</th>
    <td><span style="color:blue;">Island</span></td>
  </tr>
  <tr>
    <th>Subject Superclass</th>
    <td>Place</td>
  </tr>
  <tr>
    <th>Subject Description</th>
    <td>Piece of sub-continental land completely surrounded by water</td>
  </tr>
  <tr>
    <th>Subject Instances</th>
    <td>Saba Island, United States Virgin Islands<br>Liard Island<br>Shaw Islan</td>
  </tr>
  <tr>
    <th>Target Concept</th>
    <td><span style="color:red;">American football player</span></td>
  </tr>
  <tr>
    <th>Target Superclass</th>
    <td>Species</td>
  </tr>
  <tr>
    <th>Target Description</th>
    <td>Athlete who plays American football</td>
  </tr>
  <tr>
    <th>Target Instances</th>
    <td>Adam Bisnowaty<br>Marvin Hall<br>Siran Stacy</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">CONCEPTUAL EDITING PROMPTS</th>
  </tr>
  <tr>
    <th>Base Editing</th>
    <td>The definition of <span style="color:blue;">Island</span> is <span style="color:red;">athlete who plays American football</span></td>
  </tr>
  <tr>
    <th>Phrase</th>
    <td>To put it simply, <span style="color:blue;">island</span> refers to <span style="color:red;">athlete who plays American football</span></td>
  </tr>
  <tr>
    <th>Locality</th>
    <td>The definition of galaxy is large gravitationally bound system of stars and interstellar matter</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">INSTANCE-LEVEL EDITING PROMPTS</th>
  </tr>
  <tr>
    <th>Instance Change</th>
    <td>Whether <span style="color:blue;">Saba Island, United States Virgin Islands</span> belongs to category <span style="color:blue;">island</span>?</td>
  </tr>
  <tr>
    <th>Portability</th>
    <td>Whether Adam Bisnowaty belongs to category <span style="color:blue;">island</span>?</td>
  </tr>
  <tr>
    <th>Instance Locality</th>
    <td>Whether Salyut 6 belong to category space station?</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">CONCEPT-LEVEL EDITING PROMPTS</th>
  </tr>
  <tr>
    <th>Alignment Compare</th>
    <td>Do <span style="color:blue;">island</span> and <span style="color:red;">american football player</span> belong to the same superclass?</td>
  </tr>
  <tr>
    <th>Alignment Belong</th>
    <td>Whether <span style="color:blue;">island</span> belongs to superclass <span style="color:red;">species</span>?</td>
  </tr>
</table>

Table 5: An example of conceptual editing requests in RelEdit (under inter setting). The subject conceptual knowledge and target conceptual knowledge required in a single editing procedure are highlighted in blue and red, respectively. Due to the limitation of space, some instances of concept are omitted.

PROMPT: We employ In-context Learning to prompt LLMs to elicit the desired modification. The detailed prompt we use during editing is to include a definition about new concept before exact editing request, which is formulated as "Definition of [subject concept]: [target description]".

D Additional Experiments

D.1 More results about concept consistency

Additional to Section 6.3 Analysis Q2, we provide more results evaluated on concept consistency of Mistral-7B with ROME, MEMIT and PROMPT using GPT-4 API, as shown in Fig 8.

D.2 Impact of LLM Size on Conceptual Editing

We analyze how conceptual editing performance on RelEdit is affected by the LLM size. Specifically, we conduct conceptual editing on LLaMA2, Baichuan2 and Qwen2 with different amount of parameters, and detailed results are shown in Table 6. From this table, we can observe that: the editing performance generally improves with the increase in the number of LLM parameters, especially on factual criteria. LLMs with larger amount parameters tend to perform better on instance-level criteria, but facing a performance drop on concept-level metric like AB. These results indicate that parameter volume of LLMs positively affects the knowledge editing performance in the factual level
cases of ROME method in varying degrees of editing effectiveness. Case 1 is an ideally successful editing, as the generated sentence from post-edit model is the same as editing target in token-level. Case 2 reflects a partial successful editing, since there is an overlap between target sentence and output sentence. Case 3 shows an unsuccessful editing on concept cyclist, because the output sentence from post-edit model exactly inherited the origin conceptual knowledge.

Figure 8: The results of concept consistency on Mistral-7B with ROME, MEMIT and PROMPT in both intra and inter settings. Target means the generated sentence has close semantics to target, vice versa to Origin. Ambiguity means neither similar to both.

to a certain extent. However, they also face the challenge of updating knowledge in the ontological level, as models with a larger number of parameters involve more types of relational reasoning challenges, making it more difficult to thoroughly update ontological knowledge from the root.

D.3 Conceptual Patterns across Editing Methods

According to the main results in Table 3, we further conduct a thorough analysis on conceptual patterns that different methods captured during editing procedure. From Figure 9, we can observe that among five editing methods, PROMPT yields leading performance on instance-level and concept-level criteria like IC, PO, IL, AB and AC. On the contrary, locate-and-edit methods like ROME and MEMIT achieve strong performance on metrics Re, Ge and Lo.

D.4 Case of Editing Results

Table 8 presents several editing results with ROME on RelEdit benchmark in inter setting, listing three
<table>
  <tr>
    <th rowspan="2">Base Model</th>
    <th rowspan="2">Method</th>
    <th colspan="6">Intra Setting</th>
    <th colspan="6">Inter Setting</th>
  </tr>
  <tr>
    <th>Re</th>
    <th>Ge</th>
    <th>Lo</th>
    <th>IC</th>
    <th>PO</th>
    <th>IL</th>
    <th>AB</th>
    <th>Re</th>
    <th>Ge</th>
    <th>Lo</th>
    <th>IC</th>
    <th>PO</th>
    <th>IL</th>
    <th>AB</th>
    <th>AC</th>
  </tr>
  <tr>
    <td>LLaMA-2-7B</td>
    <td>PROMPT</td>
    <td><b>89.35</b></td>
    <td><b>87.28</b></td>
    <td>76.84</td>
    <td>6.86</td>
    <td>38.50</td>
    <td><b>89.82</b></td>
    <td><b>84.51</b></td>
    <td><b>88.88</b></td>
    <td>87.89</td>
    <td>78.06</td>
    <td>6.19</td>
    <td>30.31</td>
    <td><b>85.84</b></td>
    <td><b>79.42</b></td>
    <td><b>93.36</b></td>
  </tr>
  <tr>
    <td>LLaMA-2-13B</td>
    <td>PROMPT</td>
    <td>88.43</td>
    <td>87.08</td>
    <td><b>81.13</b></td>
    <td><b>8.19</b></td>
    <td><b>41.81</b></td>
    <td>84.29</td>
    <td>58.63</td>
    <td>88.57</td>
    <td><b>87.93</b></td>
    <td><b>81.58</b></td>
    <td><b>7.74</b></td>
    <td><b>36.73</b></td>
    <td><b>85.84</b></td>
    <td>49.56</td>
    <td>80.09</td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>PROMPT</td>
    <td><b>89.41</b></td>
    <td>87.89</td>
    <td>77.36</td>
    <td>4.87</td>
    <td><b>72.79</b></td>
    <td><b>90.71</b></td>
    <td><b>96.90</b></td>
    <td><b>89.05</b></td>
    <td>87.70</td>
    <td>78.08</td>
    <td>4.87</td>
    <td><b>70.80</b></td>
    <td><b>93.14</b></td>
    <td><b>90.71</b></td>
    <td><b>98.23</b></td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>PROMPT</td>
    <td>89.17</td>
    <td><b>88.08</b></td>
    <td><b>80.58</b></td>
    <td><b>34.29</b></td>
    <td>30.97</td>
    <td>65.27</td>
    <td>37.61</td>
    <td>88.41</td>
    <td><b>87.85</b></td>
    <td><b>80.64</b></td>
    <td><b>35.62</b></td>
    <td>24.78</td>
    <td>68.36</td>
    <td>28.10</td>
    <td>73.01</td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>PROMPT</td>
    <td>88.32</td>
    <td>86.69</td>
    <td>72.24</td>
    <td>1.11</td>
    <td><b>96.02</b></td>
    <td>58.63</td>
    <td><b>95.13</b></td>
    <td>88.48</td>
    <td>86.67</td>
    <td>73.08</td>
    <td>0.88</td>
    <td><b>95.58</b></td>
    <td>56.86</td>
    <td><b>94.03</b></td>
    <td>46.68</td>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>PROMPT</td>
    <td>88.62</td>
    <td>86.56</td>
    <td>78.61</td>
    <td><b>5.75</b></td>
    <td>63.72</td>
    <td>92.92</td>
    <td>90.71</td>
    <td>88.30</td>
    <td>87.12</td>
    <td>79.43</td>
    <td><b>8.41</b></td>
    <td>56.42</td>
    <td>91.15</td>
    <td>81.42</td>
    <td><b>92.04</b></td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>PROMPT</td>
    <td><b>88.85</b></td>
    <td><b>87.56</b></td>
    <td><b>80.53</b></td>
    <td>1.33</td>
    <td>68.36</td>
    <td><b>94.03</b></td>
    <td>56.19</td>
    <td><b>88.89</b></td>
    <td><b>87.67</b></td>
    <td><b>80.83</b></td>
    <td>1.11</td>
    <td>58.63</td>
    <td><b>93.36</b></td>
    <td>40.49</td>
    <td>79.87</td>
  </tr>
  <tr>
    <td>Qwen2-0.5B</td>
    <td>ROME</td>
    <td>98.95</td>
    <td>72.84</td>
    <td>85.32</td>
    <td>30.31</td>
    <td>55.09</td>
    <td>98.01</td>
    <td>22.79</td>
    <td>99.02</td>
    <td>71.03</td>
    <td>85.84</td>
    <td>33.41</td>
    <td>51.33</td>
    <td>98.23</td>
    <td>20.80</td>
    <td>6.86</td>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>ROME</td>
    <td>99.07</td>
    <td><b>80.77</b></td>
    <td><b>92.64</b></td>
    <td><b>21.46</b></td>
    <td><b>61.95</b></td>
    <td><b>98.67</b></td>
    <td><b>73.89</b></td>
    <td>99.65</td>
    <td><b>81.03</b></td>
    <td><b>92.81</b></td>
    <td><b>32.30</b></td>
    <td><b>57.52</b></td>
    <td><b>98.89</b></td>
    <td><b>70.13</b></td>
    <td><b>30.97</b></td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>ROME</td>
    <td><b>99.85</b></td>
    <td>75.34</td>
    <td>92.41</td>
    <td>18.81</td>
    <td>49.56</td>
    <td>98.45</td>
    <td>59.96</td>
    <td><b>99.88</b></td>
    <td>73.84</td>
    <td>92.60</td>
    <td>22.79</td>
    <td>45.58</td>
    <td><b>99.34</b></td>
    <td>53.76</td>
    <td>60.84</td>
  </tr>
</table>

Table 6: Main results of PROMPT and ROME across base models LLaMA-2, Baichuan2 and Qwen2 with varying number of parameters. The best results are in bold. Re, Ge and Lo are the abbreviation of metric Reliability, Generalization and Locality.

![Conceptual editing performance of FT, MEND, ROME, MEMIT and PROMPT on backbones LLaMA-2-7B(a) and Mistral-7B-v0.1(b). The results reported are averaged between intra and inter setting.](page_1012_1342_410_312.png)

Figure 9: Conceptual editing performance of FT, MEND, ROME, MEMIT and PROMPT on backbones LLaMA-2-7B(a) and Mistral-7B-v0.1(b). The results reported are averaged between intra and inter setting.
SPARQL for Extracting Concept Description

PREFIX bd: <http://www.bigdata.com/rdf#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX schema: <http://schema.org/>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wikibase: <http://wikiba.se/ontology#>

SELECT ?conceptLabel ?conceptDesc
WHERE {
    SERVICE wikibase:label {
        bd:serviceParam wikibase:language "en" .
        wd:<QID> rdfs:label ?conceptLabel .
        wd:<QID> schema:description ?conceptDesc .
    }
}

Table 7: SPARQL Query for extracting concept description based on concept entity id (denoted by "<QID>").
<table>
  <tr>
    <th colspan="2">CASE 1</th>
  </tr>
  <tr>
    <th>Concept</th>
    <td>noble</td>
  </tr>
  <tr>
    <th>[Origin]</th>
    <td>member of the nobility</td>
  </tr>
  <tr>
    <th>[Target]</th>
    <td>group of players organized to compete as a side in baseball</td>
  </tr>
  <tr>
    <th>[Generated]</th>
    <td style="background-color:#b3e5b3;">group of players organized to compete as a side in baseball</td>
  </tr>
  <tr>
    <th>Rewrite Accuracy</th>
    <td style="background-color:#b3e5b3;">1.000</td>
  </tr>
  <tr>
    <th colspan="2">CASE 2</th>
  </tr>
  <tr>
    <th>Concept</th>
    <td>continent</td>
  </tr>
  <tr>
    <th>[Origin]</th>
    <td>large landmass identified by convention</td>
  </tr>
  <tr>
    <th>[Target]</th>
    <td>an athlete who participates in traditional Irish sports such as Gaelic football, hurling, camogie, or handball</td>
  </tr>
  <tr>
    <th>[Generated]</th>
    <td style="background-color:#b3e5b3;">a person who participates in a sport that involves a combination of rugby, rugby league, or American football</td>
  </tr>
  <tr>
    <th>Rewrite Confidence</th>
    <td style="background-color:#b3e5b3;">0.7826</td>
  </tr>
  <tr>
    <th colspan="2">CASE 3</th>
  </tr>
  <tr>
    <th>Concept</th>
    <td>cyclist</td>
  </tr>
  <tr>
    <th>[Origin]</th>
    <td>person who rides a bike</td>
  </tr>
  <tr>
    <th>[Target]</th>
    <td>set of episodes produced for a television series</td>
  </tr>
  <tr>
    <th>[Generated]</th>
    <td style="background-color:#ffcccc;">a person who rides a bicycle</td>
  </tr>
  <tr>
    <th>Rewrite Accuracy</th>
    <td style="background-color:#ffcccc;">0.375</td>
  </tr>
</table>

Table 8: Examples of conceptual editing results with ROME in inter setting. [Origin] refers to the initial concept recognition before editing, [Target] denotes the ideal concept description after editing and [Generated] represents the generated sentence from post-edit model.

Reviewer paper summaries: This paper proposes a lightweight conceptual knowledge editing method MICE based on external memory that does not require parameter updates, as well as a new evaluation benchmark, RelEdit. Unlike previous work, which only focused on whether the model successfully memorizes concept definitions, the authors’ method is more flexible. It focuses on the impact of knowledge editing on the overall knowledge structure of the model. 

In terms of method design, the author starts by building an ontology, combining the update of neighbor relationships with the effective filtering of instances, so that the edited conceptual knowledge can be more naturally and effectively integrated into the model. 

In terms of evaluation, RelEdit systematically designs indicators of multiple dimensions at the instance level and the concept level, which can more comprehensively and fairly evaluate the chain effect after knowledge editing. Experimental results show that the proposed MICE method outperforms existing editing methods in mainstream LLMs and multiple evaluation indicators, showing good performance and the application potential of non-invasive methods in knowledge editing.

This paper proposes a novel benchmark called RelEdit, which addresses the challenge of relational reasoning in concept-level editing tasks. It provides detailed descriptions of data construction and evaluation metrics. The paper also introduces a memory-based approach called MICE, inspired by MeLLo, as a baseline. Experimental results include extensive comparisons with existing editing methods, including the performance of MICE.
Summary of strengths: Lightweight knowledge editing method: The MICE method is combined with external storage and does not require parameter updates.

Rigorous data construction process: Structured data is constructed based on the existing knowledge graph DBpedia, and instance filtering and quality control are performed in a combination of automatic and manual methods to ensure that the edited knowledge matches the existing knowledge distribution of the model, providing a guarantee for high-quality evaluation.

Comprehensive evaluation benchmarks: A variety of evaluation benchmarks are designed at the conceptual and instance levels, including Instance Change (IC), Portability (PO), Alignment Compare (AC), etc., which provide a comprehensive and accurate evaluation of the chain effect of knowledge editing.

Comprehensive experiments and strong generalization of methods: MICE is compared with major editing methods on multiple mainstream LLMs. The results show that MICE performs well in multiple indicators, demonstrating the versatility of the method.

1. The proposed benchmark is well-motivated for addressing concept-level editing, and the evaluation metric is both thoughtfully designed and novel.
2. The paper presents an extensive evaluation comparing a range of editing methods, which contributes meaningfully to the literature.
Summary of weaknesses: The current framework does not consider polysemous concepts and edited knowledge directly overwrites the original definitions. This limits the applicability of knowledge editing methods in scenarios requiring nuanced or context-sensitive understanding.

MICE relies on external memory to perform knowledge updates, which, while efficient, restricts the model’s ability to retain edited knowledge in the long term without explicit memory retrieval.

The reasoning capability of MICE may be limited, particularly in tasks that involve complex hierarchical or relational reasoning.

The method is only evaluated on the proposed RelEdit benchmark, without testing on existing datasets, raising concerns about its generalization ability across domains and tasks.

The editing task granularity is relatively coarse, focusing primarily on whole-definition replacement. The framework lacks the flexibility to edit or assess fine-grained concept attributes.

The overall evaluation pipeline is complex and potentially costly.

1. Concept-level editing is evaluated and processed separately from other editing tasks, without a unified framework. Given an edit request, it is unclear when concept-level versus instance-level editing should be applied. There is a need for designing a unified editing task framework that can handle both cases cohesively.
2. This work appears to use definition-based (or description-based) targets for editing. However, one could argue that most editing problems—even those involving entities or attributes—can be framed as modifications to their descriptions. Adopting description-based targets could offer a more general framework that encompasses both concept-level and instance-level editing. Further discussion is needed to compare the proposed tasks to a broader description-level editing perspective and to explore potential extensions in that direction.
3. In the experiments, Mistral-7B shows significantly lower performance in terms of the AB metric for both intra-setting and inter-setting evaluations, compared to other editing methods—despite performing comparably on other metrics. This discrepancy warrants further discussion and explanation.
Comments, suggestions and typos: The knowledge editing and evaluation methods proposed in this article are useful for reference, and the experiments are comprehensive. However, there is insufficient performance testing of methods on different datasets and the evaluation costs can be high. In addition, it is also possible to introduce considerations for polysemous concepts and to link knowledge editing methods to practical problems.

Typos:
Page 1, 35: delete ‘s’  
Page 1, 38: delete ‘on’  
Page 4, 218: delete ‘s’  
Page 5, 329, 342: delete one ‘the’  
Page 6, 421: are better -> are better at  
Page 7, 447: dour -> four  
Page 12, 836: numerates -> enumerate

Please see weaknesses.
Overall assessment scores: 2

4
Metareview score: 3.5
Metareview: This paper introduces RelEdit, a benchmark for evaluating conceptual knowledge editing in language models with a focus on relational consistency. It also proposes MICE, a lightweight, memory-based editing method that avoids parameter updates and achieves strong performance across multiple metrics, highlighting the potential of non-invasive editing approaches.
Summary of reasons to publish: + Relational reasoning focus: The paper highlights an under-explored but important aspect of conceptual knowledge editing—relational consistency and reasoning—beyond simple factual updates.
+ Non-invasive editing: MICE provides an effective, non-parametric editing approach using external memory, allowing efficient updates without retraining.
+ High-quality benchmark design: RelEdit is carefully constructed using structured data from DBpedia, with rigorous instance filtering and multi-dimensional evaluation metrics at both concept and instance levels.
+ Strong empirical validation: The method is extensively evaluated across multiple LLMs and editing techniques, demonstrating strong performance and generalizability.