MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality
Authors: Shuaike Li, Kai Zhang, Qi Liu, Enhong Chen
Accepted at: acl25

MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality

Shuaike Li, Kai Zhang*, Qi Liu, Enhong Chen
State Key Laboratory of Cognitive Intelligence,
University of Science and Technology of China, Hefei, China
lishuaike767@gmail.com
{kkzhang08, qiliuql, cheneh}@ustc.edu.cn

Abstract

Knowledge editing is a technique for efficiently and accurately updating the knowledge of large language models (LLMs) to alleviate obsolescence and correct errors. However, most existing methods overfit to specific models, causing edited knowledge to be discarded during each LLM update and requiring frequent re-editing, which is particularly burdensome in today’s rapidly evolving open-source community. To address this issue, we propose the problem of cross-model knowledge editing and introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models. MindBridge introduces the novel concept of memory modality, which encodes edited knowledge as an independent modality. It first performs LLM-agnostic pre-training of the memory modality and then integrates it with various LLMs. Extensive experiments on multiple LLMs and popular knowledge editing datasets demonstrate that MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs. Our code is available at here.

1 Introduction

Large language models (LLMs) have revolutionized natural language processing, demonstrating remarkable abilities in understanding and generation (Achiam et al., 2023; Touvron et al., 2023; Brown et al., 2020). These models leverage the knowledge acquired during pre-training to answer user queries. However, since the knowledge embedded in LLMs is stored in static parameters, their internal knowledge needs to be updated to keep pace with the ever-changing world and avoid obsolescence. Additionally, for personalized user information or domain-specific knowledge, customizing LLM output also requires updating the model’s knowledge. Traditional methods, such as fine-tuning, continual learning, or retraining, are computationally expensive and may inevitably degrade the model’s general capabilities(Kalajdzievski, 2024; Wang et al., 2023). Fortunately, recent advancements in knowledge editing offer a promising solution, enabling efficient and precise modification of model knowledge at a low computational cost.

Existing knowledge editing methods generally fall into two categories (Yao et al., 2023). The first one preserves LLM parameters, such as memory-based methods (Mitchell et al., 2022a; Madaan et al., 2022a) or methods that add extra parameters to the model (Huang et al., 2023; Dong et al., 2022). The second category modifies the parameters of the edited model, such as meta-learning-based methods (Mitchell et al., 2021; Tan et al., 2023) or locate-and-edit methods (Meng et al., 2022a; Fang et al., 2024). While these methods have made significant

![The cross-model knowledge editing problem and analogizing memory modality to visual modality](page_1012_355_393_246.png)

Figure 1: (a) The cross-model knowledge editing problem. Current knowledge editing methods discard previously edited knowledge after every LLM update (e.g., when the base model is updated alongside LLaMA), requiring frequent re-editing, which is labor-intensive. This motivates us to explore whether edited knowledge can transcend individual models, i.e., achieve cross-model knowledge editing. (b) Analogizing memory modality to visual modality. Different visual modality encoders exhibit low coupling with various LLM backbones, and after efficient modality alignment, they enable LLMs to see. Inspired by this, we propose memory modality, which decouples knowledge from a single model, allowing knowledge editing through modality bridging with different LLMs.
progress, most still face the issue of overfitting to a single LLM. If the knowledge to be edited is domain- or user-specific, re-editing this knowledge becomes necessary whenever the LLM is updated, which is both tedious and time-consuming, especially given the fast pace of updates in the open-source community. This motivates us to explore whether the edited knowledge can be loosely coupled with the target model and is no longer limited to a single LLM. We call this problem cross-model knowledge editing, as shown in Figure 1(a).

To address this issue, inspired by the work on multimodal large language models, we introduce the memory modality, a novel concept that encodes edited knowledge as a standalone modality to partially decouple a model’s knowledge from the LLM itself. As illustrated in Figure 1(b), the analogy to visual modality offers a clearer understanding of memory modality. The current mainstream paradigm (Liu et al., 2024) involves concatenating pre-trained visual modality encoders (e.g., ViT (Dosovitskiy, 2020), CLIP ViT (Radford et al., 2021)) with various LLM backbones (e.g., LLaMA, Vicuna (Zheng et al., 2023b)) to enable LLMs with visual perception. The low-coupling characteristic between visual modality processing and LLMs allows for independent updates of both components (Chen et al., 2024; Jain et al., 2024). This low-coupling property is precisely what is needed to solve the problem of cross-model knowledge editing. By encoding editable knowledge as a separate memory modality, pre-training a memory modality encoder to handle this knowledge, and subsequently concatenating it with multiple LLMs, we achieve efficient cross-model knowledge editing.

Based on the idea of memory modality, we propose MindBridge, a two-stage solution for cross-model knowledge editing. The first stage is memory modality pre-training, where we introduce three training objectives: memory injection, memory association, and memory existence. These objectives are designed to train the memory modality encoder so that it can acquire relevant memories, perform memory association, and determine whether certain memories exist. The second stage is memory modality bridging, where we fine-tune a simple projector to achieve efficient cross-modal alignment, enabling the output of the memory modality to be understood and utilized by LLMs. This approach allows for efficient large-scale knowledge updates, with minimal modifications required when the LLMs are updated.

To test the effectiveness of MindBridge, we conduct extensive experiments on widely used knowledge editing datasets, ZsRE and Counterfact, using multiple LLMs, including GPT2-XL, GPT-J, and LLaMA3. Leveraging the low coupling characteristic between the memory modality and LLMs, MindBridge achieves efficient and scalable knowledge editing across different models, delivering excellent editing performance even in scenarios involving tens of thousands of knowledge edits. Our contributions can be summarized as follows:

• We introduce the novel problem of cross-model knowledge editing, addressing the critical challenge of discarding previously edited knowledge and repeatedly re-editing caused by the rapid iteration of LLMs.

• We propose MindBridge, a scalable and effective solution that leverages the innovative concept of a memory modality to achieve cross-model knowledge editing.

• Extensive experiments on knowledge editing datasets and multiple LLMs validate the effectiveness and scalability of MindBridge for cross-model knowledge editing, even when handling tens of thousands of edits.

2 Related Work

2.1 Knowledge Editing

Knowledge editing, aimed at inserting new knowledge or modifying existing knowledge in LLMs to alter their behavior, falls into two main paradigms (Yao et al., 2023). Parameter-modifying editing directly adjusts model parameters. This includes locate-then-edit strategies like ROME (Meng et al., 2022a), MEMIT (Meng et al., 2022b), and Alphaedit (Fang et al., 2024), which pinpoint relevant knowledge before fine-tuning parameters. Meta-learning approaches, such as KE (De Cao et al., 2021), MEND (Mitchell et al., 2021), and MALMEN (Tan et al., 2023), also belong to this paradigm, training hypernetworks to generate parameter updates.

In contrast, parameter-preserving editing maintains original model weights and uses external components for knowledge storage. T-Patcher (Huang et al., 2023) adds neurons to the last feed-forward layer to adjust output. SERAC (Mitchell et al., 2022b) trains a counterfactual model using a classifier to determine response relevance. Database-retrieval methods, including GRACE (Hartvigsen et al., 2024) (hidden vector database) and MELO
(LoRA block database), modify LLM computation based on retrieved knowledge. MemPrompt (Madaan et al., 2022b) and IKE (Zheng et al., 2023a) utilize retrieved demonstrations as prompts, leveraging in-context learning.

While effective for targeted edits, existing knowledge editing methods are often overfit to specific LLMs and struggle with cross-model, large-scale knowledge editing. After each iteration of the LLM, the previously edited knowledge is lost, leading to the need for frequent re-editing.

2.2 MultiModal Large Language Models

Recent advancements in multimodal large language models (MM-LMMs) have enabled LLMs to process diverse modality inputs like images, video, and audio (Li et al., 2023a,b; Chu et al., 2023). Due to the high cost of training MM-LLMs from scratch, a more efficient strategy involves integrating pre-trained unimodal foundation models with the LLM backbone. This approach typically follows a two-stage pipeline: Multimodal Pre-Training, which aligns modality encoder features with the LLM, and Multimodal Instruction-Tuning, which ensures instruction following and zero-shot generalization (Zhang et al., 2024a; Liu et al., 2024).

This decoupling of the LLM backbone from modality encoders enables the reuse or iterative updates of modality encoders without affecting the LLM (Chen et al., 2024; Oquab et al., 2023), offering insights for cross-model knowledge editing. Beyond traditional modalities, we propose the novel concept of memory modality to achieve similar decoupling, partially separating model knowledge and reasoning. This enables efficient knowledge editing, allowing for the retention or independent updating of previously edited knowledge even when the LLM backbone is updated.

3 Problem Formulation

The goal of knowledge editing is to modify the knowledge stored in a model. In this paper, we focus specifically on editing memories composed of factual knowledge. More concretely, for a triplet \((s, r, o)\) consisting of a subject \(s\), relation \(r\), and object \(o\) (e.g., \(s =\) United States, \(r =\) President, \(o =\) Biden), we aim to insert a new triplet \((s, r, o^*)\) (e.g., \(s =\) United States, \(r =\) President, \(o^* =\) Trump) into the LLM to replace the previous knowledge, i.e., \((s, r, o) \rightarrow (s, r, o^*)\), where these two triplets share the same subject and relation. Specifically, \(o^*\) can also represent knowledge that does not originally exist in the LLM, i.e., \((s, r, \emptyset) \rightarrow (s, r, o^*)\).

Given a set of knowledge to be edited \(D_{edit} = \{(s_i, r_i, o_i^*) \mid i = 1, 2, \ldots, n\}\), a knowledge editing operation \(KE\), and a model to be edited \(F\), the goal of knowledge editing is to generate a new model \(F^*\) through the knowledge editing operation \(KE\). This can be formulated as follows:

\[
F^* = KE(F, D_{edit}),
\]
\[
\text{s.t. } F^*(s, r) = \begin{cases}
o^*, & \text{if } (s, r, o^*) \in \mathcal{I}(D_{edit}), \\
F(s, r), & \text{if } (s, r, o^*) \in \mathcal{O}(D_{edit}).
\end{cases}
\] (1)

Here, \(\mathcal{I}(D_{edit})\) denotes the knowledge set that requires editing and its neighborhood, such as paraphrasing and rewriting, with \(D_{edit} \subseteq \mathcal{I}(D_{edit})\). Meanwhile, \(\mathcal{O}(D_{edit})\) denotes the set of knowledge items unrelated to the edited knowledge. For simplicity, we will refer to these sets as \(\mathcal{I}\) and \(\mathcal{O}\) moving forward. In some studies, \(\mathcal{I}\) is also referred to as in-scope examples, while \(\mathcal{O}\) is termed out-of-scope examples (Mitchell et al., 2022b).

Equation 1 specifies that the edited model \(F^*\) should correctly predict the edited knowledge and its neighborhood. For inputs unrelated to the edited knowledge, \(F^*\) should maintain consistent predictions with the original model \(F\). This ensures that knowledge editing updates target knowledge while minimizing interference with unrelated knowledge.

4 Method

As illustrated in Fig. 2, MindBridge is meticulously structured into two primary stages. The first stage, Memory Modality Pre-training (detailed in Section 4.1), is specifically designed to obtain the memory modality encoder, \(E_m\). The second stage, Memory Modality Bridging (detailed in Section 4.2), is dedicated to training a projector to construct the memory-to-language bridging module, \(P_m\). Ultimately, the edited model, \(F^*\), can be mathematically represented by Eq. 2. In MindBridge, knowledge is encoded into memory modality features, denoted as \(x_{memory}\), through the synergistic utilization of the memory modality encoder \(E_m\) and the bridging module \(P_m\). These features are subsequently concatenated with the textual input \((s, r)\) as soft prompts and provided to the LLM, \(F\), for prediction.

\[
x_{memory} = P_m(E_m(s, r)),
\]
\[
F^*(s, r) = F(x_{memory} \oplus (s, r)).
\] (2)
Figure 2: Overview of MindBridge. Given a massive collection of fact knowledge subject-relation-object triplets \((s, r, o^*)\) intended for editing, we first perform stage I: Memory Modality Pre-training. This phase utilizes three training objectives – memory injection, memory association, and memory existence – to develop a memory modality encoder capable of retaining relevant memories, making associations and determining whether specific memories exist. In stage II: Memory Modality Bridging, we then train a projector to bridge the memory modality encoder with LLMs, allowing these models to obtain and effectively leverage the relevant memories.

To fully appreciate the design of MindBridge, we first review the general architecture of current mainstream multimodal large language models (MM-LLMs) (Liu et al., 2024; Bai et al., 2023). These models typically consist of three shared design elements: a modality encoder \(E\), a modality-to-language alignment module \(P\), and a frozen LLM \(F\). The final constructed MM-LLMs resemble the edited model presented in Eq.2. During inference, MM-LLMs rely on the modality encoder \(E\) and the bridging module \(P\) to extract modality features and transform them into a text feature space understandable by LLMs. Drawing inspiration from this architecture’s successful design, MindBridge adopts a similar architecture to implement the training and bridging of memory modalities. Consequently, the missing components in MindBridge are the modality encoder and the modality-to-language alignment module, which are addressed in its two stages respectively.

4.1 Memory Modality Pre-training

In multimodal large language models, modality encoders are typically pre-trained in advance to extract features from modality-specific inputs, such as the commonly used vision modality encoders ViT (Dosovitskiy, 2020) and CLIP-ViT (Radford et al., 2021). Analogously, for the memory modality, pre-training a dedicated encoder, \(E_m\), is indispensable. From the perspective of the knowledge editing workflow, this encoder should be designed to achieve the following three key objectives:

• Objective 1: Possess relevant memories for target knowledge, i.e., store tuples of \((s, r, o^*) \in D_{edit}\).

• Objective 2: Extract relevant memories based on provided context, i.e., retrieve memories associated with \((o^*)\) from the given \((s, r)\), where \((s, r, o^*) \in D_{edit}\).

• Objective 3: Distinguish whether relevant memories exist, i.e., to differentiate between the set \(\mathcal{I}\) and the set \(\mathcal{O}\).

We initialize \(E_m\) using a pre-trained BERT (Kenton and Toutanova, 2019) and use the [CLS] hidden state of its output as the extracted memory modality features, i.e., \(E_m(s, r) = \mathrm{BERT}_{[CLS]}(s, r)\). To achieve Objective 1, we adopt the Masked Language Model (MLM) training objective to inject the knowledge \((s, r, o^*) \in D_{edit}\) into \(E_m\). Specifically, we randomly mask one element of the \((s, r, o^*)\) triplet by replacing it with the [MASK] token and then have \(E_m\) reconstruct the masked element. We refer to this loss function as the memory-injection loss \(L_{inject}\), which can be expressed as follows:
\[
L_{\text{inject}} = -E_{(s,r,o^*) \sim D_{\text{edit}}} \left[ \sum_{x_i \in \{s, r, o^*\}} \log \mathbb{P}_{E_m}(x_i \mid [\text{MASK}], (s, r, o^*) \setminus x_i) \right].
\]

To achieve Objective 2, we need to enhance the memory modality features extracted from the [CLS] token so that they contain representations related to \( o^* \) based on \( (s, r) \). This process mirrors human associative memory, where observing a segment of text triggers recall of associated information. To adapt the memory modality encoder for this task, we feed the [CLS] representation output by \( E_m(s, r) \) along with \( (s, r, [\text{MASK}]) \) into another dummy model \( M \). The model \( M \) must predict \( o^* \) based on the associations made by the memory modality given \( (s, r) \). In this case, \( M \) is also a BERT model, sharing the same pre-trained initialization parameters with \( E_m \), but its parameters remain frozen throughout the training process. We refer to this loss function as the memory-association loss \( L_{\text{associate}} \), which can be formulated as follows:

\[
L_{\text{associate}} = -E_{(s,r,o^*) \sim D_{\text{edit}}} \left[ \log \mathbb{P}_M(o^* \mid E_m(s, r) \oplus (s, r, [\text{MASK}])) \right].
\]

To achieve Objective 3, which involves determining whether relevant memories exist, we need to ensure that \( E_m(s, r) \) produces distinct representations for the sets \( \mathcal{I} \) and \( \mathcal{O} \). We adopt a simple binary classification task to accomplish this goal. Specifically, we feed \( E_m(s, r) \), where \( (s, r, o^*) \in \mathcal{I} \cup \mathcal{O} \), into a classification head \( H \) composed of two linear layers to classify whether the input belongs to \( \mathcal{I} \) or \( \mathcal{O} \). This approach ensures that the internal representations of \( \mathcal{I} \) and \( \mathcal{O} \) are clearly differentiated and reside in distinct vector spaces. We refer to this loss function as the memory-existence loss \( L_{\text{exist}} \), which can be expressed as follows:

\[
\hat{y} = H(E_m(s, r)),
\]
\[
L_{\text{exist}} = -E_{(s,r,o^*) \sim \mathcal{I} \cup \mathcal{O}} \left[ y \log(\hat{y}) + (1 - y) \log(1 - \hat{y}) \right],
\]

where \( y \) is the true label indicating membership in \( \mathcal{I} \) (\( y = 1 \)) or \( \mathcal{O} \) (\( y = 0 \)).

Finally, the overall loss function for memory modality pre-training is shown in Equation 3, where \( \lambda_1, \lambda_2, \) and \( \lambda_3 \) are coefficients for different loss functions, with default values set to 1. Notably, the memory modality pre-training phase is entirely independent of the LLMs \( F \) that are to be edited. Therefore, it can be trained independently to embed a substantial amount of knowledge memory. As LLMs are updated or replaced over time, \( E_m \) can be reused across various models, facilitating efficient cross-model editing.

\[
L_{\text{pre-training}} = \lambda_1 L_{\text{inject}} + \lambda_2 L_{\text{associate}} + \lambda_3 L_{\text{exist}}.
\]

(3)

4.2 Memory Modality Bridging

After memory modality pre-training, we have obtained a memory modality encoder \( E_m \), which has stored the relevant memories of \( (s, r, o^*) \) and is capable of extracting related memories and determining their existence based on context. Revisiting Equation 2, we observe that we still need a memory-to-language module \( P_m \) to bridge the memory modality to LLMs. This module allows LLMs to understand and interpret the memories.

In mainstream MM-LLM works (Li et al., 2023a; Liu et al., 2024; Jian et al., 2024), to obtain the modality-to-language alignment modules \( P \), a modality-conditioned text generation loss is typically used as the training objective. Drawing inspiration from this, we also adopt this training objective, enabling the LLM \( F \) to correctly predict \( o^* \) based on the output of the memory modality and the prompt (composed of \( s \) and \( r \)), as specifically shown in Equation 4. Here, we use a simple two-layer fully connected network as \( P_m \), while keeping the parameters of the LLM \( F \) and the memory modality encoder \( E_m \) frozen.

\[
L_1 = -E_{(s,r,o^*) \sim D_{\text{edit}}} \left[ \log \mathbb{P}_F(o^* \mid P_m(E_m(s, r)) \oplus (s, r)) \right].
\]

(4)

However, training solely with this objective only enables LLMs to comprehend knowledge present in memory. For knowledge not contained in memories, the output from the memory modality may lead to unpredictable behavior. Ideally, for knowledge not present in memories, i.e., \( (s, r, o^*) \in \mathcal{O} \), we expect the LLMs to ignore the memory modality’s output and maintain their original predictions. To achieve this, we minimize the discrepancy between pre- and post-edit model predictions by reducing the Kullback-Leibler (KL) divergence of their prediction distributions (Cover, 1999), as illustrated in Equation 5. Benefiting from the pretrained memory modality’s ability to differentiate
representations of in-scope (\( \mathcal{I} \)) and out-of-scope (\( \mathcal{O} \)) knowledge, LLMs can more readily discern whether they possess the relevant memory.

\[
L_2 = E_{(s,r,o^*)\sim\mathcal{O}} \left[ \mathrm{KL}\left( \mathbb{P}_{F^*}(\cdot \mid (s, r)) \right. \right. \\
\left. \left. \| \mathbb{P}_{F^*}(\cdot \mid P_m(E_m(s, r)) \oplus (s, r)) \right) \right].
\] (5)

In summary, we combine the two training objectives of the bridging phase into a single loss function, as shown in Equation 6, where \( \lambda_{kl} \) is the coefficient, defaulting to 1. Although the memory modality bridging phase is not LLM backbone agnostic, it only requires fine-tuning \( P_m \). Given that the parameters involved in this fine-tuning are negligible compared to the total number of parameters, it enables an efficient and rapid implementation of memory modality bridging.

\[
L_{\text{bridge}} = L_1 + \lambda_{kl} L_2.
\] (6)

5 Experiments

5.1 Experimental Setup

We first provide a brief overview of the datasets, models, metrics, and baseline methods used in our experiments. For more detailed information on the experimental setup, please refer to Appendix A.

Models and Datasets. We conducted experiments on three LLMs of varying sizes: GPT2-XL (1.5B) (Radford et al., 2019), GPT-J (6B) (Wang and Komatsuzaki, 2021), and LLaMA3 (8B) (Dubey et al., 2024). For the benchmark, we evaluated MindBridge on two commonly used knowledge editing datasets: the ZsRE dataset (Levy et al., 2017) and the Counterfact dataset (Meng et al., 2022a). For MindBridge, we designated the knowledge in the dataset that required editing as \( D_{edit} \), and a portion of the knowledge that did not require editing as \( \mathcal{O} \). Since the quantity of \( \mathcal{I} \) in the dataset is small and typically used only for testing, we directly substitute \( D_{edit} \) for \( \mathcal{I} \) during training.

Baseline Methods. We compare MindBridge with seven baseline methods, categorized into two groups: parameter-modifying knowledge editing approaches—Fine-Tuning (FT-L) (Meng et al., 2022a), r-ROME (Gupta et al., 2024a), MEMIT (Meng et al., 2022b), EMMET (Gupta et al., 2024c) and AlphaEdit (Fang et al., 2024); and parameter-preserving knowledge editing methods—GRACE (Hartvigsen et al., 2024) and WISE (Wang et al., 2024). For these baseline methods, we utilize EasyEdit (Zhang et al., 2024c) for replication and testing, applying the default parameter settings.

Metrics. Following previous work (Fang et al., 2024; Zhang et al., 2024c), we adopt three metrics to evaluate the performance of the edited model: reliability (edit success rate), generalization (paraphrase success rate), and locality (neighborhood success rate). These are abbreviated as Rel., Gen., and Loc., respectively. We further compute the average of these three metrics, denoted as Avg., to represent the overall editing performance.

5.2 Main Results

Table 1 shows the performance comparison between MindBridge and existing knowledge editing methods after editing 10,000 facts. We can observe that, compared to baselines, MindBridge demonstrates superior performance across multiple LLMs, different datasets, and almost all metrics. For example, on LLaMA3 and GPT-J, MindBridge outperforms the best baseline by more than 20% and 15% respectively on the Avg. metric, which measures the comprehensive editing performance. On the Gen. metric, MindBridge significantly outperforms other editing methods in all experiments. Furthermore, it is particularly important to note that MindBridge is the only editing method among these that can achieve cross-model editing. When the LLM is updated, MindBridge can retain a large amount of previously edited knowledge and quickly bridge the memory modality encoder to the new LLM, enabling rapid domain knowledge adaptation while maintaining excellent editing performance.

5.3 Further Analysis

Ablation Study for Memory Modality Pre-training. In the memory modality pre-training stage, we designed three training objectives for the modality encoder, each corresponding to its intended function. To validate the effectiveness of these three training objectives, we conducted an ablation study to evaluate the impact of each objective. We performed knowledge editing on 10,000 factual statements using GPT-J. The results, as shown in Table 2, demonstrate that employing only the \( L_{inject} \) objective already yields promising editing performance (with Rel. metric reaching 71.30% on the Counterfact dataset and 94.89% on the ZsRE dataset). However, the generalization and locality of the edits are limited. Upon incorporating the \( L_{associate} \) training objective in addition to \( L_{inject} \), the generalization capability of editing is enhanced (with the Gen. metric increasing by 29.1%↑ on the Counterfact dataset and 8.73%↑
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Counterfact</th>
    <th colspan="4">ZsRE</th>
  </tr>
  <tr>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td colspan="10">Pre-edited</td>
    <td>0.28</td>
    <td>0.42</td>
    <td>\</td>
    <td>\</td>
    <td>22.44</td>
    <td>21.56</td>
    <td>\</td>
    <td>\</td>
  </tr>
  <tr>
    <td>FT-L</td><td></td><td>0</td><td>0</td><td>22.84</td><td>7.61</td><td>15.60</td><td>15.83</td><td>28.21</td><td>19.88</td>
  </tr>
  <tr>
    <td>WISE</td><td></td><td>1.79</td><td>1.99</td><td>50.38</td><td>18.06</td><td>24.88</td><td>24.84</td><td>99.98</td><td>49.90</td>
  </tr>
  <tr>
    <td>AlphaEdit</td><td></td><td>66.00</td><td>30.90</td><td>42.98</td><td>46.62</td><td>51.82</td><td>42.97</td><td>53.72</td><td>49.50</td>
  </tr>
  <tr>
    <td>EMMET</td><td></td><td>85.51</td><td>46.65</td><td>62.17</td><td>64.78</td><td>70.37</td><td>60.34</td><td>78.37</td><td>69.69</td>
  </tr>
  <tr>
    <td>GRACE</td><td>GPT2-XL</td><td>100</td><td>0.39</td><td>68.75</td><td>56.38</td><td>100</td><td>3.17</td><td>100</td><td>67.72</td>
  </tr>
  <tr>
    <td>r-ROME</td><td></td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>MEMIT</td><td></td><td>45.36</td><td>39.70</td><td>66.25</td><td>44.08</td><td>47.20</td><td>24.70</td><td>60.34</td><td>50.44</td>
  </tr>
  <tr>
    <td>MindBridge</td><td></td><td>95.60</td><td>81.10</td><td>38.18</td><td>71.62</td><td>78.14</td><td>68.17</td><td>85.67</td><td>77.33</td>
  </tr>
  <tr>
    <td colspan="10">Pre-edited</td>
    <td>0.09</td>
    <td>0.29</td>
    <td>\</td>
    <td>\</td>
    <td>23.01</td>
    <td>22.25</td>
    <td>\</td>
    <td>\</td>
  </tr>
  <tr>
    <td>FT-L</td><td></td><td>10.58</td><td>7.39</td><td>1.03</td><td>6.34</td><td>15.73</td><td>14.04</td><td>9.47</td><td>13.08</td>
  </tr>
  <tr>
    <td>WISE</td><td></td><td>18.38</td><td>12.08</td><td>3.85</td><td>11.44</td><td>36.88</td><td>34.53</td><td>99.51</td><td>56.97</td>
  </tr>
  <tr>
    <td>AlphaEdit</td><td></td><td>92.00</td><td>46.45</td><td>57.40</td><td>65.28</td><td>76.95</td><td>52.37</td><td>62.02</td><td>63.78</td>
  </tr>
  <tr>
    <td>EMMET</td><td></td><td>85.51</td><td>46.65</td><td>62.17</td><td>64.78</td><td>70.37</td><td>60.34</td><td>78.37</td><td>69.69</td>
  </tr>
  <tr>
    <td>GRACE</td><td>GPTJ</td><td>100</td><td>0.29</td><td>99.08</td><td>66.46</td><td>100</td><td>3.13</td><td>100</td><td>67.71</td>
  </tr>
  <tr>
    <td>r-ROME</td><td></td><td>0</td><td>0</td><td>0</td><td>0</td><td>0.09</td><td>0.08</td><td>0</td><td>0.05</td>
  </tr>
  <tr>
    <td>MEMIT</td><td></td><td>95.80</td><td>57.64</td><td>61.87</td><td>71.77</td><td>86.17</td><td>69.70</td><td>75.96</td><td>77.28</td>
  </tr>
  <tr>
    <td>MindBridge</td><td></td><td>94.60</td><td>82.60</td><td>93.56</td><td>90.25</td><td>99.06</td><td>83.71</td><td>93.77</td><td>92.85</td>
  </tr>
  <tr>
    <td colspan="10">Pre-edited</td>
    <td>0.7</td>
    <td>1.3</td>
    <td>\</td>
    <td>\</td>
    <td>27.70</td>
    <td>27.08</td>
    <td>\</td>
    <td>\</td>
  </tr>
  <tr>
    <td>FT-L</td><td></td><td>25.67</td><td>9.24</td><td>0.23</td><td>11.71</td><td>5.99</td><td>3.92</td><td>0.64</td><td>3.52</td>
  </tr>
  <tr>
    <td>WISE</td><td></td><td>16.08</td><td>10.68</td><td>3.87</td><td>10.21</td><td>29.18</td><td>29.18</td><td>99.40</td><td>52.07</td>
  </tr>
  <tr>
    <td>AlphaEdit</td><td></td><td>91.70</td><td>51.54</td><td>55.58</td><td>66.28</td><td>78.64</td><td>62.06</td><td>73.57</td><td>71.43</td>
  </tr>
  <tr>
    <td>EMMET</td><td></td><td>60.53</td><td>31.81</td><td>32.28</td><td>41.54</td><td>62.94</td><td>59.64</td><td>31.95</td><td>51.51</td>
  </tr>
  <tr>
    <td>GRACE</td><td>LLaMA3</td><td>100</td><td>5.33</td><td>100</td><td>68.44</td><td>100</td><td>5.33</td><td>100</td><td>68.44</td>
  </tr>
  <tr>
    <td>r-ROME</td><td></td><td>0.09</td><td>0</td><td>0.15</td><td>0.08</td><td>0.70</td><td>0.50</td><td>1.01</td><td>0.74</td>
  </tr>
  <tr>
    <td>MEMIT</td><td></td><td>84.71</td><td>41.20</td><td>64.49</td><td>63.47</td><td>66.16</td><td>59.92</td><td>80.51</td><td>68.86</td>
  </tr>
  <tr>
    <td>MindBridge</td><td></td><td>93.85</td><td>83.35</td><td>92.14</td><td>89.78</td><td>99.03</td><td>85.50</td><td>92.65</td><td>92.39</td>
  </tr>
</table>

Table 1: Comparison of MindBridge with existing methods after editing 10,000 facts. The best results are shown in bold, and the second best results are underlined.

on the ZsRE dataset). Furthermore, integrating the \( L_{exist} \) training objective alongside \( L_{inject} \) improves the locality of editing (with the Loc. metric increasing by 47.24%↑ on the Counterfact dataset and 19.05%↑ on the ZsRE dataset). These findings validate the effectiveness of each of the three training objectives. Ultimately, by combining all three training objectives, MindBridge achieves the best editing performance.

Impact on General Ability. Existing knowledge editing methods may more or less affect the general capabilities of models (Gu et al., 2024; Gupta et al., 2024b). To test the impact of MindBridge on the general capabilities of edited models, we evaluated the edited LLaMA3 (8B) on the General Language Understanding Evaluation (GLUE) benchmark (Wang, 2018) and compared it with AlphaEdit and MEMIT, which exhibit good performance in comprehensive editing effectiveness. As shown in Figure 3, MindBridge has the smallest change in F1-score compared to the pre-edited model on six tasks, indicating that MindBridge can maintain the general capabilities of the model even in the face of a large amount of knowledge editing.

![Bar chart comparing F1-score of LLaMA3 (8B) on GLUE tasks after editing using different methods](page_1012_1342_410_246.png)

Figure 3: F1-score of LLaMA3 (8B) on the GLUE benchmark after editing 10,000 facts using MindBridge, AlphaEdit, and MEMIT. The evaluation includes six tasks: SST, MRPC, CoLA, RTE, MMLU, and NLI.

Visualization of In-scope and Out-of-scope Representations. To determine whether the memory modality can truly distinguish the presence of relevant memories, we randomly selected 1,000 examples from the in-scope and out-of-scope data of two datasets. Figure 4 shows the visualization of the representations extracted by the memory
<table>
  <tr>
    <th rowspan="2">Training objectives</th>
    <th colspan="4">Counterfact</th>
    <th colspan="4">ZsRE</th>
  </tr>
  <tr>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td>Pre-edited</td>
    <td>0.2</td>
    <td>0.5</td>
    <td>\</td>
    <td>\</td>
    <td>24.20</td>
    <td>22.93</td>
    <td>\</td>
    <td>\</td>
  </tr>
  <tr>
    <td>\( L_{inject} \)</td>
    <td>71.30</td>
    <td>34.20</td>
    <td>41.16</td>
    <td>48.88</td>
    <td>94.89</td>
    <td>75.28</td>
    <td>74.23</td>
    <td>81.47</td>
  </tr>
  <tr>
    <td>\( L_{inject} + L_{associate} \)</td>
    <td>83.70</td>
    <td>63.30</td>
    <td>73.12</td>
    <td>73.37</td>
    <td>97.75</td>
    <td>84.01</td>
    <td>85.18</td>
    <td>88.98</td>
  </tr>
  <tr>
    <td>\( L_{inject} + L_{exist} \)</td>
    <td>75.40</td>
    <td>50.70</td>
    <td>88.40</td>
    <td>71.50</td>
    <td>95.86</td>
    <td>81.45</td>
    <td>93.28</td>
    <td>90.20</td>
  </tr>
  <tr>
    <td>\( L_{inject} + L_{associate} + L_{exist} \)</td>
    <td><b>94.40</b></td>
    <td><b>80.50</b></td>
    <td><b>89.26</b></td>
    <td><b>88.05</b></td>
    <td><b>99.40</b></td>
    <td><b>84.87</b></td>
    <td><b>96.15</b></td>
    <td><b>93.47</b></td>
  </tr>
</table>

Table 2: Ablation study of the three training objectives in MindBridge’s memory modality pre-training. Edited LLM: GPT-J; 10,000 facts. Best results are highlighted in bold.

![Visualization of the dimensionality-reduced distributions of representations extracted by the memory modality encoder for \( \mathcal{I} \) and \( \mathcal{O} \).](page_184_370_1082_246.png)

Figure 4: Visualization of the dimensionality-reduced distributions of representations extracted by the memory modality encoder for \( \mathcal{I} \) and \( \mathcal{O} \).

![Scaling MindBridge to 60,000 edits on ZsRE.](page_184_646_1082_246.png)

Figure 5: Scaling MindBridge to 60,000 edits on ZsRE.

modality encoder, reduced to two dimensions using t-SNE. It can be observed that the data from \( \mathcal{I} \) and \( \mathcal{O} \) are separated in space, indicating that the memory modality encoder is capable of distinguishing whether relevant factual memories exist, thereby ensuring the locality of the edited LLM.

Scale Up to 60,000 Edits. We gradually extended MindBridge from 10,000 edits to 60,000 edits on the ZsRE dataset for GPT-J and LLaMA3, and tested their editing performance. As shown in Figure 5, despite the significant increase in the number of edits, MindBridge consistently achieves stable and superior editing performance. The Avg. metric remains largely unchanged, with only a slight decrease observed in the Locality metric.

Adaptability to Heterogeneous Architectures. Our prior evaluations focused on dense model architectures such as GPT-J and LLaMA3. To further probe MindBridge’s cross-model adaptability, we extended our analysis to include a distinct heterogeneous architecture: the Mixture-of-Experts (MoE) model (Fedus et al., 2022). Table 3 details the outcomes of applying 10,000 knowledge edits to two prominent MoE models, Qwen1.5-MoE (Team, 2024) and DeepSeekMoE 16B (Dai et al., 2024). Notably, MindBridge demonstrates robust editing performance even on these different architectures, achieving an Avg. metric of approximately 90% for both. This result strongly corroborates MindBridge’s capacity for effective cross-model knowledge editing, extending its applicability across diverse architectural paradigms.

Overfitting Tests. Prior research indicates that knowledge editing can introduce overfitting challenges (Wang et al., 2025; Liu et al., 2025). To evaluate MindBridge’s susceptibility to this phenomenon, we adopted the data construction methodology from the EVOKE benchmark (Zhang et al., 2024b). Specifically, we assessed LLaMA3(8B) models, each modified with 10,000 edits on the Counterfact dataset using MEMIT, AlphaEdit, and MindBridge, respectively. The evaluation focused on two key tasks: Relation Specificity and Prefix Distraction, utilizing the DP (Direct Probability), CAP (Correct Answer Probability), and EOS (Editing Overfit Score) metrics as defined in (Zhang et al., 2024b). The results, presented in Table 4, reveal that MindBridge surpasses both MEMIT and AlphaEdit in DP and EOS metrics, indicating a superior resilience against overfitting.

Analysis of MindBridge’s Editing Efficiency. MindBridge facilitates efficient knowledge editing during model updates by reusing its pretrained memory modality encoder, requiring only a straightforward memory modality bridging step for new LLMs. To quantify this, we analyzed MindBridge’s editing time against baseline methods, as shown in Figure 6, which depicts the time taken for 10,000 edits on LLaMA3 (8B) (conducted on a Linux server equipped with 8×NVIDIA A100
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Qwen1.5-MoE</th>
    <th colspan="4">DeepSeekMoE 16B</th>
  </tr>
  <tr>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td>ZSRE</td>
    <td>Pre-edited</td>
    <td>28.00</td>
    <td>27.04</td>
    <td>-</td>
    <td>-</td>
    <td>28.56</td>
    <td>28.23</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>MindBridge</td>
    <td>99.08</td>
    <td>84.30</td>
    <td>84.21</td>
    <td>89.20</td>
    <td>96.11</td>
    <td>80.11</td>
    <td>93.84</td>
    <td>90.02</td>
  </tr>
  <tr>
    <td>Counterfact</td>
    <td>Pre-edited</td>
    <td>0.50</td>
    <td>1.00</td>
    <td>-</td>
    <td>-</td>
    <td>1.22</td>
    <td>1.50</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>MindBridge</td>
    <td>94.00</td>
    <td>83.00</td>
    <td>89.20</td>
    <td>88.73</td>
    <td>94.02</td>
    <td>79.57</td>
    <td>88.93</td>
    <td>87.50</td>
  </tr>
</table>

Table 3: Performance of MindBridge on Mixture-of-Experts (MoE) models (number of edits: 10,000).

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Prefix Distraction</th>
    <th colspan="3">Relation Specificity</th>
  </tr>
  <tr>
    <th>DP↓</th>
    <th>CAP↑</th>
    <th>EOS↑</th>
    <th>DP↓</th>
    <th>CAP↑</th>
    <th>EOS↑</th>
  </tr>
  <tr>
    <td>Pre-edited</td>
    <td>7.12%</td>
    <td>21.44%</td>
    <td>69.60%</td>
    <td>0.37%</td>
    <td>14.46%</td>
    <td>92.80%</td>
  </tr>
  <tr>
    <td>MEMIT</td>
    <td>7.37%</td>
    <td><b>25.26%</b></td>
    <td>70.60%</td>
    <td>1.24%</td>
    <td><b>18.20%</b></td>
    <td>88.40%</td>
  </tr>
  <tr>
    <td>AlphaEdit</td>
    <td>7.91%</td>
    <td>21.09%</td>
    <td>63.60%</td>
    <td>1.97%</td>
    <td>15.12%</td>
    <td>85.20%</td>
  </tr>
  <tr>
    <td>MindBridge</td>
    <td><b>6.84%</b></td>
    <td>23.95%</td>
    <td><b>71.00%</b></td>
    <td><b>0.46%</b></td>
    <td>15.81%</td>
    <td><b>93.00%</b></td>
  </tr>
</table>

Table 4: Comparison of MindBridge with MEMIT and AlphaEdit in overfitting tests on the Relation Specificity and Prefix Distraction tasks (Zhang et al., 2024b). Best results are highlighted in bold.

![Bar chart comparing time comparison of MindBridge with other knowledge editing methods for 10,000 edits on LLaMA3 (8B)](page_184_1012_579_246.png)

Figure 6: Time comparison of MindBridge with other knowledge editing methods for 10,000 edits on LLaMA3 (8B).

![Bar charts showing editing performance of Multi-MindBridge for Counterfact and ZsRE datasets](page_820_1012_579_246.png)

(a) Counterfact

(b) ZsRE

Figure 7: Editing performance of Multi-MindBridge, which simultaneously bridges two memory modality encoders trained on different datasets and evaluates them on a single dataset.

40GB PCIe GPUs). MindBridge (18.58 hours) exhibits superior editing speed compared to methods like WISE (44.92 hours), AlphaEdit (39.48 hours), and MEMIT (58.05 hours). While GRACE processes edits faster (16.48 hours), its limited editing effectiveness makes it less advantageous. For an analysis of MindBridge’s parameter overhead, please refer to Appendix B.4.

From MindBridge to Multi-MindBridge. Just as multimodal large language models are not limited to one or two modalities and can simultaneously support multiple modalities (Shu et al., 2023; Zhang et al., 2023), we propose Multi-MindBridge, which explores the editing performance of bridging multiple distinct memory modality encoders to the same LLM. We bridge encoders pretrained on the Counterfact and ZsRE datasets (each with 10,000 edits) to the LLaMA3 (8B) model. The results are shown in Figure 7. It can be observed that compared to using a single encoder, the editing performance only slightly decreases but enables the LLM to simultaneously acquire the knowledge memories from both memory modality encoders. For more implementation details and exploratory experiments of Multi-MindBridge, please refer to Appendix B.2.

6 Conclusion

In this paper, we propose MindBridge, a scalable cross-model knowledge editing method designed to address the issue that most current knowledge editing methods are overfitted to a single model, leading to the problem of discarded edited knowledge and frequent re-editing with each model update. Based on the novel concept of memory modality, MindBridge enables edited knowledge to transcend individual models. Extensive experiments conducted on two popular knowledge editing datasets and various LLMs demonstrate the effectiveness and scalability of MindBridge.
7 Limitations

Although MindBridge has demonstrated promising results in cross-model knowledge editing, it still faces several limitations. One limitation stems from resource constraints. We were unable to conduct tests on larger-scale models, restricting our experiments to models with up to 8B parameters. Furthermore, our focus was primarily on factual knowledge. We did not delve into other forms of knowledge, such as conceptual knowledge, which we leave this for future work.

Acknowledgments

This research was partially supported by the National Natural Science Foundation of China (Grants No.62406303), Anhui Provincial Natural Science Foundation (No. 2308085QF229), Anhui Province Science and Technology Innovation Project (202423k09020010), the Fundamental Research Funds for the Central Universities (No. WK2150110034).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.

Thomas M Cover. 1999. Elements of information theory. John Wiley & Sons.

Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. 2024. Deepseek-moe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066.

Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164.

Qingsiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. arXiv preprint arXiv:2210.03329.

Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2024. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. Model editing harms general abilities of large language models: Regularization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16801–16819.

Akshat Gupta, Sidharth Baskaran, and Gopala Anumanchipalli. 2024a. Rebuilding rome: Resolving model collapse during sequential model editing. arXiv preprint arXiv:2403.07175.

Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. 2024b. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07453.

Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. 2024c. A unified framework for model editing. arXiv preprint arXiv:2403.14236.

Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2024. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785.

Jitesh Jain, Jianwei Yang, and Humphrey Shi. 2024. Vocoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27992–28002.

Yiren Jian, Chongyang Gao, and Soroush Vosoughi. 2024. Bootstrapping vision-language learning with decoupled language pre-training. Advances in Neural Information Processing Systems, 36.

Damjan Kalajdzievski. 2024. Scaling laws for forgetting when fine-tuning large language models. arXiv preprint arXiv:2401.05605.

Jacob Devlin Ming-Wei Chang Kenton and Lee Christina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacl-HLT, volume 1, page 2. Minneapolis, Minnesota.

Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Bilp-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.

Tianci Liu, Zihan Dong, Linjun Zhang, Haoyu Wang, and Jing Gao. 2025. Mitigating heterogeneous token overfitting in llm knowledge editing. arXiv preprint arXiv:2502.00602.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022a. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022b. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv:2110.11309.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022a. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR.

Maxime Oquab, Timothée Darzet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

V Sanh. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.

Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. 2023. Audio-visual llm for video understanding. arXiv preprint arXiv:2312.06720.

Chenmien Tan, Ge Zhang, and Jie Fu. 2023. Massive editing for large language models via meta learning. arXiv preprint arXiv:2311.04661.

Qwen Team. 2024. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters".

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Alex Wang, 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.

Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Hua-jun Chen. 2024. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405.14768.

Pinzheng Wang, Zecheng Tang, Keyan Zhou, Juntao Li, Qiaoming Zhu, and Min Zhang. 2025. Revealing and mitigating over-attention in knowledge editing. arXiv preprint arXiv:2502.14838.

Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. 2023. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172.

Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. 2024a. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601.

Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLMA: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, Singapore. Association for Computational Linguistics.

Kai Zhang, Qi Liu, Zhenya Huang, Mingyu Cheng, Kun Zhang, Mengdi Zhang, Wei Wu, and Enhong Chen. 2022a. Graph adaptive semantic transfer for cross-domain sentiment classification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1566–1576.

Kai Zhang, Qi Liu, Hao Qian, Biao Xiang, Qing Cui, Jun Zhou, and Enhong Chen. 2021. Eatn: An efficient adaptive transfer network for aspect-level sentiment analysis. IEEE Transactions on Knowledge and Data Engineering, 35(1):377–389.

Kai Zhang, Hefu Zhang, Qi Liu, Hongke Zhao, Hengshu Zhu, and Enhong Chen. 2019. Interactive attention transfer network for cross-domain sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5773–5780.

Kai Zhang, Kun Zhang, Mengdi Zhang, Hongke Zhao, Qi Liu, Wei Wu, and Enhong Chen. 2022b. Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. arXiv preprint arXiv:2203.16369.

Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. 2024b. Uncovering overfitting in large language model editing. arXiv preprint arXiv:2410.07819.

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024c. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023a. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohou Li, Dacheng Li, Eric Xing, et al. 2023b. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

A Experimental Setup

In this section, we provide a more detailed introduction to the experimental setup, including the datasets used, a detailed explanation of the evaluation metrics, and a thorough description of the baselines.

A.1 Datasets

ZsRE. ZsRE (Levy et al., 2017) (Zero-Shot Relation Extraction) is a question-answering dataset widely used in knowledge editing tasks. Each data entry includes a question, the subject of the question, the updated answer, rephrased questions for testing the generalization of edits, and unrelated questions for testing the locality of edits. In the experiments comparing with baselines (see Section 5.2), we randomly selected 10,000 samples from this dataset as \( D_{edit} \). Due to the limited amount of in-scope data \( \mathcal{I} \) and to prevent contamination of test data, we directly used \( D_{edit} \) as a substitute for \( \mathcal{I} \) during the training of MindBridge. The remaining samples that did not require editing were used as out-of-scope data \( \mathcal{O} \). For other baseline methods, the edit data used were identical to those of MindBridge.

Counterfact. Counterfact (Meng et al., 2022a) is a more challenging knowledge editing dataset compared to ZsRE. It consists of incorrect facts that initially receive much lower scores than correct facts. Each entry in Counterfact includes a subject, an attribute of the subject to be edited, questions about attributes of non-identical subjects for testing edit locality, and paraphrases for testing
edit generalization. The construction of the editing data is similar to that of ZsRE.

A.2 Metrics
In this paper, we use three evaluation metrics—reliability, generalization, and locality—to represent performance. For simplicity, they are abbreviated as Rel., Gen., and Loc., respectively. Their specific formulas are as follows, where \( \mathbb{1}(\cdot) \) denotes the indicator function.

\[
Rel. = \frac{1}{|D_{edit}|} \sum_{(s, r, o^*) \in D_{edit}} \mathbb{1}(F^*(s, r) = o^*),
\]
\[
Gen. = \frac{1}{|\mathcal{I}|} \sum_{(s, r, o^*) \in \mathcal{I}} \mathbb{1}(F^*(s, r) = o^*),
\]
\[
Loc. = \frac{1}{|\mathcal{O}|} \sum_{(s, r, o^*) \in \mathcal{O}} \mathbb{1}(F^*(s, r) = F(s, r)).
\]

Here, Rel. measures the proportion of edits that have been successfully applied to a model, Gen. indicates the proportion of edits to which the model generalizes after editing, and Loc. reflects the extent to which the edited model retains knowledge of unrelated facts. Note that the Loc. metric is only evaluated on the edited model. Higher values for all three metrics indicate better editing performance.

A.3 Baselines
Here, we introduce several knowledge editing baselines that are compared in this paper. We utilize EasyEdit(Zhang et al., 2024c) for the reproduction of these baselines. For the selection of hyperparameters among them, we follow the default settings provided in the code.

• FT-L is a fine-tuning method proposed by (Meng et al., 2022a). FT-L directly fine-tunes the feed-forward network (FFN) of a specific layer, identified by causal tracing in ROME, by maximizing the probability of all tokens in the target sequence through last token prediction.

• GRACE (Hartvigsen et al., 2024) is a lifelong editing method. It writes new mappings into a pre-trained model’s latent space, creating a discrete, local codebook of edits without altering model weights. The phenomenon of GRACE’s generalization collapse under extensive editing has been evidenced and discussed in (Wang et al., 2024).

• WISE (Wang et al., 2024) is a method specifically designed for lifelong model editing. It inserts side memory in the FFN layers to preserve edited memory and trained a router to select which memory module to activate. To further enhance the support for continual editing, WISE incorporated a knowledge-sharding mechanism to enable different edits to be maintained in distinct parameter subspaces.

• r-ROME is an improvement upon ROME. Prior work has demonstrated that ROME can suffer from disabling edits, leading to immediate model collapse (Gupta et al., 2024b). r-ROME identifies that this is caused by irregularities in ROME’s implementation, specifically the asymmetric usage of key-vectors in its update equation, and proposes a more stable implementation, r-ROME.

• MEMIT (Meng et al., 2022b) is a scalable multi-layer update algorithm that employs explicitly calculated parameter updates to insert new memories. Building upon the direct editing approach of ROME, it designs an edit-distribution algorithm to distribute parameter updates uniformly across multiple layers of parameters. This enables MEMIT to update thousands of new pieces of knowledge.

• EMMET (Gupta et al., 2024c) unifies two editing methods, ROME and MEMIT, under a single optimization objective, namely, the preservation memorization objective. Furthermore, it improves upon ROME by employing equality constraints to support batched editing, achieving comparable performance to MEMIT.

• AlphaEdit (Fang et al., 2024) extends the locating-and-editing method by projecting the perturbation introduced during the editing process onto the null-space of the knowledge to be preserved. Subsequently, it applies this projection to the model parameters. This mechanism ensures the model’s preservation of its original knowledge following the edit.

B More Experimental results

B.1 Impact of Different Memory Modality Encoders
We use BERT-Base (110M) as the default modality encoder. To evaluate how different-sized BERT
models affect editing performance, we compared DistillBERT (66M) (Sanh, 2019) and BERT-Large (340M). As shown in Table 5, DistillBERT delivers strong performance across GPT-J and LLaMA3. In most cases, BERT-Base and DistillBERT outperform the others, with DistillBERT achieving the best overall results on Counterfact and BERT-Base excelling on ZsRE. Despite its larger size, BERT-Large performs worse than the other two models.

B.2 Implementation Details and Further Experiments of Multi-MindBridge

In Section 5.3, inspired by the idea that multimodal large language models are not restricted to one or two modalities, we proposed the implementation of Multi-MindBridge. This allows LLMs to be jointly bridged by multiple memory modality encoders while possessing the corresponding knowledge memories. Here, we elaborate on its implementation details.

Given n pre-trained memory modality encoders \( E_{m}^{1}, E_{m}^{2}, \ldots, E_{m}^{n} \) with different knowledge memories and their corresponding memory modality-language modules \( P_{m}^{1}, P_{m}^{2}, \ldots, P_{m}^{n} \), similar to Equation 2, the model \( F_{multi}^{*} \) after editing with Multi-MindBridge can be expressed as follows:

\[
x_{multi} = \bigoplus_{i=1}^{n} P_{m}^{i}(E_{m}^{i}(s, r)),
\]
\[
F_{multi}^{*}(s, r) = F(x_{multi} \oplus (s, r)).
\]

Here, \( \oplus \) denotes the concatenation of outputs from the modality encoders \( E_{m}^{1} \) to \( E_{m}^{n} \) to form \( x_{multi} \). The resulting \( x_{multi} \) is then provided as a soft prompt to the LLM, together with the text prompt, a multi-source integration strategy with parallels in other NLP models that utilize interactive processing or fuse diverse data structures like sequences and graphs (e.g., (Zhang et al., 2019, 2022a)). Notably, in Multi-MindBridge, each memory modality encoder is independently trained using the same pipeline as MindBridge, and subsequently, they are combined and integrated into the edited LLM.

In Section 5.3, we tested the performance of Multi-MindBridge when bridging modality encoders trained on different datasets. The results showed that compared to using a single encoder, the editing performance only slightly decreased but allowed the LLM to simultaneously acquire the knowledge memory from two encoders. Here, we test the editing performance of bridging multiple encoders trained on the same dataset. Specifically, we selected the ZsRE dataset and randomly sampled 50,000 edits, training one memory modality encoder for every 10,000 edits (with no overlap between edits). We evaluated the editing performance of bridging 1 to 5 encoders simultaneously on LLaMA3. The experimental results are shown in Figure 8. As can be observed, as the number of simultaneously bridged encoders increases, the editing performance gradually decreases but remains satisfactory. Even when bridging up to 5 encoders, the Avg score still reaches 88.59%.

![Editing performance of multiple memory modality encoders bridged to LLaMA3 on the ZsRE dataset, where each encoder handles 10,000 edits (non-overlapping).](page_1012_120_377_246.png)

Figure 8: Editing performance of multiple memory modality encoders bridged to LLaMA3 on the ZsRE dataset, where each encoder handles 10,000 edits (non-overlapping).

B.3 Impact of Editing Volume on General Capabilities

(Gupta et al., 2024b) observed that when using some model editing methods, the general capabilities of the model tend to decline as the number of edits increases, a challenge that relates to the broader goal of efficiently transferring and adapting learned capabilities across different domains or tasks (e.g., (Zhang et al., 2021)). Here, we test whether MindBridge exhibits a similar phenomenon. We conduct experiments using LLaMA3 (8B), gradually increasing the number of edits from 10,000 to 60,000. Similar to Section 5.3, we evaluate its performance on the GLUE benchmark, which includes six tasks: SST, MRPC, CoLA, RTE, MMLU, and NLI. The experimental results are shown in Figure 9. As can be seen, with the increase in the number of edits, the F1 scores for all tasks except NLI and CoLA remain stable, and in some cases, even outperform the pre-edited model. For NLI and CoLA, the F1 scores exhibit noticeable fluctuations as the number of edits increases, but they do not show a consistent downward trend. Overall, MindBridge demonstrates good scalability in preserving the model’s general capabilities.
<table>
  <tr>
    <th rowspan="2">Modality Encoder</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Counterfact</th>
    <th colspan="4">ZsRE</th>
  </tr>
  <tr>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
    <th>Rel.↑</th>
    <th>Gen.↑</th>
    <th>Loc.↑</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td>DistilBERT (66M)</td>
    <td>GPT-J</td>
    <td><b>98.21</b></td>
    <td><b>90.64</b></td>
    <td>92.50</td>
    <td><b>93.79</b></td>
    <td>97.60</td>
    <td>82.47</td>
    <td>96.52</td>
    <td>92.20</td>
  </tr>
  <tr>
    <td>BERT-Base (110M)</td>
    <td></td>
    <td>94.71</td>
    <td>83.57</td>
    <td><b>93.61</b></td>
    <td>90.63</td>
    <td><b>99.09</b></td>
    <td><b>84.22</b></td>
    <td>95.95</td>
    <td><b>93.09</b></td>
  </tr>
  <tr>
    <td>BERT-Large (340M)</td>
    <td></td>
    <td>94.43</td>
    <td>81.86</td>
    <td>91.07</td>
    <td>89.12</td>
    <td>96.16</td>
    <td>79.29</td>
    <td><b>97.22</b></td>
    <td>90.89</td>
  </tr>
  <tr>
    <td>DistilBERT (66M)</td>
    <td>LLaMA3</td>
    <td><b>97.71</b></td>
    <td><b>89.31</b></td>
    <td>90.17</td>
    <td><b>92.39</b></td>
    <td>97.12</td>
    <td>84.46</td>
    <td>93.81</td>
    <td>91.80</td>
  </tr>
  <tr>
    <td>BERT-Base (110M)</td>
    <td></td>
    <td>93.97</td>
    <td>83.58</td>
    <td><b>92.08</b></td>
    <td>89.88</td>
    <td><b>98.50</b></td>
    <td><b>85.02</b></td>
    <td>92.44</td>
    <td><b>91.99</b></td>
  </tr>
  <tr>
    <td>BERT-Large (340M)</td>
    <td></td>
    <td>92.80</td>
    <td>80.81</td>
    <td>89.99</td>
    <td>87.87</td>
    <td>96.43</td>
    <td>83.15</td>
    <td><b>95.34</b></td>
    <td>91.64</td>
  </tr>
</table>

Table 5: Impact of different modality encoders on editing performance, 10,000 edits. Best results are shown in bold.

<table>
  <tr>
    <th>Model</th>
    <th>Model Params (B)</th>
    <th>Additional Params (B)</th>
    <th>Total Add. Params (B)</th>
    <th>% Increase</th>
  </tr>
  <tr>
    <td>LLaMA3</td>
    <td>8.030</td>
    <td>\( E_m : 0.110,\ P_m : 0.008 \)</td>
    <td>0.118</td>
    <td>1.46</td>
  </tr>
  <tr>
    <td>GPT-J</td>
    <td>6.050</td>
    <td>\( E_m : 0.110,\ P_m : 0.010 \)</td>
    <td>0.120</td>
    <td>1.93</td>
  </tr>
  <tr>
    <td>Qwen1.5-MoE</td>
    <td>14.316</td>
    <td>\( E_m : 0.110,\ P_m : 0.004 \)</td>
    <td>0.114</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>DeepSeekMoE 16B</td>
    <td>16.376</td>
    <td>\( E_m : 0.110,\ P_m : 0.004 \)</td>
    <td>0.114</td>
    <td>0.70</td>
  </tr>
</table>

Table 6: Comparison of model parameters and additional parameters introduced by MindBridge across different LLMs. \( E_m \) denotes the memory modality encoder, and \( P_m \) denotes the bridging module. (B) stands for billions. Percentage increase is relative to the original model parameters.

![Line chart showing F1-score on GLUE benchmark for edited LLaMA3 (8B) as number of edits varies, with six tasks: SST, MRPC, CoLA, RTE, MMLU, NLI.](page_312_1042_484_246.png)

Figure 9: Change in F1-score on the GLUE benchmark for the edited LLaMA3 (8B) as the number of edited facts varies. 0 denotes the pre-edited model. Evaluation includes six tasks: SST, MRPC, CoLA, RTE, MMLU, and NLI.

B.4 Analysis of Additional Parameters in MindBridge

To evaluate MindBridge’s computational footprint, this section analyzes the additional parameters introduced during knowledge editing across various Large Language Models (LLMs), utilizing BERT-Base as the default memory modality encoder. Table 6 provides a detailed breakdown.

MindBridge introduces two lightweight components: a shared memory modality encoder (\( E_m \), 0.110B) and a model-specific bridging module (\( P_m \), 0.004–0.010B). As shown in Table 6, the encoder remains consistent across all models, thanks to its reusability, which facilitates smooth knowledge transfer. In contrast, the bridging module is fine-tuned for each LLM during the memory modality bridging stage, with significantly fewer parameters—thereby ensuring parameter efficiency during this stage.

As a result, the overall parameter overhead is indeed minimal. For example, MindBridge adds only 0.118B parameters to LLaMA3 (8.030B), a 1.46% increase, and 0.120B to GPT-J (6.050B), or 1.93%. For larger Mixture-of-Experts models such as Qwen1.5-MoE (14.316B) and DeepSeekMoE 16B (16.376B), the relative increase drops even further to 0.80% and 0.70%, respectively, with only 0.114B added.

This analysis confirms MindBridge as a parameter-efficient knowledge editing solution, a goal pursued through various techniques when adapting pre-trained models for specific functionalities (e.g., (Zhang et al., 2022b)). The added parameters remain consistently low, with under 2% increase across all evaluated models and below 1% for larger MoE architectures, emphasizing its lightweight design and minimal impact on base model size. This efficiency facilitates seamless integration and contributes to MindBridge’s broad applicability across diverse LLMs.

Reviewer paper summaries: This study proposes a framework named *MindBridge*, introducing the concept of "memory modality" to encode knowledge as an independent modality and leverage the decoupling properties of multimodal models. This addresses cross-model knowledge reuse and scalability challenges in large language model (LLM) editing. The method consists of two stages—memory modality pretraining and bridging—and designs three core training objectives (*Linject*, *Lassociate*, and *Lexist*). It achieves stable large-scale editing across models like GPT-J, LLaMA3, and datasets such as ZsRE and Counterfact.

The paper introduces MindBridge, a scalable and cross-model knowledge editing framework that addresses the limitations of existing methods that are tightly coupled with specific large language models (LLMs). Inspired by multi-modal large language models (MM-LLMs), MindBridge proposes a novel memory modality that encodes edited knowledge as an independent modality. The approach involves two main stages:

1. Memory Modality Pre-training: Training a memory modality encoder to store, retrieve, and differentiate knowledge.
2. Memory Modality Bridging: Fine-tuning a projector to align the memory modality with different LLMs.

The experimental results on multiple LLMs (GPT-XL, GPT-J, LLaMA3) and benchmark datasets (ZsRE, Counterfact) demonstrate that MindBridge outperforms existing knowledge editing methods in terms of reliability, generalization, and locality while enabling cross-model knowledge transfer.

The paper introduces MindBridge, a knowledge editing method using an external knowledge store for a model that is treated as a separate modality (called the memory modality). The external knowledge store is a small additional model, trained to encode knowledge, associate information it is tasked to "remember" with other relevant information, and predict whether knowledge is within the weights. A per-model projector is then trained to integrate this with a frozen LLM downstream.
Summary of strengths: ​Innovation: Introduces a "memory modality" encoding scheme for cross-model knowledge editing, enabling novel cross-model knowledge transfer and resolving compatibility issues between knowledge reuse and model scalability.

​Methodology: Features a systematic design with stage-wise pretraining, bridging mechanisms, and three training objectives (Linject, Lassociate, Lexist), forming a coherent framework for knowledge editing with strong theoretical and experimental logic.

- Novel Contribution: The introduction of cross-model knowledge editing and the memory modality concept offers a fresh perspective on updating LLMs without frequent re-editing.
- Scalability and Adaptability: MindBridge can handle tens of thousands of knowledge edits while remaining adaptable to different LLMs, making it highly practical for real-world applications.
- Strong Empirical Validation: The paper presents extensive experiments on multiple datasets and models, demonstrating superior performance over existing knowledge editing methods.
- Clear Motivation: The paper effectively highlights the problem of knowledge loss in current editing methods and justifies why a low-coupling approach is beneficial.
- Comprehensive Evaluation: The evaluation includes comparison with multiple baselines, ablation studies, scalability tests, and general ability assessments, providing a well-rounded analysis.

S1. I haven't seen work using a BERT/small LM as an external knowledge store before; I think this is an interesting idea. And the paper draws an interesting connection between external knowledge stores, multimodal settings, and knowledge editing. 

S2. The method is more robust to adding large numbers of edited facts than prior work. I think the combinations of memory stores is also an interesting application. 

S3. Though this is not the area I work most closely in, the datasets and evaluation methods seem reasonably chosen and the baselines appear to be comprehensive.
Summary of weaknesses: Experimental Credibility: Significant discrepancies with baselines (e.g., ROME[1], WISE[2]) require further validation of data collection and evaluation processes. Ablation studies and reproducible comparisons are needed to strengthen credibility.

​Methodological Comparison: Lacks systematic comparisons with memory-based methods (e.g., Memory classes). Multidimensional analyses (e.g., storage efficiency, editing granularity) should be added to highlight unique advantages.

​Cross-Model Applicability: Experiments are limited to dense architectures (e.g., GPT-XL, GPT-J, LLaMA3). Testing on heterogeneous models like MoE is recommended to validate universality.

​Technical Value: Quantitative analysis of "memory modality" efficiency (e.g., parameter increments, storage capacity, inference speed) is missing. Practical value in knowledge base updates and lightweight deployment should be argued.

[1] Gupta A, Baskaran S, Anumanchipalli G. Rebuilding rome: Resolving model collapse during sequential model editing[J]. arXiv preprint arXiv:2403.07175, 2024.

[2] Wang P, Li Z, Zhang N, et al. Wise: Rethinking the knowledge memory for lifelong model editing of large language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 53764-53797.

Limited Evaluation on Large-Scale Models: The experiments focus on models up to 8B parameters, which may not fully capture the challenges of knowledge editing in larger models (e.g., GPT-4, LLaMA3-70B).

Lack of Real-World Deployment Analysis: While the approach is theoretically sound, practical deployment considerations, such as computational overhead and integration complexity, are not fully discussed.

Limited Exploration of Non-Factual Knowledge Editing: The paper primarily focuses on factual knowledge editing, while conceptual or reasoning-based knowledge updates remain unexplored.

W1. While the paper engages closely with work in knowledge editing, I think it needs to more clearly explain how this is different from augmenting a model with a datastore or knowledge graph-- which there is extensive work on. (I do understand that the memory store here is a model, not a datastore, but if anything that seems like a _disadvantage_ of the approach-- how does this compare in efficiency and in reliability of edited information to a reasonable datastore/knowledge graph based solution?)

W2. While the inspiration in multimodality is interesting, I think the claim that this paper introduces "the novel concept of a memory modality" is not really reasonable. There's a wide number of things that are already called memory, from long-context modeling to retrieval-augmented LMs to persona modeling, and these are generally a separate store of information that must be integrated with text (and that is not necessarily itself text, e.g. embeddings or knowledge graph nodes). I think the paper _integrates_ memory as a modality in an interesting way, but the claims about the novelty of a separate knowledge base as a separate modality are not well-substantiated.
Comments, suggestions and typos: none

- Page 2, Line 46: “enabling efficient and precise modification of model knowl- edge” → “knowledge” should not be hyphenated.
- Page 5, Equation 1: Consider clarifying the notation for I(D_edit) and O(D_edit) to improve readability.

The Figure 2 is a nice illustration and clearly drawn. 

What motivated choosing BERT/similar as a memory store? Since these models already have some knowledge from pretraining, is there a concern that training the memory store has just shifted the knowledge editing problem from the bigger model to this smaller model?

Late fusion approaches to multimodal LLMs have also proven popular-- and some knowledge editing techniques also modify logits or later layers of the model. Did you consider integrating memory information later in the model, rather than in the prompt? (This is just something I'm curious about, not a factor in my scores.)
Overall assessment scores: 2.5

3

3
Metareview score: 3.5
Metareview: This paper introduces MindBridge —a lightweight, trainable external knowledge module treated as a separate modality for scalable, cross model knowledge editing. The framework enables scalable, cross-model knowledge editing and consists of two key stages: memory modality pretraining and modality bridging with LLMs. Evaluations across multiple datasets (ZsRE, Counterfact) and LLMs (GPT-J, GPT-XL, LLaMA3) suggest improved performance over prior methods.
Summary of reasons to publish: * The core idea of MindBridge is a novel and practical framework for scalable, cross-model knowledge editing. There was a general consensus among reviewers that treating an external memory store as a separate modality—inspired by multimodal LLMs—offered a fresh and flexible perspective on integrating edited knowledge without tightly coupling it to any specific language model.

* The proposed method showed solid and comprehensive empirical results, with efficiency and scalability, especially the ability to handle large-scale edits with minimal additional parameters and negligible inference slowdown.

* The authors were highly responsive to reviewer feedback, providing additional experiments, addressing theoretical concerns, and clearly clarifying distinctions from existing methods.