MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Prompts

## Abstract

Abstract LLMs hold great promise for healthcare applications, but the rapid evolution of medical knowledge and errors in training data often cause them to generate outdated or inaccurate information, limiting their applicability in high-stakes clinical practice. Model editing has emerged as a potential remedy without full retraining. While parameter-based editing often compromises locality and is thus ill-suited for the medical domain, retrieval-based editing offers a more viable alternative. However, it still faces two critical challenges: (1) representation overlap within the medical knowledge space often causes inaccurate retrieval and reduces editing accuracy; (2) existing methods are restricted to single-sample edits, while batch-editing remains largely unexplored despite its importance for real-world medical applications. To address these challenges, we first construct MedVersa, an enhanced benchmark with broader coverage of medical subjects, designed to evaluate both single and batch edits under strict locality constraints. We then propose MedREK, a retrieval-based editing framework that integrates a shared query-key module for precise matching with an attention-based prompt encoder for informative guidance. Experimental results on various medical benchmarks demonstrate that our MedREK achieves superior performance across different core metrics and provides the first validated solution for batch-editing in medical LLMs.
Our code and dataset are available at https://github.com/mylittleriver/MedREK .

## 1 Introduction

The remarkable success of large language models (LLMs) (Touvron et al., 2023a; b; Bai et al., 2023) in recent years has attracted significant attention from the medical community, leading to the emergence of specialized medical LLMs such as BioGPT (Luo et al., 2022), Med-PaLM (Singhal et al., 2023), ChatDoctor (Li et al., 2023) and PMC-LLaMA (Wu et al., 2024). However, due to the rapid evolution of medical knowledge and limitations in training data, these models may generate inaccurate or even fabricated responses (hallucinations), which can be particularly harmful in real-world medical advising and decision-making scenarios (Tian et al., 2024; He et al., 2025).

To address this issue, model editing (Meng et al., 2022; 2023; Wang et al., 2024) has emerged as a promising approach for updating the knowledge of pre-trained LLMs without requiring full retraining. Despite its growing popularity, model editing remains underexplored in the medical domain. A pioneering effort, MedLaSA (Xu et al., 2024), follows the locate-then-edit paradigm and constructs medical benchmarks for evaluating single-edit scenarios. However, its benchmark remains confined to simple single-edit settings and overlooks the more realistic batch-edit scenario. Moreover, locate-then-edit methods, which modify a small subset of parameters, often induce side effects on unrelated knowledge and compromise locality (Fang et al., 2025). This limitation is particularly concerning in medical applications, where reliability and consistency are critical.

To enable more realistic evaluation, we introduce MedVersa, the first benchmark designed to explore batch-editing in medical scenarios. It better reflects real-world use cases in which multiple related facts require simultaneous updates. We also substantially broaden subject coverage, yielding a more comprehensive benchmark that facilitates future research on knowledge editing for medical LLMs.
Under batch-editing scenarios, we first investigate retrieval-based approaches (Huang et al., 2023a; Yu et al., 2024) as an alternative to parameter-based editing, since they store new knowledge in an external memory module without altering the model’s original parameters and thereby better preserve locality.
However, our preliminary experiments show that the state-of-the-art retrieval-based method RECIPE (Chen et al., 2024b) often fails to retrieve the correct factual entries on the challenging medical evaluations. Further analysis indicates that the medical domain contains many textually similar but factually distinct knowledge items, which causes representation overlap in the retrieval space and thus hinders accurate knowledge matching. Based on these observations, we propose Medical Retrieval-based Editing with Key-aware prompts (MedREK).

As shown in Fig. [1](https://arxiv.org/html/2510.13500v2#S1.F1), our method introduces two key components: (1) a shared query-key MLP, which unifies the representation space of queries and keys for more precise knowledge retrieval;
(2) an attention-based prompt encoder, which generates more informative prompts to guide editing.
Building on these components, our MedREK achieves strong performance in both single-edit and batch-edit evaluations.
Comprehensive experiments on various medical benchmarks further demonstrate that our MedREK achieves state-of-the-art performance across Efficacy, Generality, and Locality metrics, validating the effectiveness of the proposed shared query-key MLP and attention-based prompt encoder. In summary, our main contributions are as follows:

Figure: Figure 1: Pipeline of our MedREK. Left: Construction of the knowledge base by encoding medical knowledge into key-value pairs. Right: Inference process where different types of queries are encoded to retrieve relevant knowledge and generate attention-based prompts for precise model editing.
Refer to caption: figures/fig0_0925.png

- •
We construct MedVersa, a medical factual knowledge editing benchmark that enables a more realistic batch-editing setting and offers broader subject coverage than existing benchmarks.
- •
We are the first to explore retrieval-based model editing for medical LLMs, proposing two novel components that separately enhance key-query alignment and prompt quality.
- •
We propose MedREK, a novel method that markedly improves knowledge editing performance by enhancing key-query alignment. Extensive experiments confirm that MedREK achieves state-of-the-art results across core metrics, demonstrating particularly strong gains in locality.

## 2 Related Works

Model Editing aims to efficiently update a pre-trained model’s behavior in response to new or corrected knowledge, without full retraining or negatively affecting unrelated predictions. Existing methods fall into three categories: locate-then-edit methods (Meng et al., 2022; Li et al., 2024), meta-learning-based strategies (Tan et al., 2024; Mitchell et al., 2022a), and retrieval-based approaches (Wang et al., 2024; Chen et al., 2024b).

- •
Locate-then-edit methods, such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023), identify the parameters associated with specific knowledge in LLMs and directly modify them to incorporate new information.
- •
Meta-learning-based methods take a different route by manipulating gradients to perform global parameter updates, aiming for more generalizable knowledge integration. For example, both KE (Cao et al., 2021) and MEND (Mitchell et al., 2022a) employ a lightweight hyper-network to adjust the updating gradient. InstructEdit (Wang et al., 2023a) builds on MEND by introducing instruction tuning for handling various tasks. Moreover, MALMEN (Tan et al., 2024) further generalizes MEND to batch-editing under memory constraints.
- •
Retrieval-based approaches edit model output by storing new knowledge in external memory modules, without altering pre-trained weights. These modules can take the form of codebooks, neurons, or auxiliary models, as demonstrated in DualEdit (Shi et al., 2025), SERAC (Mitchell et al., 2022b), T-Patcher (Huang et al., 2023a), GRACE (Hartvigsen et al., 2023), and MELO (Yu et al., 2024). More recently, WISE (Wang et al., 2024) designs dual memory and routes the model to the side memory in FFN for editing. RECIPE (Chen et al., 2024b) utilizes continuous prompt learning to prefix knowledge and dynamically retrieves knowledge from the knowledge base.

Accurate and up-to-date medical knowledge is critical for the safe and reliable application of large language models in healthcare. As medical facts evolve rapidly with new research and clinical guidelines, the ability to edit existing knowledge without retraining is essential.
Given the importance of medical knowledge editing, MedLaSA (Xu et al., 2024) pioneers this area and introduces the MedCF benchmark, but direct model edits can degrade locality and are limited to single-edit protocols. Building on this medical model editing line, we extend the setting to batch-editing and adopt the retrieval-based strategy. We further identify medical domain retrieval failures and remedy them with targeted improvements, culminating in MedREK, which delivers more reliable knowledge updates.

Prompt Tuning.
Efficiency has become increasingly important given the massive parameter scales of modern pre-trained models (Lin et al., 2024b; a; 2025; Yang et al., 2024; 2025; Wang et al., 2025; Wu et al., 2022; 2025; Xing et al., 2025; Jiang et al., 2025; Xiao et al., 2024; 2025b; 2025a).
As a representative parameter-efficient fine-tuning technique, prompt tuning is widely used in adapting foundation models to downstream tasks (Wang et al., 2022; 2023b; Piao et al., 2024).
It includes discrete prompts, expressed as actual text strings, and continuous prompts, encoded directly within the embedding space of the language model. Specifically, discrete prompts (Shin et al., 2020; Schick & Schütze, 2021a; b; Gao et al., 2021) are manually designed or automatically searched to elicit desired behavior from the model, while continuous prompts (Li & Liang, 2021; Liu et al., 2022; 2023; Xu et al., 2023) are trainable embeddings learned through optimization, capable of capturing more nuanced task-specific information. In this work, we train a prompt encoder to generate a continuous prompt that serves as external memory and enables targeted medical knowledge editing. These lightweight prompts can be stored efficiently, enabling effective editing with minimal resource overhead.

## 3 Motivation

### 3.1 Preliminaries

Model Editing. Let $f_{\theta}\in\!\mathcal{F}:\mathcal{Q}\!\mapsto\!\mathcal{A}$ denote the large language model which can map an input query ${q}\in\mathcal{Q}$ to the output answer $a\!=\!f_{\theta}(q)$. Given an edit sample pair $(q_{e},o_{e})$ that $f_{\theta}(q_{e})\neq o_{e}$, model editing hopes to modify $f_{\theta}$ to $f_{\theta}^{\prime}$ so that,

$$ $f_{\theta}^{\prime}=\mathrm{ME}(f_{\theta},q_{e},o_{e}),\quad f_{\theta}^{\prime}(q_{e})=o_{e},$ (1) $$

where $\mathrm{ME}(\cdot,\cdot,\cdot)$ means the model editor.

For a factual knowledge triple $\mathrm{KN}_{e}=$ ($s_{e},r_{e},o_{e}$), the components $s_{e}$, $r_{e}$, and $o_{e}$ denote the subject, relation, and object, respectively (Mallen et al., 2023). We define the pair $\mathrm{KN}_{e/o}:=(s_{e},r_{e})$ as the pre-defined knowledge key, and thus $\mathrm{KN}_{e}=(\mathrm{KN}_{e/o},o_{e})$.

Evaluation Metrics.
Beyond accurate knowledge editing (i.e., $f_{\theta}^{\prime}(q_{e})=o_{e}$), an ideal model editor should meet a range of additional requirements, each evaluated through specific metrics (Huang et al., 2023b; Yao et al., 2023; Zhang et al., 2024).

Generality. The edited model $f_{\theta}^{\prime}$ is expected to generalize beyond the editing query $q_{e}$ to correctly answer similar questions $q_{g}$.

$$ $\mathbb{E}_{(q_{g},o_{e})}~\mathbb{I}\{f_{\theta}^{\prime}(q_{g})=o_{e}\},$ (2) $$

Locality. The edited model $f_{\theta}^{\prime}$ should preserve the overall capabilities of the original large language model (i.e., $f_{\theta}$) by ensuring that examples unrelated to the editing target remain unaffected.

$$ $\mathbb{E}_{(q_{l},o_{l})}~\mathbb{I}\{f_{\theta}^{\prime}(q_{l})=f_{\theta}(q_{l})=o_{l}\},$ (3) $$

where $(q_{l},o_{l})$ denotes question-answer pairs that are semantically irrelevant to the edited samples $(q_{e},o_{e})$ and should ideally remain unaffected.
Note that, given the high precision requirements in medical domains, this metric becomes particularly critical when performing knowledge editing.

Efficacy. This metric calculates the accuracy of the modified model $f^{\prime}_{\theta}$ on the edited examples $(q_{e},o_{e})$, thereby reflecting the effectiveness of the edit.

$$ $\mathbb{E}_{(q_{e},o_{e})}~\mathbb{I}\{f_{\theta}^{\prime}(q_{e})=o_{e}\}.$ (4) $$

Fluency. This metric is designed to reflect the linguistic coherence of the response. It is quantified by computing the weighted average of bi- gram and tri-gram entropies, given by:

$$ $\sum_{k}f_{n}(k)\,\log_{2}f_{n}(k),$ (5) $$

where $f_{n}(\cdot)$ is the $n$-gram frequency distribution, and $k$ denotes a model-generated output.

Editing Setting.
Single-editing refers to modifying one knowledge item at a time, while batch-editing involves applying multiple edits simultaneously. Real-world updates such as revised treatment guidelines, newly discovered drug interactions, or retracted clinical findings often affect multiple related facts simultaneously. This makes batch-editing a realistic and practically valuable setting.
However, within medical domains, current approaches are primarily designed for single-edit scenarios, leaving the more realistic challenge of batch-editing largely unexplored. This further highlights the necessity of more precise retrieval mechanisms.

### 3.2 Locate-then-edit Methods Harm Locality

Locate-then-edit approaches (Meng et al., 2023; 2022) typically use causal tracing to identify influential model components and modify the corresponding parameters. However, such parameter updates often introduce undesirable side effects on unrelated knowledge, leading to a notable drop in locality metrics (Fang et al., 2025). This makes them ill-suited for medical applications, where high locality (i.e., preserving irrelevant knowledge) is essential to ensure reliability and safety.
To mitigate these issues, we turn our attention to the alternative retrieval-based methods, which avoid direct parameter modification and generally offer better locality preservation.

### 3.3 Retrieval-based Methods Struggle with Overlapping Knowledge

Figure: Figure 2: (a) Visualization of key representations in retrieval-based editing methods. Key overlap raises retrieval errors, decreasing editing accuracy. (b) Example of incorrect retrieval caused by key overlap. (c) Cosine similarity measurements quantifying degree of overlap among key representations.
Refer to caption: x1.png

In medical scenarios, many knowledge entries may be textually similar but factually unrelated, making accurate retrieval especially challenging. This calls for a more precise matching mechanism between queries and stored knowledge.
In particular, competing retrieval-based methods, such as RECIPE (Chen et al., 2024b), fail to retrieve the correct factual entry from the knowledge base on the medical knowledge editing benchmark. As shown in Figure [2](https://arxiv.org/html/2510.13500v2#S3.F2) (a), the representation space of RECIPE exhibits significant overlap among distinct knowledge items, leading to frequent confusion. This issue is exemplified in Figure [2](https://arxiv.org/html/2510.13500v2#S3.F2) (b):
for the query "Which drug is involved in a drug-drug interaction with Clofedanol?", the model incorrectly retrieves the unrelated entry "Which drug is involved in a drug-drug interaction with Camostat? Ozanezumab" from the knowledge base, due to overlapping key representations.

To quantify this phenomenon, Figure [2](https://arxiv.org/html/2510.13500v2#S3.F2) (c) further measures the overlap among key representations. Specifically, we report the percentage of unique samples involved in at least one pair with cosine similarity $>$ 0.6, denoted as “High-Sim. Samples (%)”, which reflects how many representations are highly aligned with others in the same batch. It can be observed that, as batch size increases, the share of samples participating in at least one high-similarity pair rises, reaching 51.09%, indicating greater representation overlap in larger batches. This provides quantitative support for our motivation and underscores the need for more precise retrieval tailored to medical knowledge editing.

## 4 Method

Figure: Figure 3: Comparison of MedVersa and MedCF++ in terms of medical subjects and dataset statistics. We provide the full distribution of medical subjects in the two datasets in Appendix.
Refer to caption: figures/compare_dataset.png

To address the limitations discussed above, we first construct MedVersa (i.e., Medical Versatile Knowledge Editing Dataset), an enhanced benchmark with broader coverage of medical subjects, designed to systematically examine batch-edit scenarios in the medical domain.
We then present the MedREK algorithm (i.e., Medical Retrieval-based Editing with Key-aware prompts), designed to mitigate knowledge forgetting from direct parameter updates and effectively tackle the issue of irrelevant retrievals highlighted in Figure [2](https://arxiv.org/html/2510.13500v2#S3.F2).

### 4.1 Construction of MedVersa Dataset

**Table 1: An example from the MedVersa dataset.**
| Medical Versatile Knowledge Editing Dataset |
| --- |
| Efficacy Question: What is the treatment for multiple carboxylase deficiency? |
| Generality Question: What is the therapeutic management for multiple carboxylase deficiency? |
| Locality Question: At what age does purposeful movement typically start in infants? |
| Ground Truth: Biotin |
| Counterfactual Edit Target: Thiamine |
| Locality Ground Truth: 6 months |
| Efficacy QA Pair: (Efficacy Question, Counterfactual Edit Target) |
| Generality QA Pair: (Generality Question, Counterfactual Edit Target) |
| Locality QA Pair: (Locality Question, Locality Ground Truth) |

The current MedCF (Xu et al., 2024) benchmark–built on the Drug Repurposing Knowledge Graph (DRKG) that links compounds, diseases, biological processes, side effects, and symptoms–presents two limitations: (i) Some prompts are reused for both Efficacy and Locality across the train/validation/test splits, which confounds batch-edit evaluation because edits targeting the Efficacy answer should not influence unrelated Locality responses. (ii) MedCF (Xu et al., 2024) remains limited in its topical breadth: most selected entities come from drugs and compounds, concentrating the benchmark in pharmacology. To address these issues, we construct MedVersa built on MedMCQA (Pal et al., 2022), which eliminates Efficacy–Locality prompt overlap to enable reliable batch-edit assessment and broadens coverage across medical subjects for more comprehensive evaluation. For clarity, the intermediate version that fixes only limitation (i) is denoted as MedCF++. An illustrative example is shown in Table [1](https://arxiv.org/html/2510.13500v2#S4.T1), and additional construction details are provided in Appendix [B](https://arxiv.org/html/2510.13500v2#A2) and [C](https://arxiv.org/html/2510.13500v2#A3).

Comparison of MedVersa and MedCF++. In Figure [3](https://arxiv.org/html/2510.13500v2#S4.F3), we present a detailed comparison of MedVersa and MedCF++. MedVersa spans a broader range of 20 medical subjects, including areas absent in MedCF++ such as Pediatrics and Social & Preventive Medicine. In contrast, MedCF++ covers only 12 medical subjects, with Pharmacology accounting for the majority (71.8%). The broader and more balanced subject coverage of MedVersa fills the gap in evaluating medical knowledge editing across a wider range of medical domains, enabling more comprehensive assessment.

### 4.2 The Proposed MedREK Algorithm

As shown in Figure [1](https://arxiv.org/html/2510.13500v2#S1.F1), our proposed MedREK comprises two main components: 1 a representation model featuring a unique shared query-key MLP (i.e., $\mathrm{MLP}_{qk}$) designed for effective knowledge retrieval, and 2 an attention-based prompt encoder for generating more informative prompts.

#### 4.2.1 The Shared Query-Key MLP

To ensure precise retrieval in model editing, we design a retrieval mechanism that satisfies both relevance and selectivity. Specifically, it aims to (1) accurately retrieve the specific knowledge associated with the given query, which reflects the Efficacy and Generality criteria, and (2) avoid unnecessary retrieval when the query is unrelated to stored knowledge, which corresponds to the locality requirement. To this end, we employ a shared query-key MLP encoder that encodes both memory keys and incoming queries into a unified representation space. Importantly, by constructing keys solely from the subject–relation pair $(s_{i},r_{i})$ instead of the full triplet $(s_{i},r_{i},o_{i})$, our design avoids incorporating irrelevant object information, thereby reducing retrieval noise.

Given an input sentence (either in the form of queries $q_{e}$, $q_{g}$ or $q_{l}$, knowledge key $\mathrm{KN}_{e/o}$ or a full knowledge triplet $\mathrm{KN}_{e}$), we first tokenize it using a pre-trained RoBERTa (Liu et al., 2019) tokenizer and obtain contextualized embeddings from the last hidden layer:

$$ $\mathbf{H}\in\mathbb{R}^{L\times d},\quad\mathbf{p}\in\mathbb{R}^{d}$ (6) $$

where $L$ is the sequence length and $d$ is the hidden dimension. We aggregate these embeddings by concatenating the CLS token embedding $\mathbf{p}$ with pooled statistics (mean, max, and min):

$$ $\displaystyle\mathbf{x}=\left[\mathbf{p};\operatorname{mean}(\mathbf{H});\operatorname{max}(\mathbf{H});\operatorname{min}(\mathbf{H})\right]\in\mathbb{R}^{4d}.$ (7) $$

Next, this vector $\mathbf{x}$ is fed into the representation model’s MLP layers. For queries $q_{e}$, $q_{g}$ or $q_{l}$ and knowledge key $\mathrm{KN}_{e/o}$, we apply a shared MLP encoder MLP${}_{\text{qk}}$:

$$ $\mathbf{z}_{q}=\text{MLP}_{\text{qk}}(\mathbf{x})=\text{ReLU}(\mathbf{W}_{q2}(\mathbf{W}_{q1}\mathbf{x}))+\mathbf{W}_{q1}\mathbf{x},$ (8) $$

$$ $\mathbf{z}_{k}=\text{MLP}_{\text{qk}}(\mathbf{x})=\text{ReLU}(\mathbf{W}_{q2}(\mathbf{W}_{q1}\mathbf{x}))+\mathbf{W}_{q1}\mathbf{x},$ (9) $$

which projects them into a shared representation space for accurate matching in the retrieval stage. For full knowledge triplets $\mathrm{KN}_{e}$ used in prompt generation, a separate MLP encoder MLPv is applied:

$$ $\mathbf{z}_{v}=\text{MLP}_{v}(\mathbf{x})=\text{ReLU}(\mathbf{W}_{k2}(\mathbf{W}_{k1}\mathbf{x}))+\mathbf{W}_{k1}\mathbf{x},$ (10) $$

where $\mathbf{z}_{k}$ is stored as the key of the $(k,v)$
pair in the knowledge base, while
$\mathbf{z}_{v}$ is further passed to the prompt encoder to generate continuous prompt as discussed next.

#### 4.2.2 Attention-Based Prompt Encoder

To enable precise edits, we further design a prompt encoder that maps each knowledge representation into a sequence of continuous prompt tokens via multi-head attention. Unlike traditional prompt tuning with fixed prompts for broad tasks, medical knowledge editing demands fine-grained, fact-specific modifications. Given the subtle differences between medical facts, dynamically generating prompts conditioned on the input knowledge is essential. Our prompt encoder learns to produce knowledge-specific prompts, leading to more accurate and effective editing.
Specifically, given the knowledge representation $\mathbf{z}_{v}\in\mathbb{R}^{d{\text{in}}}$ obtained from the representation model, we first project it into a set of query vectors for prompt tokens, and a single key and value vector:

$$ $\mathbf{Q}=\text{reshape}(\mathbf{W}_{q}\mathbf{z}_{v})\in\mathbb{R}^{T\times d},\quad\mathbf{K}=\mathbf{W}_{k}\mathbf{z}_{v}\in\mathbb{R}^{d\times 1},\quad\mathbf{V}=\mathbf{W}_{v}\mathbf{z}_{v}\in\mathbb{R}^{d\times 1},$ $$

where $T$ is the number of prompt tokens, $d_{\text{in}}$ and $d$ denote input and output dimensions, and $\mathbf{W}q\in\mathbb{R}^{Td\times d_{\text{in}}}$, $\mathbf{W}_{k},\mathbf{W}v\in\mathbb{R}^{d\times d_{\text{in}}}$ are learnable parameters in the attention mechanism, following the standard formulation in (Vaswani et al., 2017).
Next, a multi-head attention module is used to allow each prompt token to attend to the same knowledge vector and obtain contextualized prompt representations:

$$ $\mathbf{p}=\text{MultiHeadAttn}(\mathbf{Q},\mathbf{K},\mathbf{V})\in\mathbb{R}^{T\times d}.$ (11) $$

In this way, $\mathbf{p}_{\text{final}}\in\mathbb{R}^{B\times T\times d}$, which is stored as the value of the $(k,v)$ pair in the knowledge base. The number of continuous prompt tokens $T$ is a hyper-parameter, and we describe its selection for different datasets in Section [5.1](https://arxiv.org/html/2510.13500v2#S5.SS1).

### 4.3 Retrieval Pipeline and Training

Retrieval Pipeline. We employ a trainable knowledge prototype representation $\mathbf{z}_{pt}$
as a dynamic threshold for retrieval in the representation model.
During inference, retrieval is performed before the query tokens are fed into the embedding layer:

$$ $f_{r}(q)=\begin{cases}\mathbf{p}_{i},&\text{if }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{z}_{q}}\cdot{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbf{z}_{k_{i}}}>\mathbf{z}_{q}\cdot\mathbf{z}_{pt}\\ \varnothing,&\text{otherwise}\end{cases}$ (12) $$

where $f_{r}(\cdot)$ denotes the retrieval process for the query $q$, and $\textbf{z}_{k_{i}}$ is the most similar key representation in the knowledge base.
A prompt is retrieved only if it is more similar to the query than the learned prototype. If no prompt is returned, the model proceeds as usual, with inference unaffected.

Training.
Following RECIPE (Chen et al., 2024b), we utilize the training loss as $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{contra}}+\mathcal{L}_{\text{edit}}$, where $\mathcal{L}_{\text{edit}}=\frac{1}{B}\sum_{i=1}^{B}\left(\mathcal{L}_{\text{eff}}^{(i)}+\mathcal{L}_{\text{gen}}^{(i)}+\mathcal{L}_{\text{loc}}^{(i)}\right)$.
Details of each sub-loss are provided in Appendix [E](https://arxiv.org/html/2510.13500v2#A5).

## 5 Experiments

In this section, we conduct experiments to answer the following research questions: RQ1: Does MedREK outperform strong baseline editors on medical LLMs across core metrics and under batch-editing? RQ2: Do the proposed modules contribute significant gains individually and jointly? RQ3: Is the retrieval mechanism effective at locating the correct knowledge?

Table: Table 2: The overall results for single-editing ( # Editing=1) and batch-editing ( # Editing>1) using Meditron-7B on MedCF++ and MedVersa datasets.

### 5.1 Experimental Setup

Settings & Benchmarks.
We evaluate model editors under both Single-editing and Batch-editing settings to comprehensively assess their robustness and generalization capabilities.
Experiments are conducted on the improved MedCF++ benchmark and our newly constructed MedVersa dataset.
Unlike prior work (Xu et al., 2024), which only considers single-editing, we explore more realistic large-scale update scenarios by evaluating performance under 10/50/100-edit configurations.

Implementation Details.
Following MedLaSA (Xu et al., 2024), we use LLaMA2-based (Touvron et al., 2023c) Meditron-7B (Li et al., 2023) as the primary model and include LLaMA3-based (Grattafiori et al., 2024) HuatuoGPT-o1-8B (Chen et al., 2024a) for additional evaluation.
We train MedREK for 200 epochs and report the results using the checkpoint with the smallest loss. We use 3 and 8 prompt tokens for MedCF++ and MedVersa, respectively. See Appendix [A](https://arxiv.org/html/2510.13500v2#A1) for details of hyper-parameters.

Baselines.
We compare MedREK with several
knowledge editing baselines, including MEND (Mitchell et al., 2022a), MEMIT (Meng et al., 2023), MedLaSA (Xu et al., 2024), and RECIPE (Chen et al., 2024b).
Note that MedLaSA (Xu et al., 2024) is excluded from batch-editing experiments, as its parameter modification strategy is inherently designed for single-edit settings and cannot be directly extended to handle multiple simultaneous edits.
This limitation underscores the importance of developing batch-capable editing methods.
We re-implement RECIPE in the medical domain for fair comparison.

Table: Table 3: The overall results for single-editing ( # Editing=1) and batch-editing ( # Editing>1) using HuatuoGPT-o1-8B on MedCF++ and MedVersa datasets.

Evaluation metrics.
In line with MedLaSA (Xu et al., 2024), we adopt four evaluation metrics: Efficacy (Eff.), Generality (Gen.), Locality (Loc.), Fluency (Flu.).
For Locality, we report the original four sub-metrics on MedCF++, and a simplified overall score on MedVersa. Definitions of each Loc. sub-metric are provided in Appendix [A](https://arxiv.org/html/2510.13500v2#A1).
The weighted average (Avg.) is computed as in MedLaSA (Xu et al., 2024) to capture the trade-off between editing success (Eff. and Gen.) and Loc.

$$ $\text{Average}=\left(\frac{\text{Eff.}+\text{Gen.}}{2}+\frac{\sum\limits_{m\in\text{Loc.}}m}{|\text{Loc.}|}\right)\Big/2.$ (13) $$

### 5.2 Results on MedCF++ and Medversa (RQ1)

Single-editing. From Table [2](https://arxiv.org/html/2510.13500v2#S5.T2) and Table [3](https://arxiv.org/html/2510.13500v2#S5.T3), we observe that MedREK achieves competitive single-editing performance across both MedCF++ and MedVersa, using different LLM backbones. It outperforms baselines on most sub-metrics, with a notably improved overall average. Interestingly, MedLaSA (Xu et al., 2024) exhibits a significant drop in Locality on both datasets, supporting our earlier claim (Section [3.2](https://arxiv.org/html/2510.13500v2#S3.SS2))
that parameter-modifying methods introduce side effects on unrelated knowledge.
In contrast, MedREK performs consistently well across all evaluation setups, highlighting its robustness.

Batch-editing. Detailed evaluations with 10, 50, and 100 edits are shown in Table [2](https://arxiv.org/html/2510.13500v2#S5.T2) and Table [3](https://arxiv.org/html/2510.13500v2#S5.T3).
We observe that RECIPE (Chen et al., 2024b) suffers a clear performance drop compared to single-editing, with degradation worsening as the number of edits increases.
This suggests that overlapping key representations limit its effectiveness under batch-editing scenarios.
In contrast, MedREK consistently performs well across all sub-metrics, achieving the best or second-best results, which highlights
the strength of our improved representation model and prompt encoder in handling large-scale edits.

Remark. The experimental results on the original MedCF dataset are provided in Appendix [9](https://arxiv.org/html/2510.13500v2#A4.T9).

Figure: Figure 4: Distribution of query and corresponding key representations (i.e., the keys of the $k$–$v$ pairs in the knowledge base) under a batch of 100 edits using Meditron-7B on the MedCF++ dataset.
Refer to caption: figures/query_key_match.png

### 5.3 Abation Study (RQ2)

Table: Table 4: Ablation of Important Modules on MedCF++ using Meditron-7B.

To understand the contribution of each proposed component, i.e., the shared query-key MLP and the attention-based prompt encoder, we conduct an ablation study on batch-editing with 100 edits using MedCF++.
From Table [4](https://arxiv.org/html/2510.13500v2#S5.T4), we observe both components contribute significantly to the final performance of our MedREK.
The shared query-key MLP is critical for precise alignment between queries and stored knowledge, enabling more effective retrieval. Without it, performance drops notably due to mismatched representations, highlighting the importance of this component for accurate query-key interaction.
The attention mechanism in the prompt encoder also plays a key role by generating high-quality prompts, which further enhances overall editing performance.

Figure: Figure 5: (a-b) Distribution of relative similarity between the query, the ground-truth knowledge, the most similar knowledge and the prototype for test samples in 100-edit batch-editing using Meditron-7B on MedCF++. For Eff. and Gen., x values < 1 indicate incorrect or no retrieval; values > 1 indicate correct retrieval. For Loc., x values = 1 indicate no retrieval (locality preserved); values > 1 imply unintended retrieval. (c) Retrieval statistics across Eff., Gen., and Loc.
Refer to caption: figures/scatter_plot.png

### 5.4 Knowledge Retrieval Effectiveness (RQ3)

Obs1: MedREK achieves better retrieval via precise query-key alignment.
We analyze the distribution of query representations and key representations from the $k$–$v$ pairs in one 100-edit batch.
As shown in Figure [4](https://arxiv.org/html/2510.13500v2#S5.F4), MedREK aligns query and key representations well for both Efficacy and Generality inputs, indicating precise retrieval.
In contrast, RECIPE (Chen et al., 2024b) shows more scattered and distant representations, leading to lower scores in Eff. (68.30 vs. 77.96) and Gen. (70.74 vs. 79.88).
These observations suggest that RECIPE struggles to retrieve the correct knowledge, while MedREK benefits from more accurate query-key alignment, resulting in better editing success.

Obs2: MedREK achieves more accurate and controlled retrieval.
To evaluate retrieval effectiveness, we visualize the distribution of test samples across Eff., Gen., and Loc. metrics in the 100-edit batch setting on MedCF++, along with quantitative retrieval statistics (Figure [5](https://arxiv.org/html/2510.13500v2#S5.F5)).
For Eff. and Gen., the x-axis denotes the ratio between the query’s similarity to ground-truth knowledge vs. to the prototype. Values < 1 indicate incorrect or no retrieval.
Samples are categorized as correct (green), incorrect, or no retrieval.
MedREK achieves a significantly higher rate of correct retrievals than RECIPE (Chen et al., 2024b).
For Loc., the x-axis represents the ratio between the query’s similarity to the most similar knowledge entry (including prototype) and the prototype itself. Values > 1 indicate unintended retrieval of real knowledge, which harms locality.
MedREK shows fewer such cases, indicating better locality preservation.
Quantitative results further confirm
these observations: MedREK retrieves correct knowledge in nearly all Eff. and Gen. cases while avoiding unwanted retrievals in Loc., in stark contrast to RECIPE.
More details are provided in Appendix [A](https://arxiv.org/html/2510.13500v2#A1).

## 6 Conclusion

To address the practical challenge of updating clinical knowledge in medical LLMs, we introduce MedVersa, a benchmark for batch-wise model editing with broad coverage of medical domains.
We further propose MedREK, a retrieval-based editing framework tailored for medical LLMs. By incorporating a shared query–key MLP and an attention-based prompt encoder, MedREK enables precise retrieval and effective knowledge updates.
Experiments showcase the superiority of MedREK.

## Ethics Statement

This research follows the ICLR Code of Ethics. Our work focuses on developing method for knowledge editing in medical LLMs. The primary goal is to improve the adaptability and reliability of LLMs when incorporating updated or corrected medical information, thereby supporting more accurate and trustworthy downstream applications.

## Reproducibility Statement

To support reproducibility, we provide a complete description of our proposed MedREK in Section [4.2](https://arxiv.org/html/2510.13500v2#S4.SS2), with training and evaluation setups detailed in Section [5.1](https://arxiv.org/html/2510.13500v2#S5.SS1). Hyper-parameter settings are included in Appendix [A](https://arxiv.org/html/2510.13500v2#A1). Dataset details are described in Section [4.1](https://arxiv.org/html/2510.13500v2#S4.SS1) and Appendix [C](https://arxiv.org/html/2510.13500v2#A3).

## References

- Bai et al. (2023)
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
*arXiv preprint arXiv:2309.16609*, 2023.
- Cao et al. (2021)
Nicola De Cao, Wilker Aziz, and Ivan Titov.
Editing factual knowledge in language models.
2021.
URL [https://arxiv.org/abs/2104.08164](https://arxiv.org/abs/2104.08164).
- Chen et al. (2024a)
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang.
Huatuogpt-o1, towards medical complex reasoning with llms, 2024a.
URL [https://arxiv.org/abs/2412.18925](https://arxiv.org/abs/2412.18925).
- Chen et al. (2024b)
Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue.
Lifelong knowledge editing for llms with retrieval-augmented continuous prompt learning.
In *EMNLP*, 2024b.
URL [https://2024.emnlp.org/program/accepted_main_conference/](https://2024.emnlp.org/program/accepted_main_conference/).
- Fang et al. (2025)
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua.
Alphaedit: Null-space constrained model editing for language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=HvSytvg3Jh](https://openreview.net/forum?id=HvSytvg3Jh).
- Gao et al. (2021)
Tianyu Gao, Adam Fisch, and Danqi Chen.
Making pre-trained language models better few-shot learners.
In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pp. 3816–3830, Online, August 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.acl-long.295.
URL [https://aclanthology.org/2021.acl-long.295/](https://aclanthology.org/2021.acl-long.295/).
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, and Bethany Biron.
The llama 3 herd of models, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Hartvigsen et al. (2023)
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi.
Aging with GRACE: Lifelong model editing with discrete key-value adaptors.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=Oc1SIKxwdV](https://openreview.net/forum?id=Oc1SIKxwdV).
- He et al. (2025)
Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria.
A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics.
*Information Fusion*, 118:102963, 2025.
ISSN 1566-2535.
doi: https://doi.org/10.1016/j.inffus.2025.102963.
URL [https://www.sciencedirect.com/science/article/pii/S1566253525000363](https://www.sciencedirect.com/science/article/pii/S1566253525000363).
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9).
- Huang et al. (2023a)
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong.
Transformer-patcher: One mistake worth one neuron.
*arXiv preprint arXiv:2301.09785*, 2023a.
- Huang et al. (2023b)
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong.
Transformer-patcher: One mistake worth one neuron.
In *The Eleventh International Conference on Learning Representations*, 2023b.
URL [https://openreview.net/forum?id=4oYUGeGBPm](https://openreview.net/forum?id=4oYUGeGBPm).
- Jiang et al. (2025)
Yue Jiang, Haokun Lin, Yang Bai, Bo Peng, Zhili Liu, Yueming Lyu, Yong Yang, Jing Dong, et al.
Image-level memorization detection via inversion-based inference perturbation.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Li & Liang (2021)
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pp. 4582–4597, Online, August 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.acl-long.353.
URL [https://aclanthology.org/2021.acl-long.353/](https://aclanthology.org/2021.acl-long.353/).
- Li et al. (2024)
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu.
Pmet: Precise model editing in a transformer.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp. 18564–18572, 2024.
- Li et al. (2023)
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang.
Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge.
*Cureus*, 15(6), 2023.
- Lin et al. (2024a)
Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun.
Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 27370–27380, 2024a.
- Lin et al. (2024b)
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei.
Duquant: Distributing outliers via dual transformation makes stronger quantized llms.
*Advances in Neural Information Processing Systems*, 37:87766–87800, 2024b.
- Lin et al. (2025)
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan.
Toklip: Marry visual tokens to clip for multimodal comprehension and generation.
*arXiv preprint arXiv:2505.05422*, 2025.
- Liu et al. (2022)
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.
P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.
In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pp. 61–68, Dublin, Ireland, May 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.acl-short.8.
URL [https://aclanthology.org/2022.acl-short.8/](https://aclanthology.org/2022.acl-short.8/).
- Liu et al. (2023)
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.
Gpt understands, too, 2023.
URL [https://arxiv.org/abs/2103.10385](https://arxiv.org/abs/2103.10385).
- Liu et al. (2019)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach, 2019.
URL [https://arxiv.org/abs/1907.11692](https://arxiv.org/abs/1907.11692).
- Luo et al. (2022)
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.
Biogpt: generative pre-trained transformer for biomedical text generation and mining.
*Briefings in Bioinformatics*, 23(6):bbac409, 09 2022.
ISSN 1477-4054.
doi: 10.1093/bib/bbac409.
URL [https://doi.org/10.1093/bib/bbac409](https://doi.org/10.1093/bib/bbac409).
- Mallen et al. (2023)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.546.
URL [https://aclanthology.org/2023.acl-long.546/](https://aclanthology.org/2023.acl-long.546/).
- Meng et al. (2022)
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.
Locating and editing factual associations in gpt.
*Advances in neural information processing systems*, 35:17359–17372, 2022.
- Meng et al. (2023)
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau.
Mass-editing memory in a transformer.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=MkbcAHIYgyS](https://openreview.net/forum?id=MkbcAHIYgyS).
- Mitchell et al. (2022a)
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning.
Fast model editing at scale.
In *International Conference on Learning Representations*, 2022a.
URL [https://openreview.net/pdf?id=0DcZxeWfOPt](https://openreview.net/pdf?id=0DcZxeWfOPt).
- Mitchell et al. (2022b)
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning.
Memory-based model editing at scale.
In *International Conference on Machine Learning*, 2022b.
URL [https://arxiv.org/pdf/2206.06520.pdf](https://arxiv.org/pdf/2206.06520.pdf).
- Pal et al. (2022)
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu.
Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.
In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.), *Proceedings of the Conference on Health, Inference, and Learning*, volume 174 of *Proceedings of Machine Learning Research*, pp. 248–260. PMLR, 07–08 Apr 2022.
URL [https://proceedings.mlr.press/v174/pal22a.html](https://proceedings.mlr.press/v174/pal22a.html).
- Piao et al. (2024)
Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei.
Federated continual learning via prompt-based dual knowledge transfer.
In *Forty-first International Conference on Machine Learning*, 2024.
- Schick & Schütze (2021a)
Timo Schick and Hinrich Schütze.
Exploiting cloze-questions for few-shot text classification and natural language inference.
In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pp. 255–269, Online, April 2021a. Association for Computational Linguistics.
doi: 10.18653/v1/2021.eacl-main.20.
URL [https://aclanthology.org/2021.eacl-main.20/](https://aclanthology.org/2021.eacl-main.20/).
- Schick & Schütze (2021b)
Timo Schick and Hinrich Schütze.
It‘s not just size that matters: Small language models are also few-shot learners.
In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 2339–2352, Online, June 2021b. Association for Computational Linguistics.
doi: 10.18653/v1/2021.naacl-main.185.
URL [https://aclanthology.org/2021.naacl-main.185/](https://aclanthology.org/2021.naacl-main.185/).
- Shi et al. (2025)
Zhiyi Shi, Binjie Wang, Chongjie Si, Yichen Wu, Junsik Kim, and Hanspeter Pfister.
Dualedit: Dual editing for knowledge updating in vision-language models.
In *Second Conference on Language Modeling*, 2025.
URL [https://openreview.net/forum?id=X5vFauyVWr](https://openreview.net/forum?id=X5vFauyVWr).
- Shin et al. (2020)
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.
In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 4222–4235, Online, November 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.emnlp-main.346.
URL [https://aclanthology.org/2020.emnlp-main.346/](https://aclanthology.org/2020.emnlp-main.346/).
- Singhal et al. (2023)
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.
Large language models encode clinical knowledge.
*Nature*, 620(7972):172–180, Aug 2023.
ISSN 1476-4687.
doi: 10.1038/s41586-023-06291-2.
URL [https://doi.org/10.1038/s41586-023-06291-2](https://doi.org/10.1038/s41586-023-06291-2).
- Tan et al. (2024)
Chenmien Tan, Ge Zhang, and Jie Fu.
Massive editing for large language models via meta learning.
In *International Conference on Learning Representations*, 2024.
URL [https://openreview.net/pdf?id=L6L1CJQ2PE](https://openreview.net/pdf?id=L6L1CJQ2PE).
- Tian et al. (2024)
Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C Comeau, Rezarta Islamaj, Aadit Kapoor, Xin Gao, and Zhiyong Lu.
Opportunities and challenges for chatgpt and large language models in biomedicine and health.
*Briefings in Bioinformatics*, 25(1):bbad493, 01 2024.
ISSN 1477-4054.
doi: 10.1093/bib/bbad493.
URL [https://doi.org/10.1093/bib/bbad493](https://doi.org/10.1093/bib/bbad493).
- Touvron et al. (2023a)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023a.
- Touvron et al. (2023b)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023b.
- Touvron et al. (2023c)
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and Daniel M. Bikel and.
Llama 2: Open foundation and fine-tuned chat models.
*ArXiv*, abs/2307.09288, 2023c.
URL [https://api.semanticscholar.org/CorpusID:259950998](https://api.semanticscholar.org/CorpusID:259950998).
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.
URL [https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf).
- Wang et al. (2024)
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen.
WISE: Rethinking the knowledge memory for lifelong model editing of large language models.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=VJMYOfJVC2](https://openreview.net/forum?id=VJMYOfJVC2).
- Wang et al. (2023a)
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka.
Instructedit: Improving automatic masks for diffusion-based image editing with user instructions.
*arXiv preprint arXiv:2305.18047*, 2023a.
- Wang et al. (2023b)
Quanziang Wang, Renzhen Wang, Yichen Wu, Xixi Jia, and Deyu Meng.
Cba: Improving online continual learning via continual bias adaptor.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 19082–19092, 2023b.
- Wang et al. (2022)
Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng.
Imbalanced semi-supervised learning with bias adaptive classifier.
*arXiv preprint arXiv:2207.13856*, 2022.
- Wang et al. (2025)
Zhiwu Wang, Yichen Wu, Renzhen Wang, Haokun Lin, Quanziang Wang, Qian Zhao, and Deyu Meng.
Singular value fine-tuning for few-shot class-incremental learning.
*arXiv preprint arXiv:2503.10214*, 2025.
- Wu et al. (2024)
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang.
Pmc-llama: toward building open-source language models for medicine.
*Journal of the American Medical Informatics Association*, 31(9):1833–1843, 04 2024.
ISSN 1527-974X.
doi: 10.1093/jamia/ocae045.
URL [https://doi.org/10.1093/jamia/ocae045](https://doi.org/10.1093/jamia/ocae045).
- Wu et al. (2022)
Yichen Wu, Long-Kai Huang, and Ying Wei.
Adversarial task up-sampling for meta-learning.
*Advances in Neural Information Processing Systems*, 35:31102–31115, 2022.
- Wu et al. (2025)
Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei.
SD-loRA: Scalable decoupled low-rank adaptation for class incremental learning.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=5U1rlpX68A](https://openreview.net/forum?id=5U1rlpX68A).
- Xiao et al. (2024)
Yicheng Xiao, Lin Song, Jiangshan Wang, Siyu Song, Yixiao Ge, Xiu Li, Ying Shan, et al.
Mambatree: Tree topology is all you need in state space model.
*Advances in Neural Information Processing Systems*, 37:75329–75354, 2024.
- Xiao et al. (2025a)
Yicheng Xiao, Lin Song, Yukang Chen, Yingmin Luo, Yuxin Chen, Yukang Gan, Wei Huang, Xiu Li, Xiaojuan Qi, and Ying Shan.
Mindomni: Unleashing reasoning generation in vision language models with rgpo.
*arXiv preprint arXiv:2505.13031*, 2025a.
- Xiao et al. (2025b)
Yicheng Xiao, Lin Song, Rui Yang, Cheng Cheng, Zunnan Xu, Zhaoyang Zhang, Yixiao Ge, Xiu Li, and Ying Shan.
Haploomni: Unified single transformer for multimodal video understanding and generation.
*arXiv preprint arXiv:2506.02975*, 2025b.
- Xing et al. (2025)
Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Wanpeng Zhang, Haokun Lin, Guoqi Li, and Jiajun Zhang.
Efficientllm: Scalable pruning-aware pretraining for architecture-agnostic edge language models.
*arXiv preprint arXiv:2502.06663*, 2025.
- Xu et al. (2024)
Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, Wanyu Wang, Yuyang Ye, Xiangyu Zhao, Enhong Chen, and Yefeng Zheng.
Editing factual knowledge and explanatory ability of medical large language models.
In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*, CIKM ’24, pp. 2660–2670, New York, NY, USA, 2024. Association for Computing Machinery.
ISBN 9798400704369.
doi: 10.1145/3627673.3679673.
URL [https://doi.org/10.1145/3627673.3679673](https://doi.org/10.1145/3627673.3679673).
- Xu et al. (2023)
Ziyun Xu, Chengyu Wang, Minghui Qiu, Fuli Luo, Runxin Xu, Songfang Huang, and Jun Huang.
Making pre-trained language models end-to-end few-shot learners with contrastive prompt tuning.
In *Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining*, WSDM ’23, pp. 438–446, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9781450394079.
doi: 10.1145/3539597.3570398.
URL [https://doi.org/10.1145/3539597.3570398](https://doi.org/10.1145/3539597.3570398).
- Yang et al. (2024)
Lianwei Yang, Haisong Gong, Haokun Lin, Yichen Wu, Zhenan Sun, and Qingyi Gu.
Dopq-vit: Towards distribution-friendly and outlier-aware post-training quantization for vision transformers.
*arXiv preprint arXiv:2408.03291*, 2024.
- Yang et al. (2025)
Lianwei Yang, Haokun Lin, Tianchen Zhao, Yichen Wu, Hongyu Zhu, Ruiqi Xie, Zhenan Sun, Yu Wang, and Qingyi Gu.
Lrq-dit: Log-rotation post-training quantization of diffusion transformers for text-to-image generation.
*arXiv preprint arXiv:2508.03485*, 2025.
- Yao et al. (2023)
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang.
Editing large language models: Problems, methods, and opportunities, 2023.
URL [https://arxiv.org/abs/2305.13172](https://arxiv.org/abs/2305.13172).
- Yu et al. (2024)
Lang Yu, Qin Chen, Jie Zhou, and Liang He.
Melo: Enhancing model editing with neuron-indexed dynamic lora.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp. 19449–19457, 2024.
- Zhang et al. (2024)
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen.
A comprehensive study of knowledge editing for large language models, 2024.
URL [https://arxiv.org/abs/2401.01286](https://arxiv.org/abs/2401.01286).
- Zhou et al. (2024)
Yinan Zhou, Yuxin Chen, Haokun Lin, Shuyu Yang, Li Zhu, Zhongang Qi, Chen Ma, and Ying Shan.
Doge: Towards versatile visual document grounding and referring.
*arXiv preprint arXiv:2411.17125*, 2024.

## Appendix

## Appendix A More Implementation Details.

**Table 5: Hyper-parameters for MedREK on MedCF++ and MedVersa.**
| Hyper-parameter | MedCF++ | MedVersa |
| --- | --- | --- |
| Meditron-7B |  |  |
| Learning Rate | $1\times 10^{-5}$ | $1\times 10^{-5}$ |
| Batch Size | 8 | 8 |
| Repsresentation Dimension | 4096 | 4096 |
| Model Hidden Size | 4096 | 4096 |
| # Prompt Tokens | 3 | 8 |
| # Knowledge Prototype Tokens | 10 | 10 |
| HuatuoGPT-o1-8B |  |  |
| Learning Rate | $1\times 10^{-5}$ | $1\times 10^{-5}$ |
| Batch Size | 8 | 8 |
| Repsresentation Dimension | 4096 | 4096 |
| Model Hidden Size | 4096 | 4096 |
| # Prompt Tokens | 3 | 8 |
| # Knowledge Prototype Tokens | 10 | 10 |

Training Setup.
We train our MedREK and RECIPE (Chen et al., 2024b) on the training set of MedCF++ and MedVersa. As we observe the checkpoints after around 150 epochs all exhibit a trend of increasing loss for both methods, we stop our training at epoch 200.
The hyper-parameters for training and evaluation are kept the same as the baseline methods, except for the number of continuous prompt tokens, which we tune for RECIPE (Chen et al., 2024b). For RECIPE (Chen et al., 2024b) and MedREK, we both use 3 prompt tokens on MedCF++ and 8 prompt tokens on MedVersa. The complete hyper-parameter settings of training for MedREK are shown in Table [5](https://arxiv.org/html/2510.13500v2#A1.T5). All experiments are conducted on NVIDIA RTX 5090 GPUs.

Locality Metrics of MedCF and MedCF++. Following MedLaSA (Xu et al., 2024), the sub-metrics of Locality in MedCF (Xu et al., 2024) and MedCF++ are defined as follows:

- •
Target Distribution (TD): Does the editing operation alter the probability
distribution of the ground truth tokens?
- •
Entity Mapping (EM): Does the editing operation solely learn the
mapping relationship between head and tail entities?
- •
Structural Similarity (SS): Does the editing operation influence
unrelated knowledge with similar graph structures?
- •
Textual Similarity (TS): Does the editing operation have an impact
on unrelated knowledge that contains similar semantic text?

Details for Figure [5](https://arxiv.org/html/2510.13500v2#S5.F5).
We present the visualization of the distribution of test samples for each metric, focusing on the effectiveness of the knowledge retrieval, i.e., whether the correct piece of knowledge is retrieved for Eff. and Gen, and whether the knowledge is “ignored” for Loc. The result is obtained with batch-editing of 100 edits on MedCF++. The y-axis corresponds to the batch index the test samples belonging to.

Recall that the prompt selection criteria for each metric is as follows: For Eff. and Gen., the correct prompt is supposed to be selected. We calculate the similarity between the query and the top-1 similar key of the knowledge entry in the knowledge base, $sim(q,\operatorname{top}_{1})$, and the similarity between the query and the knowledge prototype, $sim(q,proto)$. If $sim(q,\operatorname{top}_{1})$ > $sim(q,proto)$, we select the prompt that generates $sim(q,\operatorname{top}_{1})$. Note that there are three cases here: (1) Correct selection, where the prompt corresponding to $sim(q,\operatorname{top}_{1})$ is the target knowledge. (2) Wrong selection, where the prompt corresponding to $sim(q,\operatorname{top}_{1})$ is a piece of unrelated knowledge. (3) No selection, where $sim(q,\operatorname{top}_{1})$ $=sim(q,proto)$, meaning the most similar entry is the prototype, and no retrieval is triggered. For Loc., no prompt is supposed to be selected. There are two cases: (1) If $sim(q,\operatorname{top}_{1})$ $=$ $sim(q,proto)$, which is the desired behavior, no prompt will be retrieved. (2) If $sim(q,\operatorname{top}_{1})$ $>$ $sim(q,proto)$, the query will falsely retrieve a prompt.

We calculate the relative magnitude between $sim(q,\operatorname{top}_{1})$, $sim(q,gt)$ (for Eff. and Gen.), and $sim(q,proto)$ to obtain the x value. For editing success (Eff. and Gen.), we present cases of correct retrieval, wrong retrieval, and no retrieval, and label the three cases out. For Loc., we do not label correct or wrong, instead, if x $=$ 1, it means the query selects no prompt as desired, and if x $>$ 1, it means the query falsely selects a prompt, which is not desired.
For editing success, more samples labeled as Correct Retrieval (green) means better retrieval performance. For Loc., less samples with x $>$ 1 means better retrieval.

Analysis of MedLaSA.
We find that MedLaSA (Xu et al., 2024) has the following critical design flaw and major weaknesses. First, it violates the spirit of knowledge editing since it considers the rephrase and locality queries for evaluation as known knowledge and applies causal tracing on them to calculate the corresponding layer-wise scaling factors. When testing, it retrieves the scaling factors calculated beforehand with the query text as key. In real-world scenarios, input queries are often unknown or unavailable in advance, making it impractical to apply causal tracing to determine the scaling factors for the adapters. As a result, effective knowledge editing cannot be achieved in practice. Second, despite the utilization of scaling factors to control the impact of adapters on each layer, it does not guarantee precise control. The LoRA (Hu et al., 2022) process may still introduce unintended intervention in a model where no modification is desired, thereby affecting unrelated knowledge. In contrast, our method transforms the queries into representations and matches them with the knowledge key representations for prompt retrieval without operations on the queries in advance, making it practical in real-world scenarios. Additionally, we achieve better locality by leaving the original model parameters frozen and only retrieving the prompt when necessary, which is crucial for editing medical LLMs.

## Appendix B Construction of MedCF++ Dataset.

As mentioned in [4.1](https://arxiv.org/html/2510.13500v2#S4.SS1), the only existing medical factual knowledge editing benchmark, MedCF (Xu et al., 2024), suffers from design issues that prevent it from supporting batch-editing, which we address by proposing an improved version, MedCF++.
Specifically, we remove any records in which the same prompt is used for both Eff. and Loc., as well as any records where Eff. and Loc. share the same prompt across different data entries. As a result, 181 records are removed from the training set, 47 from the validation set, and 47 from the test set. The cleaned dataset MedCF++ avoids prompt overlap to ensure reliable evaluation for batch-editing.

## Appendix C Construction of MedVersa Dataset.

Although MedCF++ supports batch-editing, it is still limited in its size and range of medical domains, as most of the entities it selects from the source dataset DRKG are drugs and compounds, causing the majority of the knowledge to fall under the pharmacology domain. To address these limitations, we construct the MedVersa dataset derived from MedMCQA (Pal et al., 2022), which allows for broader coverage of medical knowledge and supports batch-editing.

**Table 6: The prompts for querying Gemini 2.0 Flash.**
| Prompts for Querying LLMs |
| --- |
| Question Transformation: |
| You are given a phrase and its corresponding answer. Please rewrite the phrase into a clear question. |
| Input: |
| Phrase: {original question} |
| Answer: {correct answer} |
| Output only the rewritten question. |
| Question Rephrase: |
| Please rephrase the following question using precise medical terminology, ensuring that the original meaning is fully preserved. |
| Question: {question} |
| Output only the rephrased question. |

**Table 7: The distribution of medical subjects in MedVersa (%).**
| Medical Subject | Percentage (%) |
| --- | --- |
| Anatomy | 11.22 |
| Microbiology | 10.18 |
| Physiology | 10.08 |
| Surgery | 10.02 |
| Social & Preventive Medicine | 9.97 |
| Gynaecology & Obstetrics | 8.14 |
| Ophthalmology | 7.94 |
| Forensic Medicine | 6.78 |
| Pediatrics | 6.61 |
| ENT | 6.02 |
| Medicine | 2.71 |
| Pathology | 2.62 |
| Pharmacology | 2.05 |
| Biochemistry | 1.98 |
| Orthopaedics | 0.94 |
| Radiology | 0.82 |
| Psychiatry | 0.64 |
| Anaesthesia | 0.47 |
| Dental | 0.44 |
| Skin | 0.37 |

**Table 8: The distribution of medical subjects in MedCF++ (%).**
| Medical Subject | Percentage (%) |
| --- | --- |
| Pharmacology | 71.78 |
| Biochemistry | 6.96 |
| Pathology | 5.24 |
| Medicine | 4.48 |
| Anatomy | 2.98 |
| Gynaecology & Obstetrics | 1.94 |
| Physiology | 1.86 |
| Psychiatry | 1.44 |
| Ophthalmology | 1.31 |
| Skin | 0.73 |
| Orthopaedics | 0.73 |
| Surgery | 0.55 |

Efficacy and Generality Data Construction. The Efficacy QA pair ($q_{e}$, $o_{e}$) is used to measure the effectiveness of model editing. We utilize the MedMCQA (Pal et al., 2022) dataset, which contains medical knowledge in the form of multiple-choice questions, each comprising a correct answer and three incorrect options. Since the original “question” field in MedMCQA (Pal et al., 2022) is often expressed as a phrase instead of an interrogative, we first employ Gemini to rewrite it into a clear question form to ensure linguistic clarity and consistency. To construct the Efficacy QA pair, the rewritten question is paired with the correct answer as the ground truth, while one incorrect option is sampled as the counterfactual edit target. For example, as shown in Table [1](https://arxiv.org/html/2510.13500v2#S4.T1), the original question in MedMCQA (Pal et al., 2022) “Treatment of multiple carboxylase deficiency” is reformulated into the well-formed interrogative “What is the treatment for multiple carboxylase deficiency?”. The original correct answer “Biotin” is retained as the ground truth, while one incorrect option “Thiamine” is selected as the edit target. For the Generality QA pair ($q_{g}$, $o_{e}$), which evaluates editing effectiveness on similar questions, we employ Gemini to rephrase the Efficacy question. The prompts for querying Gemini are shown in Table [6](https://arxiv.org/html/2510.13500v2#A3.T6).

Locality Data Construction. Locality aims to evaluate whether the edited model preserves unrelated knowledge, and the model is expected to generate the same answer as before editing. To construct the Locality data, we sample a different entry from MedMCQA (Pal et al., 2022) than the one used for Efficacy, but within the same medical subject, leveraging the “subject name" field in MedMCQA (Pal et al., 2022). The original question is then converted into a standard interrogative, with the correct answer as the ground truth for the Locality question. For example, as shown in Table [1](https://arxiv.org/html/2510.13500v2#S4.T1), the Locality question “At what age does purposeful movement typically start in infants?" represents a different piece of knowledge from the Efficacy question, while sharing the same medical subject “Pediatrics". This design assesses the reliability of the model in preserving unrelated but same-domain knowledge after editing.

Comparison of Medical Subjects in MedVersa and MedCF++. Table [7](https://arxiv.org/html/2510.13500v2#A3.T7) and Table [8](https://arxiv.org/html/2510.13500v2#A3.T8) show the full distribution of medical subjects in MedVersa and MedCF++. Compared with MedCF++, which is dominated by Pharmacology, MedVersa exhibits a more balanced coverage across subjects. In addition, MedVersa includes 8 subjects that are absent in MedCF++. These underscore the broader coverage of medical domains of MedVersa, enabling a more comprehensive evaluation of medical knowledge editing.

## Appendix D More Experimental Results.

Table: Table 9: The overall results for single editing using Meditron-7B on original MedCF dataset.

Results of Single Editing on MedCF.
In Table [9](https://arxiv.org/html/2510.13500v2#A4.T9), we present the results of single-editing using Meditron-7B on the original MedCF (Xu et al., 2024) dataset, which contains duplicate prompts for Efficacy and Locality in a single record. MedREK performs consistently well on it, and obtains better results, especially in Loc.-EM on the cleaned dataset MedCF++ as shown in Table [2](https://arxiv.org/html/2510.13500v2#S5.T2). This supports our earlier claim (Section [4.1](https://arxiv.org/html/2510.13500v2#S4.SS1)) that duplicated prompts in MedCF (Xu et al., 2024) can lead to unreliable evaluations.

Edit Time of Different Methods.
To compare the efficiency of different methods, we conduct single-edit experiments using Meditron-7B (Li et al., 2023) on MedVersa and report the average edit time in Table [10](https://arxiv.org/html/2510.13500v2#A4.T10). We observe that MedREK and RECIPE (Chen et al., 2024b)—both retrieval-based methods—significantly outperform parameter-modifying methods. In particular, MedLaSA (Xu et al., 2024) is a parameter-modifying method that leverages LoRA (Hu et al., 2022) finetuning to perform edits. However, it requires 70 epochs of finetuning for each single edit, which results in substantially longer editing time. This makes it less efficient and impractical for scenarios requiring rapid or frequent knowledge updates.

**Table 10: Average edit time taken across different methods for single-editing using Meditron-7B on MedVersa.**
| Method | Edit Time (s) |
| --- | --- |
| MEND | 3.301 |
| MEMIT | 18.239 |
| MedLaSA | 16.787 |
| RECIPE | 0.006 |
| MedREK (Ours) | 0.012 |

## Appendix E Training Loss of MedREK

Following RECIPE (Chen et al., 2024b), the loss functions are defined as follows:

$$ $\mathcal{L}^{(i)}_{\text{eff}}=-\log\hat{f}_{\theta}\left(o^{(i)}_{e}\mid p^{(i)}\oplus f_{\text{emb}}(q^{(i)}_{e})\right)\qquad\text{(1)}$ $$

$$ $\mathcal{L}^{(i)}_{\text{gen}}=-\log\hat{f}_{\theta}\left(o^{(i)}_{g}\mid p^{(i)}\oplus f_{\text{emb}}(q^{(i)}_{g})\right)\qquad\text{(2)}$ $$

$$ $\mathcal{L}^{(i)}_{\text{loc}}=\operatorname{KL}\left(f_{\theta}(q^{(i)}_{l})\parallel\hat{f}_{\theta}\left(p^{(i)}\oplus f_{\text{emb}}(q^{(i)}_{l})\right)\right)\qquad\text{(3)}$ $$

where $f_{\theta}$ is the large language model (LLM) to be editied, and $\hat{f}_{\theta}$ is $f_{\theta}$ with the embedding layer $f_{\text{emb}}$ removed. The editing loss is then defined as $\mathcal{L}_{\text{edit}}=\frac{1}{B}\sum_{i=1}^{B}\left(\mathcal{L}_{\text{eff}}^{(i)}+\mathcal{L}_{\text{gen}}^{(i)}+\mathcal{L}_{\text{loc}}^{(i)}\right)$.
The contrastive learning loss for prompt learning are defined as follows:

$$ $\mathcal{L}^{(i)}_{\text{no}}=\delta\left(\mathbf{z}^{(i)}_{q_{e}},\mathbf{z}^{(i)}_{v},R\right)+\delta\left(\mathbf{z}^{(i)}_{q_{g}},\mathbf{z}^{(i)}_{v},R\right)\qquad\text{(4)}$ $$

$$ $\mathcal{L}^{(i)}_{\text{so}}=\delta\left(\mathbf{z}^{(i)}_{q_{l}},\mathbf{z}_{pt},R\right)+\delta\left(\mathbf{z}^{(i)}_{q_{e}},\mathbf{z}_{pt},R\setminus v_{i}\right)+\delta\left(\mathbf{z}^{(i)}_{q_{g}},\mathbf{z}_{pt},R\setminus v_{i}\right)\qquad\text{(5)}$ $$

$$ $\mathcal{L}_{\text{contra}}=\frac{1}{b}\sum_{i=1}^{b}\left(\mathcal{L}^{(i)}_{\text{no}}+\mathcal{L}^{(i)}_{\text{so}}\right)\qquad\text{(6)}$ $$

where $R=\{\mathbf{z}^{(i)}_{v}\}_{i=1}^{b}\cup\{r_{\Theta}\}$ and $R\setminus\mathbf{z}^{(i)}_{v}=R\setminus\{\mathbf{z}^{(i)}_{v}\}$. $\mathbf{z}^{(i)}_{v}$ is the representation of the editing knowledge triple $\mathrm{KN}^{(i)}_{e}$ transformed through Equation (9) in the paper. The query representations $\mathbf{z}^{(i)}_{q_{e}}$, $\mathbf{z}^{(i)}_{q_{g}}$, and $\mathbf{z}^{(i)}_{q_{l}}$ for $q_{e}^{(i)}$, $q_{g}^{(i)}$, and $q_{l}^{(i)}$ are attained via Equation ([8](https://arxiv.org/html/2510.13500v2#S4.E8)) in the paper, respectively. $\delta$ is the InfoNCE loss [2], formulated as:

$$ $\delta\left(q,\mathrm{KN}_{e}^{+},\{\mathrm{KN}^{(i)}_{e}\}_{i=1}^{n}\right)=-\log\left(\frac{\exp(q\cdot\mathrm{KN}_{e}^{+}/\tau)}{\sum_{i=1}^{n}\exp(q\cdot\mathrm{KN}^{(i)}_{e}/\tau)}\right)\qquad\text{(7)}$ $$

where $\tau$ is the temperature, typically set to 1 by default.
Finally, the total training loss is defined as $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{contra}}+\mathcal{L}_{\text{edit}}$.

## Appendix F Use of LLMs

We leverage Gemini 2.0 Flash for constructing the Efficacy, Generality, and Locality data in MedVersa, as detailed in [C](https://arxiv.org/html/2510.13500v2#A3). For Efficacy and Locality questions, the original phrases from MedMCQA (Pal et al., 2022) are transformed into interrogatives using Gemini. For Generality questions, we employ Gemini to rephrase the corresponding Efficacy questions. The prompts used for querying the LLM are provided in Table [6](https://arxiv.org/html/2510.13500v2#A3.T6). We took measures to ensure that the rewritten and rephrased questions maintained their original meaning. We used ChatGPT solely to improve the fluency and clarity of writing.
All content was written by the authors, and the model served only as a language assistant.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: see the weaknesses
soundness: 2
strengths: 1. It pioneers the construction of a batch-editing benchmark (MedVersa) for medical LLMs, addressing the gap in prior benchmarks (e.g., MedCF, MedCF++) that only support single-editing. MedVersa’s 20 medical subjects (including Pediatrics and Social & Preventive Medicine) fill the void of narrow domain coverage in existing benchmarks.
2. It innovatively adapts retrieval-based editing to the medical domain, designing a shared query-key MLP to resolve representation overlap (a critical issue in medical knowledge retrieval) and an attention-based prompt encoder to generate fine-grained prompts, overcoming the limitations of parameter-based editing (e.g., MedLaSA) that harms Locality.
3. Experimental design is rigorous: it validates MedREK on two medical LLMs (Meditron-7B, HuatuoGPT-o1-8B) and two benchmarks (MedCF++, MedVersa), covering single (1-edit) and batch (10/50/100-edit) scenarios. Ablation studies confirm the individual contributions of the shared query-key MLP and attention prompt encoder.
4. Benchmark construction is systematic: it first refines MedCF into MedCF++ to eliminate prompt overlap between Efficacy and Locality, then expands to MedVersa using MedMCQA, ensuring each data entry includes Efficacy/Generality/Locality QA pairs for comprehensive evaluation.
summary: This paper addresses the limitations of medical Large Language Models (LLMs) in generating outdated or inaccurate information due to rapid medical knowledge evolution and training data errors. It proposes two core contributions: (1) MedVersa, a medical knowledge editing benchmark that supports both single and batch-editing scenarios, with broader coverage of 20 medical subjects (vs. 12 in prior MedCF++) to avoid over-reliance on pharmacology; (2) MedREK, a retrieval-based editing framework integrating a shared query-key MLP (for precise query-knowledge alignment) and an attention-based prompt encoder (for knowledge-specific prompts). Experiments on MedCF++ and MedVersa with Meditron-7B and HuatuoGPT-o1-8B show MedREK outperforms baselines (e.g., MEND, MEMIT, RECIPE) across Efficacy, Generality, and Locality metrics, providing the first validated batch-editing solution for medical LLMs.
weaknesses: 1. The paper uses Gemini 2.0 Flash to rewrite MedMCQA’s original phrases into interrogative questions (for Efficacy/Generality) but provides no evidence of semantic consistency or medical term accuracy. There is no mention of manual verification (e.g., by medical professionals) or automatic metrics (e.g., BERTScore) to rule out ambiguity or term misuse, which may undermine MedVersa’s reliability.
2. RoBERTa is a general pre-trained model, but the paper does not compare it with medical-specific models (e.g., BioBERT, PubMedBERT) in terms of embedding quality for medical text. Without this comparison, it cannot be confirmed that RoBERTa is optimal for capturing medical knowledge representations.
3. Batch-editing is only tested on 10/50/100 edits. The paper does not explore larger batch sizes (e.g., 500 edits) to assess performance degradation, nor does it analyze the scalability of MedREK when the knowledge base expands to millions of medical facts (e.g., whether retrieval speed degrades and if index optimizations like FAISS are needed).
4. The ethics section only mentions compliance with the ICLR Code of Ethics but lacks details: (1) whether MedVersa’s source data (MedMCQA) contains patient-related information and has undergone de-identification; (2) whether ethical committee (IRB) approval was obtained for dataset construction; (3) how user medical queries are protected during MedREK’s deployment.

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 2
questions: - Could the authors provide targeted case studies of failed edits or unintended locality violations, particularly in batch-editing?
- How sensitive is the performance to the selection of RoBERTa as the encoder for keys/queries? Would other encoders or domain-specific pretraining yield different trade-offs?
- What steps were taken to ensure that the automatic rewriting and rephrasing steps (using Gemini) did not introduce subtle semantic distortions? Is there any human (expert) validation of sampled MedVersa questions and counterfactuals?
soundness: 3
strengths: - The paper addresses knowledge editing within the medical domain. By focusing on the "locality" requirement—ensuring edits do not corrupt unrelated knowledge—it tackles a critical safety and reliability problem specific to high-stakes applications.
- Dual contribution. The authors provide both a new resource and a new method.
- MedREK ablations. The paper includes an ablation study on MedREK's two primary components (the shared query-key MLP and the attention-based prompt encoder). This analysis, as referenced in the "Key Findings" and experimental setup, aims to validate the authors' design choices by isolating the contribution of each module to the framework's overall performance.
summary: **Problem & Motivation**

The paper addresses the unreliability of LLMs in high-stakes medical applications. Due to the rapid evolution of medical science and potential training data errors, LLMs often generate outdated or inaccurate information (hallucinations). While "model editing" (updating model knowledge without full retraining) is a potential solution, standard methods face critical limitations in the medical domain.
The authors identify three categories of model editing, but focus on the critical flaws of the two parameter-modifying approaches:
1. *Locate-then-edit*: These methods directly modify a small subset of the model's weights. The paper argues this is ill-suited for medicine because it "compromises locality," meaning an edit to one fact can unintentionally alter unrelated knowledge.
2. *Meta-learning-based*: These methods manipulate gradients to perform global parameter updates.
3. *Retrieval-based editing*: This approach avoids altering the original LLM parameters by storing new knowledge in an external memory (a key-value store) and retrieving it at inference time; e.g., RECIPE uses continuous prompt learning to prefix knowledge to the input query. This is the paper's preferred approach as it inherently preserves locality.

The paper's method is motivated by two specific failures of existing retrieval-based editors in the medical domain:
- Medical knowledge contains many textually similar but factually distinct items (e.g., "drug-drug interaction with Clofedanol" vs. "drug-drug interaction with Camostat"). This causes representation overlap in the retrieval space, leading the model to retrieve the wrong fact and reducing editing accuracy.
- Lack of batch-editing evaluation: existing benchmarks (like MedCF) focus on single-fact edits. However, real-world medical updates (e.g., new guidelines, retracted studies) often require updating multiple related facts at once (batch-editing).

**Method**

- **MedVersa.** To address the lack of a suitable benchmark, the authors first create MedVersa. It is constructed from the MedMCQA dataset and is designed to be superior to the previous MedCF benchmark.
   - Batch-editing support. It fixes a flaw in MedCF where "efficacy" (what you edit) and "locality" (what should not change) prompts overlapped, which made true batch-editing evaluation not possible.
   - Broader coverage. It spans 20 medical subjects, instead of 12 of MedCF.

- **MedREK (Medical Retrieval-based Editing with Key-aware prompts)**, the main contribution, a new retrieval-based editing framework. Its mechanism is trained in two phases. First, a knowledge base is built where each new fact (s, r, o) is encoded into a key-value pair: a Shared Query-Key MLP (MLP_qk) encodes the (s, r) pair to create the Key, and an Attention-based Prompt Encoder encodes the full (s, r, o) triplet to create the Value. A user's query is encoded by the same MLP_qk. If the query vector matches a Key (and is not an unrelated query, as determined by a threshold), the corresponding Value embedding is retrieved and prepended to the query, thus acting as a continuous prompt. The total loss L_total = L_edit + L_contra trains the modules for these two distinct goals:
- L_edit: An editing loss that trains the Prompt Encoder to generate "Values" (continuous prompts) that successfully guide the LLM to the new answer.
- L_contra: A contrastive loss (InfoNCE) that trains the Shared MLP_qk to pull matching query/key pairs together and push mismatched pairs apart, mitigating the representation overlap problem.

**Experimental setup**

- The framework is evaluated on two medical-domain LLMs: Meditron-7B (LLaMA2-based) and HuatuoGPT-01-8B (LLaMA3-based).
- Baselines. MedREK is compared against:
   - Parameter-based: MEND, MEMIT, and MedLaSA (a medical-specific editor).
   - Retrieval-based: RECIPE (the previous state-of-the-art retrieval editor).
- Benchmarks: MedCF++ and the new MedVersa.

**Key findings**

- Superior performance. MedREK achieves the highest average scores across all metrics (Efficacy, Generality, Locality) on both benchmarks and both LLMs.
- Batch-editing success. MedREK's performance remains high even in batch-editing (10, 50, 100 edits), while the baseline retrieval-editor (RECIPE) sees its performance degrade significantly as the batch size increases.
- Locality. Parameter-based methods (MedLaSA, MEMIT) show a significant drop in Locality, validating the paper's motivation for using a retrieval-based approach.
weaknesses: - *Limited comparison.* The results position MedREK primarily against RECIPE. While RECIPE is a strong baseline, the claim of SOTA performance would be substantially stronger if MedREK were benchmarked against a wider array of recent retrieval-based editors. The conclusion that MedREK solves the retrieval problem for medical text is contingent on RECIPE being a sufficient proxy for all retrieval-based methods.
- *Absence of statistical significance testing.* The presented results appear robust. However, for a paper making strong claims of superiority, the lack of statistical significance testing (e.g., p-values from t-tests or similar) is a methodological omission. It is unclear if the reported average performance gains are statistically significant across the entire test set or if they are driven by a high-leverage subset of the benchmark.
- *Prompt encoder motivation.* While the prompt encoder uses multi-head attention to generate prompt tokens, it is not fully transparent from the text or ablation exactly why attention-based prompts yield such significant locality gains. More interpretability analysis (e.g., what information is captured by learned prompts) would enhance understanding and trust in the mechanism.
- *Lack of cross-domain generalization results.* The evaluation is tightly focused on medical LLMs and medical datasets. Experiments on out-of-domain tasks could demonstrate the generality or limits of the approach, especially since the core architectural ideas (retriever + prompt generation) are not specific to the medical domain.
- *Limited model generalizability.* The framework is only validated on two medical LLaMA-based LLMs (Meditron-7B and HuatuoGPT-01-8B). This is a very small model pool, and it remains unproven how MedREK would perform with other model architectures or non-medical-domain backbones.
- *Questionable encoder choice.* The shared query-key MLP is built on a general-purpose RoBERTa. Given the paper's strong emphasis on the medical domain, the failure to test or justify this choice against domain-specific encoders is a clear methodological gap. A domain-specific encoder might inherently resolve some of the "representation overlap" issues.
- *Unspecified efficiency.*  The paper does not provide a clear analysis of the computational overhead of MedREK, as well as exact VRAM requirements.
- *Presentation quality.* The paper suffers from presentation issues that hinder readability. The submission retains a template title ("FORMATTING INSTRUCTIONS..."), and the methodological description (particularly the interaction between the two encoders) is not always easy to parse. Different typos have been identified, e.g., missing punctuation in Section 5.1.
- *No stated data/code sharing plan.* The paper lacks a clear commitment or plan for sharing the MedVersa benchmark or the MedREK implementation code, even for peer review. This omission hinders reproducibility and prevents independent verification of the experimental claims.

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: 1. The motivation of batch edits is reasonable and the proposed benchmark can potentially inspire future studies.
2. The experiment is comprehensive and demonstrates the proposed model’s effectiveness.
summary: This paper studies the problem of knowledge editing in the medical domain and introduces a retrieval-based framework designed for batch editing, along with a new benchmark for evaluating batch edit performance. To mitigate the issue of overlapping keys, the proposed method embeds subject–relation pairs and employs an attention-based encoder to generate soft prompts. Experimental results demonstrate that the proposed approach achieves state-of-the-art performance across multiple evaluation metrics.
weaknesses: 1. Regarding the methodology, it appears that this paper largely follows the design of RECIPE, which also formulates knowledge editing as training retrieved soft prompts. Although the authors introduce some specific modifications, such as using a pretrained RoBERTa encoder and a shared MLP projection, these seem to be more engineering adjustments rather than fundamentally new technical contributions.
2. The idea of the attention-based prompt encoder is also unclear to me. Since the attention is performed over a single embedding, the resulting soft prompt essentially comes from distributing attention scores over the projected value embedding. I am uncertain about the underlying criterion here and wonder how the performance would compare if the model simply projected z_v into T embeddings directly, without the attention operation.
3. Moreover, it remains unclear how these proposed techniques specifically address the challenges of batch editing.
4. As for the ablation study, `w/o shared MLP` means removing the MLP (only use H?) or using separate MLPs? What does `w/o Attn Prompt Enc.` represent? How do you obtain the soft prompts in this variant?