Title: A$^3$E: Towards Compositional Model Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model Editing
Abstract: Model editing has become a *de-facto* practice to address hallucinations and outdated knowledge of large language models (LLMs). However, existing methods are predominantly evaluated in isolation, i.e., one edit at a time, failing to consider a critical scenario of compositional model editing, where multiple edits must be integrated and jointly utilized to answer real-world multifaceted questions. For instance, in medical domains, if one edit informs LLMs that COVID-19 causes "fever" and another that it causes "loss of taste", a qualified compositional editor should enable LLMs to answer the question "What are the symptoms of COVID-19?" with both "fever" and "loss of taste" (and potentially more). In this work, we define and systematically benchmark this compositional model editing (CME) task, identifying three key undesirable issues that existing methods struggle with: *knowledge loss*, *incorrect preceding* and *knowledge sinking*. To overcome these issues, we propose A$^3$E, a novel compositional editor that (1) ***a**daptively combines and **a**daptively regularizes* pre-trained foundation knowledge in LLMs in the stage of edit training and (2) ***a**daptively merges* multiple edits to better meet compositional needs in the stage of edit composing. Extensive experiments demonstrate that A$^3$E improves the composability by at least 22.45\% without sacrificing the performance of non-compositional model editing.

A 3 E: Towards Compositional Model Editing

Abstract

Model editing has become a de-facto practice to address hallucinations and outdated knowledge of large language models (LLMs). However, existing methods are predominantly evaluated in isolation, i.e., one edit at a time, failing to consider a critical scenario of compositional model editing, where multiple edits must be integrated and jointly utilized to answer real-world multifaceted questions. For instance, in medical domains, if one edit informs LLMs that COVID-19 causes 'fever' and another that it causes 'loss of taste', a qualified compositional editor should enable LLMs to answer the question 'What are the symptoms of COVID-19?' with both 'fever' and 'loss of taste' (and potentially more). In this work, we define and systematically benchmark this compositional model editing (CME) task, identifying three key undesirable issues that existing methods struggle with: knowledge loss , incorrect preceding and knowledge sinking . To overcome these issues, we propose A 3 E, a novel compositional editor that (1) a daptively combines and a daptively regularizes pre-trained foundation knowledge in LLMs in the stage of edit training and (2) a daptively merges multiple edits to better meet compositional needs in the stage of edit composing. Extensive experiments demonstrate that A 3 E improves the composability by at least 22.45% without sacrificing the performance of non-compositional model editing. The code is available at https://github.com/piaohongming/A3E .

1 Introduction

LLMs learn extensive knowledge from massive pre-training corpora and utilize the learned knowledge during inference to meet a wide range of tasks [1, 2, 3, 4]. Despite their impressive capabilities, especially as the pre-training data and model size scale [5], LLMs remain prone to factual hallucinations [6] and outdated knowledge [7]. These errors often emerge gradually after deployment and significantly impairs system reliability, which necessitates timely and effective corrections. Conventional solutions such as re-training are not only time-consuming, but fine-tuning without access to pre-training data also poses a high risk of interfering with irrelevant knowledge. Therefore, model editing [8] has emerged as a promising approach that enables targeted updates to LLMs in a data-efficient manner (minimal reliance on pre-training data), localized scope (limited impact on irrelevant knowledge), and with high reliability (accurate correction of the targeted knowledge).

Efforts toward model editing center around improving reliability [9, 10], generalization [10, 11], and locality [8, 12], with further studies exploring editing ripple effects [13], bidirectional generalization [14], and its impact on downstream tasks [15, 16, 17, 18]. Progress along with these attempts is

Hao Wang

Ying Wei †

Jack forged an identity document.

Which article of the Criminal Law and

the Public Security Administration

Punishments Law did he violate?

edit training

COVID-19

>> fever

COVID-19

>> loss of taste

edit composing

w/ composability

fever,  loss of taste.

w/o composability

fever,  loss of fever.

What are the symptoms of

COVID-19?

(a) Multi-answer composition (MAC)

(b) Multi-question composition (MQC)

forging an identity

document

in the Criminal Law

>> Article 280

forging an identity

document

in the Public … Law

>> Article 26

w/ composability

Article 280, Article 26.

w/o composability

Article 280, Article 28.

edit training

edit composing

unanimously evaluated under a single-edit assumption ; that is, LLMs recall each atomic edit in isolation. For example, if an edit previously taught an LLM that COVID-19 causes 'loss of taste', evaluations only test whether this LLM can correctly recall this isolated fact, e.g., 'I have loss of taste, do I have COVID-19?' Unfortunately, this evaluation paradigm diverges significantly from real-world demands. In many practical domains, such as medicine and law (as shown in Fig. 1), questions likely require access to multiple edits simultaneously. In Fig. 1(a), for instance, answering a multi-symptom diagnostic question requires composing two pieces of knowledge (multi-answer composition, MAC); if these facts were introduced through prior edits, LLMs must effectively leverage both. Similarly, in Fig. 1(b), questions may require reasoning over multiple pieces of edited knowledge to form a unified answer (multi-question composition, MQC). In this paper, we formally define and benchmark tasks that demand such compositional capability as Compositional Model Editing (CME) tasks. The success of CME hinges on two criteria: (1) in the edit training stage, individual edits must be performed in a way that guards their composability for downstream inference; (2) in the edit composing stage, question-related edits must be effectively merged.

As shown in Fig. 2, state-of-the-art editing methods suffer from knowledge loss , incorrect preceding and knowledge sinking , which comprehensively encompass all the failure cases we observed. MEND [9] and ROME [8] modify the entire matrix in the feed-forward netwok (FFN) through hypernetworks and closed-form solutions, respectively, which yields significant knowledge loss when composing different edits. AlphaEdit [19] exhibits a similar level of knowledge loss , likely because it prevents interference between edits by projecting each update into the null space of unrelated questions. However, in the MAC and MQC settings we target, where a single question depends on multiple edits, this orthogonality collapses and conflicts between edits still persist. WISE [20] alleviates knowledge loss by randomly masking parameter updates of each individual edit, though the likelihood of mask overlap grows significantly as the number of edits increases and thus composability degrades. GRACE [21], MELO [22], and T-patcher [23] replace hidden states or insert new key-value pairs in the FFN layer in a low-rank form. Such low storage overhead for each edit enables them to treat all edits as an external vector database attached to the original parameters. Despite reduced knowledge loss , all three methods suffer from the pronounced issue of incorrect preceding . Collectively, these limitations reveal a pressing gap, i.e., how to train and compose different edits to improve edit composability?

Our response to the question is the proposed A 3 E . Specifically, previous works [24, 25] have pointed out that the FFN layer in Transformer [26] is a key-value neural memory [27], where the down projection matrix stores robust and generalizable foundation knowledge pre-trained on large datasets. The composition of these foundation knowledge pieces forms multiple advanced knowledge needed in the generation process of LLMs. We hypothesize and empirically verify that leveraging the pre-trained foundation knowledge in the down projection matrix is sufficient for not only composing the new knowledge but also boosting composability. Thus, in order to train edits with better composability , we A daptively combine the pre-trained foundation knowledge in the down projection matrix to reduce knowledge loss . By A daptively regularizing the use of selected knowledge at the last answer token and non-label logits, we effectively alleviate knowledge sinking and incorrect preceding . In order to compose edits to further enhance composability , we utilize a vector database [21, 22] to filter out irrelevant edits in a question-wise way at inference and A daptively merge relevant edits with proper combinations of pre-trained foundation knowledge. In summary, our contributions are four-fold.

fever, loss of fever.

fever, loss of taste.

What are the symptoms of COVID-19?

fever, Stomachache, loss of taste.

fever fever is a …

(a) Knowledge Loss

(b) Incorrect Preceding

(c) Knowledge Sinking

We are the first to define and benchmark the CME evaluation setup, paving the way for future model editors to support multi-edit reasoning that is crucial for real-world use cases.

We conduct a systematic analysis of existing methods under CME, revealing three pivotal failure modes in composability, including knowledge loss , incorrect preceding and knowledge sinking .

We develop a dual-stage framework, edit training (adaptive combining and regularizing) and edit composing (adaptive merging), that preserves the effectiveness of individual edits while enabling their synergistic integration during inference.

Through extensive experiments on two datasets and four CME tasks, we improves the composability by at least 22.45%, without sacrificing the performance of non-compositional model editing.

2 Preliminaries

In this section, we first provide a formal definition of compositional model editing, and then demonstrate how to perform edit training and edit composing in FFN. Please refer to Tab. 3 for the definition of notations.

2.1 Compositional Model Editing (CME)

In model editing (ME), let f Θ : X ↦→ Y , parameterized by Θ , represents a model mapping the input x to the output f Θ ( x ) . Given a model Θ 0 before a single edit ( x e , y e ) , let I ( x e ) denote the in-scope input with the same semantics as the edit question. Then the objective of ME is:

Compositional model editing (CME) contains two stages: edit training and edit composing . For the edit training stage, an edit is trained for ( x e , y e ) . For the edit composing stage, the model needs to compose the edit for ( x e , y e ) with other edits at inference time to answer a single question. Let C ( X e ) represent the question that contains the semantics of multiple edit questions X e = { x e , x 1 e , x 2 e , ..., x c -1 e } including x e , where c is the composition number. Let C ( Y e ) represent the target output that contains corresponding multiple answers Y e = { y e , y 1 e , y 2 e , ..., y c -1 e } including y e . The shared objective for the two stages becomes

It is worth noting that, first, the edits involved in edit composing are not necessarily all contained within ( X e , Y e ) , which introduces interference from edits useless to the current problem. Additionally, within our definition, all edits may appear asynchronously over time and be distributed spatially. Therefore, our definition is applicable to both lifelong model editing and federated model editing.

∆ℎ

!"#

$

∆a

1

B

%

B

$

Pre-trained

Weights

Pre-trained

Weights

Pre-trained

Weights

∆ℎ

!"#

%

∆ℎ

!"#

B

$

B

%

∆

a

a

ℎ

&'

ℎ

&'

∆ℎ

!"#

∆

a

a

ℎ

&'

ℎ

&'

∆ℎ

!"#

ℎ

!"#

ℎ

!"#

+

+

A

%

A

$

A

$

A

%

W

"(

∆W

"(

$

W

"(

Pre-trained

Weights

W

)!*'

Edit 1

∆W

)!*'

$

Edit 2

∆W

)!*'

%

+

Pre-trained

Weights

W

)!*'

∆W

)!*'

%

Edit 2

∆W

)!*'

$

Edit 1

+

+

WISE [20]

+

Edit 1

∆W

"(

%

Edit 2

+

Edit 1

Edit 2

∆W

"(

$

∆W

"(

%

MEND [9] & ROME [8] & AlphaEdit [19]

+

+

a

ℎ

&'

ℎ

!"#

W

"(

Pre-trained

Weights

W

)!*'

+

T-patcher [23]

a

ℎ

&'

ℎ

!"#

W

"(

Pre-trained

Weights

Pre-trained

Weights

W

)!*'

ℎ

&'

GRACE [21]

ℎ

&'

ℎ

!"#

W

"(

Pre-trained

Weights

Pre-trained

Weights

W

)!*'

∆ℎ

!"#

+

MELO [22]

∆

a

a

+

B

$

B

%

A

$

A

%

ℎ

&'

∆a

2

2.2 Edit Training and Edit Composing in FFN

Based on the results in Fig. 2, we categorize the factors influencing the composability of edits into Composable and Context-preserving . Whether the edits are composable refers to whether there is significant interference between the trained edits, which impacts knowledge loss and incorrect preceding . Context-preserving, on the other hand, refers to whether the edits affect the model's ability to follow the context to generate other answers, thereby influencing knowledge sinking . In this section, we discuss whether edit training stage of existing methods effectively have these properties and whether their edit composing preserves these properties as shown in Table 1.

Most existing model editing methods insert new knowledge by modifying the parameters or outputs of the FFN layer during the forward process of different tokens. The FFN operates as key-value neuron memories, where the values (rows) in the down projection layer, store pre-trained foundation knowledge obtained from vast amounts of data [24, 25]. During the forward process, it retrieves values from the down projection matrix W down ∈ R n × m by matching the keys in the up projection matrix W up ∈ R m × n with the input h in ∈ R 1 × m :

Here, Act is the activation function (e.g., Relu [28], Gelu [29], and SwiGLU [30]). a is the vector of activation values and serves as the weights to retrieve different pre-trained foundation knowledge in W down. b down ∈ R 1 × m and b up ∈ R 1 × n are bias vectors.

We summarize edit training and edit composing in FFN for existing methods in Fig. 3. Among them, GRACE only edits the forward process of the last input token while achieving contextpreserving. The other methods modify the forward process of all tokens but lack contextpreserving. MEND, ROME, AlphaEdit and WISE modify the entire W down, or W up and W down to insert new knowledge in edit training , thus different ∆ W up and ∆ W down are added together to utilize multiple pieces of knowledge in edit composing . AlphaEdit only edits W down and projects ∆ W down onto the null space of the preserved knowledge including other edits be-

fore applying it to W down to avoid interference between edits. However, in the CME settings we focus on, where a single question depends on multiple edits, this orthogonality becomes ineffective and conflicts between edits still persist. WISE learns more composable edits by masking part of W up and W down in edit training , but the mask on the whole matrix with no carefully designed composing algorithm struggles to keep a large number of edits composable in edit composing . T-patcher concatenates multiple keys A ∈ R m × r and corresponding values B ∈ R r × m to W up and W down respectively. For GRACE, the output h out are replaced with the sum of edit vectors ∆ h out ∈ R 1 × m for multiple pieces of knowledge. MELO trains a LoRA [31] ( A , B ) for each edit, where A ∈ R n × r and B ∈ R r × m . The combination of different knowledge is achieved by concatenating A and B of different LoRAs, respectively. Thanks to the low storage overhead of their low-rank form, GRACE and MELO employ a vector database to store all edits and filter out useless ones during inference,

which achieves more composable edit composing with a large number of edits than others but the results contain more incorrect preceding and are far from ideal.

3 Rethinking the Composability in Model Editing

This section examines existing model editing methods to explore three key questions: How to train composable edits? How to train context-preserving edits? How to compose edits to further enhance composability? Our answers are Finding 1 , Finding 2 and Finding 3 , respectively.

Experimental settings. We conducted analytical experiments using the Llama3-8B model [32] on the PEAK-CF dataset [33] with 50 2-edit composition samples, During inference, we generate 30 tokens for each question, which is enough to output all edited answers. We analyze the performance of existing model editing methods with the metric SR-S = 1 |D test | ∑ |D test | t =1 I ( ∀ a ∈ Y t , rank ( a ) < |Y t | ) , which is the rate of question t in test dataset D test that all edited answers Y t are output before irrelevant content, where rank ( · ) represents the rank of an answer in all answers in the output and |Y t | represents the number of edited answers. Please refer to Appendix H for an example.

Finding 1. The combination of pre-trained foundation knowledge is enough for ME while boosting the composability. As introduced in Sec. 1, W down stores pre-trained foundation knowledge obtained from vast amounts of data. Our hypothesis is that the ability of LLMs to combine different advanced knowledge stems from these well-trained foundation knowledge elements. These elements boosts the composability with a more robust and generalizable knowledge representation from large-scale pre-training. We validate our hypothesis by the average performance of methods editing full matrix and in low-rank form when editing W up or W down at different layers. As shown in Fig. 4(a) and Fig. 4(b), editing W up is sufficient to guarantee the performance compared with editing W down as well as editing both W up and W down. This means the pre-trained foundation knowledge in W down is sufficient to express new knowledge. The generally higher SR-S of editing W up in different layers shown in Fig. 4(c) confirms our hypothesis. For a more comprehensive discussion about the sufficiency of editing W up for higher SR-S and less knowledge loss , please refer to Appendix I.

(a)

(b)

(c)

Finding 2. Less change of the hidden state of the last answer token [la], more context-preserving. We demonstrate in Fig. 5(a) the relationship between the context-preserving rate and the change of the hidden state at the edited FFN layer when inferring with [la] (e.g, 'What are the symptoms of COVID-19? fever'). The results show an evident inverse relationship between these two factors in the baselines, indicating that a smaller change in the hidden state of [la] can achieve better context-preserving. The finding

(a)

(b)

also answers why GRACE achieves context-following, because it has no parametric interference on the forward process of [la] by editing the last input token only.

B

!

B

"

ℎ #

#$

Adaptive   merging

ℎ

%&'

Values in

W

(%)$

A

&

"

A

&

!

a

∆a

1

∆a

2

∆a

mask

a

'

Retrieve values

Edit 1

Edit 2

W

&*

zero

positive

negative

B

!

B

"

ℎ #

#$

ℎ

%&'

Values in

W

(%)$

A &

"

a

∆a

1

Adaptive

regularization

ℒ

+

ℒ

,

Adaptive combination

a

'

Retrieve values

Edit 1

W

&*

edit composing

edit training

Finding 3. Setting the elements with different signs in different edits to zero (Adaptive Merging) during edit composing enhances the composability. The elements with opposite signs between different edits may contain conflicting knowledge. In Fig. 6, inspired by [34], we validate that setting these opposing elements in different edits to zero better balances the knowledge forgetting in the edits and their negative interference on each other. As can be seen, the mean performance of methods editing the full matrix and in low-rank form with Adaptive Merging outperforms directly composing different edits.

4 The Proposed Method

Although the three findings mentioned above contributes the composability of model editing, there still lacks of a unified framework to fully integrate their advantages. How to train composable and context-preserving edits? Based on Finding 1, we adaptively select the pre-trained foundation knowledge using a mask and in a low-rank form to utilize the vector database, namely Adaptive Combination of pre-trained foundation knowledge. We propose two Adaptive Regularization to regularize the use of pre-trained foundation knowledge inspired by Finding 2 while further alleviating incorrect preceding in low-rank form. How to compose edits to further enhance composability? We utilize a vector database to filter out useless edits following [21, 22] while proposing Adaptive Merging for different combinations of pre-trained foundation knowledge inspired by Finding 3. The method is shown in Fig. 7. Please refer to Appendix D for the vector database and Alg. 1-2 for the complete editing progress.

4.1 How to train composable and context-preserving edits?

Adaptive Combination. Inspired by Finding 1, we perform model editing only on W up in a low-rank form similar to LoRA but incorporate an adaptive mask on B . The adaptive mask aims to only preserve the most related pre-trained foundation knowledge to achieve a better trade-off between the composability and learning ability. Specifically, an ( A , B ) pair is trained for each edit following [22]. The forward process of the edited FFN is calculated as:

A ∈ R m × r , B ∈ R r × n are the components of LoRA while ¯ h in and ¯ A are normalized h in and A to avoid the weights of using different B have significant differences and lead to knowledge loss . M ∈ R r × n is the mask to select top k related values in W down, where the correlation is calculated by the dot product between values in W down and the answer tokens' embedding:

Adaptive Regularization. Based on Finding 2, we design a regularization L c to restrict the additional use of pre-trained foundation knowledge by the last answer token [la], thereby enabling the contextpreserving to generate the next answer. Specifically:

where ¯ h [ la ] in represents the normalized input hidden state of the FFN layer at [la]. L c achieves context-preserving by encouraging a low ∆ a in Eq. (4).

In order to further alleviate incorrect preceding in low-rank form, we propose another regularization L o to restrict the logits of tokens h [a] logit except labels by:

where [a] represents the positions of an edit sample with labels, W [a] represents the set of all token IDs except the label of the position [a]. During the edit training stage, we optimize ( A , B ) with the loss:

where L e are cross-entropy loss, α and β are hyperparameters.

4.2 How to compose edits?

Adaptive Merging. During the inference time, after filtering out useless edits by the vector database S db (Appendix D), we need to compose different combinations of pre-trained foundation knowledge and different edits may conflict in the utilization of pre-trained foundation knowledge. Specifically, some edits aim to leverage the j -th value in W down to compose the target answer, so [∆ a ] j tends to be positive. In contrast, other edits may want to erase the knowledge in the j -th row to compose the target answer, so [∆ a ] j tends to be negative. Inspired by Finding 3, when such conflicts occur, we set [∆ a ] j of all used edits to 0 to avoid the overly strong impact on each other. For simplicity, we represent this merging process of different ∆ a as ˆ ∑ and the ∆ a of preserved useful edits by vector database as S . Thus the forward process during inference changes to:

5 Experiment

5.1 Evaluation Benchmarks

Dataset. We utilize the PEAK-CF and PEAK-T datasets [33], which contain a large number of questions with multiple answers. We keep the questions for which Llama3-8B [32] and Mistral-7B

Algorithm 2 The Edit Composing Stage

1: Input The initial LLM f θ , the vector database S db, the test dataset D test .

2: Output The answers of the queries in D test .

3: for each query X t ∈ D test do

4: Retrieve edits and calculate the ∆ a set and S t related to X t with its representation h db (Eq. (9)).

5: Generate the output with replacing h out = FFN ( h in ) (Eq. (3)) with h out = FFNtest ( h in , S t ) (Eq. (8))

6: end for

[35] models have more than four unknown answers, with 1,949 instances in PEAK-CF and 922 instances in PEAK-T. We directly use the rephrase instances and locality instances from PEAK-CF and PEAK-T to test the generalization and locality. Please refer to Appendix G for the specific construction process.

Evaluation metrics. Inspired by [33], we evaluate baselines and our methods with SR-S = 1 |D test | ∑ |D test | t =1 I ( ∀ a ∈ Y t , rank ( a ) < |Y t | ) , which represents the success rate with all edited answers in the output without incorrect preceding . In Appendix H, we provide examples about SR-S while discussing its effectiveness for evaluation and the results of its variants; SR-1 = 1 c |D test | ∑ |D test | i =1 ∑ a ∈Y t I ( a ∈ ˆ Y t ) , which represents the success rate that edited answers a ∈ Y t are in the output answer set ˆ Y t . c represents the composition number, which means c pieces of edited knowledge are needed for a question; GSR = 1 |D rep | ∑ |D rep | t =1 I ( ∀ a ∈ Y t , rank ( a ) < |Y t | ) , but the questions are changed into rephrased ones with the same semantic to evaluate the generalization; LSR = 1 |D loc | ∑ |D loc | t =1 Rouge-L ( ˆ Y loc t , Y loc t ) , which represents the f1 score of Rouge-L [36] between the edited outputs ˆ Y loc t for locality questions and origin output Y loc t . Please refer to Appendix K for examples of the input.

Baselines. We compared our method with FT, ROME, KE, MEND, AlphaEdit, WISE, T-patcher, GRACE, and MELO. Please refer to Appendix G for more details.

5.2 Experimental Results

Different datasets and CME tasks . To evaluate the effectiveness of the proposed A 3 E, we conduct large-scale experiments with all instances of the PEAK-CF and PEAK-T datasets, covering four CMEtasks: independent multi-answer composition (IMAC), independent multi-question composition (IMQC), multi-answer composition (MAC) and multi-question composition (MQC). In independent scenarios, we manually filter out useless edits for each question to directly verify the effectiveness of adaptive combination, regularization and merging without the effect of vector database. In MAC and MQC, the vector database is used to evaluate in real-world scenarios with a large number of edits asynchronously and distributedly. We set the composition number to 2, which means two pieces of edited knowledge are needed for a question. The results shown in Table 2 indicate that A 3 E consistently performs better on four CME tasks across two datasets. For IMAC on PEAK-CF, A 3 E surpasses all baselines by at least 22.45% in SR-S , at least 13.63% in SR-1 and at least 18.36% in GSR . Even on challenging MAC, A 3 E consistently achieves the best editing performance and generalization while maintaining similar level locality. The lead of A 3 E on PEAK-T is even more obvious, with at least 67.16% and 406.73% in SR-S for IMQC and MQC respectively.

Different number of edits. To evaluate the performance of A 3 E throughout the lifelong usage process, we compare the performance among A 3 E, GRACE and MELO, which have relatively strong MACand MQC capability. The results of MAC on PEAK-CF, as shown in Fig. 8(a), indicate that A 3 E maintains the most outstanding performance until 3898 edits. Please refer to Appendix. J.1 for more results.

Different composition number. So far, we have conducted experiments with a composition number of 2. To demonstrate that A 3 E still has superiority when it needs to simultaneously utilize more edited knowledge, we compared the performance of A 3 E, GRACE, and MELO with composition numbers ranging from 1 to 4, each containing 50 ( 50 × 1), 100 ( 50 × 2), 150 ( 50 × 3) and 200 ( 50 × 4)

edits, respectively. As shown in Fig. 8(b), although the SR-S gradually decreases with increasing composition number, A 3 E always maintains a significantly better performance. Please refer to Appendix. J.1 for more results.

Non-compositional ME. A stronger composability should not come at the expense of worse non-compositional ME performance. Therefore, we compare the average non-compositional ME performance of A 3 E and baselines. The results, as shown in Fig. 8(c), demonstrate that our method is on par with baselines in SR-1 , GSR and LSR . We provide non-compositional results of other datasets in Appendix J.6.

Different backbones and prompts. To investigate the impact of different backbones (Llama3-8B and Mistral-7B) and few-shot prompts (please refer to Appendix K) on the proposed method, we conduct experiments in MAC setting with 3898 edits.

(a)

(c)

(d)

(b)

The results in Fig. 8(d) show that: (1) A 3 E has a leading SR-S when using different backbones and different prompts. (2) A 3 E and baselines generally have better editing performance on Llama3-8B. (3) When using different few-shot prompts to combine multiple questions, there is a certain performance difference in A 3 E. This may be due to the fact that, although we manually ensure that few-shot prompts do not contain answers and content obviously related to answers, there may still be some implicit associations between them and the answers. However, this does not affect the significant improvement of A 3 E on the composability of model editing. Please refer to Appendix. J.3 and Appendix. J.4 for more comprehensive results. We also provide more discussions on failure cases in Appendix J.7 and generalization in Appendix J.8.

5.3 Ablation Study

As shown in Fig. 9(a), we conduct experiments in MAC with 3898 edits to evaluate seven ablation versions removing the proposed L o , L c , adaptive combination, adaptive merging, norm and editing only W up, respectively. All the components contribute to the composability of A 3 E. Please refer to Appendix. J.5 for more results. We further demonstrate three advantages of the components.

(a)

(b)

(c)

Advantage 1. Adaptive combination achieves performance beyond the trade-off between composability and learning ability. As can be seen from Fig. 9(b), although adding a random mask can achieve good composability, the larger standard deviation indicates that it is significantly affected by the random seed. Meanwhile, as the unmask size increases, the learning ability gradually improves with the increasing overlap between edits. We overcome this trade-off by using adaptive combination to select more useful pre-trained foundation knowledge.

Advantage 2. L c makes better context-preserving. See Fig. 5(a), by incorporating L c , our method significantly reduces the change in the hidden state at the last answer token compared to our methods without L c , thus improving the context-preserving capability and achieving a level comparable to GRACE. Meanwhile, as shown in Fig. 5(b), our stronger context-preserving capability successfully translates into a higher SR-S .

Advantage 3. Adaptive merging and adaptive combination mutually reinforce each other. In Fig. 9(c), we compare the impact of test-time random masks on B for editing W down and W up in the noncompositional ME scenario, where the mask size is set to the mean size of the parts that require adaptive merging in the CME scenario. It shows that editing W up is less likely to suffer from forgetting caused by the presence of adaptive merging.

6 Conclusion

In this paper, we first define and benchmark the CME task, which aims to evaluate the ability of model editing methods to simultaneously utilize multiple edited knowledge. By revisiting existing model editing methods, we point out their lack of composability manifests as knowledge loss , incorrect preceding and knowledge sinking . To mitigate them, we propose A 3 E to train more composable and context-preserving edits and compose edits with enhanced composability. Extensive experiments under various scenarios, backbones, and prompts demonstrate the effectiveness of the proposed A 3 E.

References

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 , 2019.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910 , 2020.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys , 55(12):1-38, 2023.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164 , 2021.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359-17372, 2022.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309 , 2021.

Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18564-18572, 2024.

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286 , 2024.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172 , 2023.

Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics , 12:283-298, 2024.

Jun-Yu Ma, Jia-Chen Gu, Zhen-Hua Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310.10322 , 2023.

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arXiv preprint arXiv:2401.04700 , 2024.

Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07453 , 2024.

Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, and Lifu Huang. Navigating the dual facets: A comprehensive evaluation of sequential memory editing in large language models. arXiv preprint arXiv:2402.11122 , 2024.

Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. Perturbationrestrained sequential model editing. arXiv preprint arXiv:2405.16821 , 2024.

Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang, Xiangnan He, and Tat-seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355 , 2024.

Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405.14768 , 2024.

Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems , 36, 2024.

Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuronindexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19449-19457, 2024.

Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785 , 2023.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913 , 2020.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696 , 2021.

A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems , 2017.

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems , 28, 2015.

Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 807-814, 2010.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 , 2020.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Jun-Yu Ma, Zhen-Hua Ling, Ningyu Zhang, and Jia-Chen Gu. Neighboring perturbations of knowledge editing on large language models. arXiv preprint arXiv:2401.17623 , 2024.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems , 36, 2024.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74-81, 2004.

Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. arXiv preprint arXiv:2311.04661 , 2023.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Massediting memory in a transformer. arXiv preprint arXiv:2210.07229 , 2022.

Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning , pages 15817-15831. PMLR, 2022.

Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795 , 2023.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021.

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems , 33:512-523, 2020.

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849 , 2022.

Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems , 35:17703-17716, 2022.

Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character , 222(594-604):309-368, 1922.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems , 36, 2024.

Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378 , 2022.

Protection Regulation. Eu general data protection regulation. available at, available at: https://eur-lex. europa. eu/eli/reg/2016/679/oj/(accessed 10 September 2023) , 2016.

Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115 , 2017.

Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai-Wei Chang. Deepedit: Knowledge editing as decoding with constraints. arXiv preprint arXiv:2401.10471 , 2024.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740 , 2023.

Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, and Wei Han. Retrieval meets reasoning: Dynamic in-context editing for long-text understanding. arXiv preprint arXiv:2406.12331 , 2024.

Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, et al. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492 , 2024.

Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages 2056-2066, 2024.

Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, and Weiran Xu. Assessing and post-processing black box large language models for knowledge editing. In Proceedings of the ACM on Web Conference 2025 , pages 1716-1732, 2025.

Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. arXiv preprint arXiv:2401.06855 , 2024.

Zihao Zhao, Yuchen Yang, Yijiang Li, and Yinzhi Cao. Ripplecot: Amplifying ripple effect of knowledge editing in language models via chain-of-thought in-context learning. arXiv preprint arXiv:2410.03122 , 2024.

Changyue Wang, Weihang Su, Qingyao Ai, Yujia Zhou, and Yiqun Liu. Decoupling reasoning and knowledge injection for in-context knowledge editing. arXiv preprint arXiv:2506.00536 , 2025.

Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. Atm: Adversarial tuning multiagent system makes a robust retrieval-augmented generator. arXiv preprint arXiv:2405.18111 , 2024.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Distributed pruning towards tiny neural networks in federated learning. In 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS) , pages 190-201. IEEE, 2023.

Hong Huang, Weiming Zhuang, Chen Chen, and Lingjuan Lyu. Fedmef: Towards memoryefficient federated dynamic pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 27548-27557, 2024.

Hong Huang, Hai Yang, Yuan Chen, Jiaxun Ye, and Dapeng Wu. Fedrts: Federated robust pruning via combinatorial thompson sampling. arXiv preprint arXiv:2501.19122 , 2025.

Hong Huang and Dapeng Wu. Quaff: Quantized parameter-efficient fine-tuning under outlier spatial stability hypothesis. arXiv preprint arXiv:2505.14742 , 2025.

Junda Zhu, Jun Ai, Yujun Li, Yichun Yin, Yasheng Wang, Lifeng Shang, and Qun Liu. RidgeloRA: Matrix ridge enhanced low-rank adaptation of large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025.

Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. arXiv preprint arXiv:2501.13198 , 2025.

Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei. Federated continual learning via prompt-based dual knowledge transfer. In Forty-first International Conference on Machine Learning , 2024.

Appendices Contents

A Notation

B Related Work

Model Editing. Model editing has become a popular technique to correct LLMs without the costly re-training. Existing model editing methods are mainly divided into constrained fine-tuning, metalearning, locating-and-editing, and memory-based methods. Meta-learning methods such as MEND [9] and KE [7] train a hypernetwork to generate model updates, with MALMEN [37] addressing the cancellation effects. Locating-and-editing methods like ROME [8] identify factual associations in the feed-forward layer and update them. Building on this, MEMIT [38] extends it to a batch edit setting, while AlphaEdit [19] projects perturbation onto the null space of the preserved knowledge to achieve better lifelong model editing. To avoid conflicts among numerous edits and the impact on original knowledge, memory-based methods such as SERAC [39], T-patcher [23], GRACE [21], MELO [22] and WISE [20] construct extra working memory in different knowledge preservation forms, from which useless knowledge is filtered out during inference.

Model Editing Evaluation. From single edit to batch edit and then to lifelong edit, the evaluation of model editing methods is becoming increasingly close to real-world applications and more challenging in terms of the ability to handle interference between different edits and with the original knowledge. In addition to the classic assessments of reliability [9, 10], generalization [10, 11], and locality [8, 12], Cohen et al. [13] proposed evaluating the ripple effect of model editing, which means a series of knowledge that needs to be changed along with the new knowledge. Ma et al. [14] evaluated the knowledge inserted into the model bidirectionally, while these works [15, 16, 17, 18] assessed the impact of model editing methods on general downstream tasks. Unlike previous works, we are the first to point out that model editing methods should possess composability, namely enabling the model to use multiple edited knowledge simultaneously. On the one hand, our work extends locality to scenarios where multiple edits are needed for the same input (multi-question composition &multi-answer composition), which needs to avoid interference between useful edits. On the other hand, our work expands generalization in two ways. From the input perspective, we explore how to make an edit generalize to scenarios where only part of the input requires the edit (multi-question composition). From the output perspective, we explore how to make an edit generalize to scenarios when there is some content that corresponds to multiple other edits in the output (multi-question composition & multi-answer composition). This work takes model editing a step further towards more complete answers and more complex questions. It is noteworthy that a seemingly related

work [40] considers the success of a single edit in a multi-hop question with reasoning, instead of compositional question/answers that require multiple edits in our focus.

Model Merging. Model merging aims to merge the weights of different models while retaining their respective knowledge. However, due to permutation symmetry [41] and disjoint optimization trajectories [42], simply averaging two models can lead to a catastrophic drop in their performance. RegMean [43] proposed a closed-form solution by solving a local linear regression problem for each linear layer, but it requires additional data transmission and inference overhead. Fisher merging [44] weighs the parameters in each model with the Fisher Information Matrix [45, 46], but suffers from memory overhead and conflicts during the merging of multiple models. Recently, since different finetuned models initialized from the same pre-trained model effectively share a part of the optimization trajectory, model merging methods based on pre-trained models, such as TIES-Merging [34], task arithmetic in tangent space [47], and ColD Fusion [48], have achieved leading results. Our method draws inspiration from these works but differs in that our proposed compositional model editing aims to enable the model to utilize model merging to use multiple edited knowledge simultaneously during the inference.

C Compositional vs. Collect & Edit

compositional model editing addresses a different scenario where: during the edit training for 'fever' and the edit training for 'loss of taste', they are unaware of each other . This means we cannot collect all of the knowledge to be edited and then directly compose them at text level. We will explain why this scenario is meaningful below.

As stated in Sec. 2.1 (cf. Lines 97-99) and illustrated in Fig. 1, we believe that it is necessary to support model editing in a federated manner. Specifically, when an open-source model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.

In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [49], different deployers are only able to share trained edits . In such cases, when a deployer receives trained edits from other deployers, it does not know the questions of the edits. It is impossible to get complete answers during edit training because:

The deployer has to iterate through all previous edit questions to determine which previous edits need to be composed with the newly received edit to form a complete answer.

However, storing all previous edit questions is impossible in lifelong model editing .

When exchanging raw data is allowed, local deployers still have to perform immediate model editing to mitigate potential adverse effects without waiting to receive errors discovered by other deployers. In this scenario, assuming that each of n deployers updates an answer to the same question and exchanges these n different answers with each other.

-Continuously injecting complete knowledge would require n times edit training for each deployer at most .

-Exchanging independently trained edits would only require 1 time edit training for each deployer .

Training edits independently while ensuring their composability supports model editing in a federated manner much more efficiently . This is especially important in the field of model editing, which emphasizes immediate correction of errors .

Meanwhile, for multi-question composition , compositional model editing is clearly meaningful because,

Edits for any two different questions may need to be composed during testing.

Different errors to be edited may be discovered at different time.

Therefore, it is impossible to exhaustively cover all possible composition cases for complete knowledge during the edit training process.

D The Design of Vector Database

To enhance the edit composing stage, we adapt the vector database in [21, 22] in a simple but effective way. We retrieve useful edited knowledge with a threshold γ to ensure locality and a topp selection to balance the completeness of knowledge and interference from useless edits:

where S represents the retrieved edits from the vector database S db with K ∈ R 1 × m as the key and ( A , B ) as the value. K is saved from a selected hidden state of the edit question. h db ∈ R 1 × m represents the selected hidden state of the test question, which are hyperparameters. We use h db to retrieve edits by matching K . Because of the mask M , we only need to save the unmasked B and the unmasked position. Please refer to Alg. 1 and Alg. 2 for the construction and utilization of S db. Please also refer to Alg. 2 for the complete test procedure.

E Handling Continuous Editing Situations

When adding an edit for 'loss of taste,' if we recognize that the existing edit for 'fever' is outdated, our method can easily remove the outdated edit with the following locate-and-delete algorithm:

Iterate and mask each edit ( A , B ) in the retrieved part S of the vector database during the generation process → locate the edit that leads to the outdated answer.

Delete the located edit.

We consider a three-answer composition scenario with 3 × 50 successful edits to test the effectiveness of the locate-and-delete algorithm above, where each question receives two correct edits and one outdated edit. The accuracy of deleting the outdated edits is 100% . We encourage the construction of outdated edits feedback and avoidance mechanisms when deploying A 3 E.

F Computational and Memory Overhead

In Tab. 4 and Tab. 5, we provide a more detailed analysis of the computational and memory overhead compared to baselines with Llama3-8B and Llama3-70B from four aspects: storage overhead , trainable parameters , extra FLOPs for inference (infer.) , extra FLOPs for training (train.) . We only provide the extra FLOPs for training of T-patcher, GRACE, MELO and A 3 E because other methods that update the whole matrix or train a hypernetwork have significantly larger training overhead. Specifically,

Storage overhead refers to the additional parameters that need to be stored compared to the pre-trained model for 1000 edits.

Trainable parameters denote the parameters that are updated during training for each edit.

Infer. denotes extra FLOPs compared to the pre-trained model for the inference of u tokens with v input tokens.

Train. denotes extra FLOPs compared to the vanilla forward-backward progress with cross-entropy loss for w training steps of an edit with z -token answer.

For Llama3-8B ,

We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:

Adaptive merging of A 3 E: 1 . 64 u × 10 -5 GFLOPs

Adaptive combination of A 3 E: 0 . 03 z GFLOPs

Adaptive regularization of A 3 E: 1 . 28 wz × 10 -4 +1 . 64 w × 10 -5 GFLOPs

In A 3 E, w is set to 50 and z < u for a instance. The extra FLOPs for training and inference are very small compared to the generation of u tokens with v input tokens with an 8B model, which is about 16 u +16 v GFLOPs.

For Llama3-70B ,

We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:

Adaptive merging of A 3 E: 1 . 19 u × 10 -4 GFLOPs

Adaptive combination of A 3 E: 0 . 09 z GFLOPs

Adaptive regularization of A 3 E: 1 . 28 wz × 10 -4 +2 . 87 w × 10 -5 GFLOPs

In A 3 E, w is set to 50 and z < u for a instance. The extra FLOPs for training and inference are very small compared to the generation of u tokens with v input tokens with an 70B model, which is about 140 u +140 v GFLOPs.

G Experiment Details

For the baselines, we follow the reproduction code and hyperparameters as described in [11]. For A 3 E, we set the unmasked size k to 896 for PEAK-CF and to 448 for PEAK-T. We set the loss weight α to 8 and β to 1 to balance their utilities. We store the output of the 5-th down projection layer at the last subject token as K and h db. We set the edited FFN layer to 31, the learning rate to 0.01, and train each edit for 50 epochs. To better evaluate the priority of different answers within the model, for all baselines and A 3 E, we assign a penalty of 10 to the answers that have already been generated. How to achieve effective compositional model editing under both instruction-tuned models and conventional inference settings remains an area requiring urgent exploration. For GRACE, MELO and A 3 E, we use the vector database in Appendix D with the same p = 4 and γ = 0 . 5 . The experiments are conducted on a server with 8 NVIDIA RTX 5880 Ada GPUs.

For the benchmark construction with PEAK-CF as an example: (1) The original PEAK-CF is a model editing dataset containing questions with multiple answers. (2) We queried Llama3-8B and Mistral-7B, selecting questions from PEAK-CF where both models missed four or more answers, along with their corresponding missing answers in PEAK-CF. (3) For multi-answer composition, the retained questions and their randomly selected c missing answers form a test instance, where c is the composition number. (See Tab. 17 for an example) (4) For multi-question composition, we randomly selected c retained questions without repetition and one randomly chosen missing answer per question to form a test instance. (See Tab. 19 for an example)

H Discussions of SR-S

Fig. 10 shows a correct example and a false example of SR-S . SR-S aims to accurately measure the interference between different edits to evaluate the composability without considering the case of outputting unrelated answers after all correct answers. This is to mitigate the confounding effect of conventional model editing, where 'outputting unrelated answers after correct answers' is also a problem, and warrant fairness in evaluating composability. We report the metric SR-AS when 'outputting unrelated answers after all correct answers' is considered as an error in Table 6. A 3 E maintains a clearly superior performance.

fever, loss of taste. What are the symptoms of COVID-19?

fever, Stomachache, loss of taste.

(a) SR-S

fever, loss of taste, Stomachache.

fever, loss of taste.

What are the symptoms of COVID-19?

fever, loss of taste.

fever, loss of taste, Stomachache.

fever, Stomachache, loss of taste.

(b) SR-AS

I The Sufficiency of Editing W up

I.1 For higher SR-S

From an intuitive perspective , new information does not appear out of thin air, it will be related to past information to some extent.

From an empirical perspective , we have shown in Fig. 4(a) that using existing columns to form new knowledge (edit W up) can achieve results comparable to or even slightly better than editing W down alone, as well as editing both W up and W down simultaneously. In Fig. 4(b), we conduct the experiment above with fake relation in the input (e.g. share border with -> share boodee with) and fake answer (e.g. Syria -> Sykie) to simulate unseen knowledge. It can be seen that the performance does not change a lot. These experiments indicate that editing W up is enough to guarantee the performance.

From a mathematical perspective , W down is often of full rank, which to some extent guarantees its expressive power in its output space.

I.2 For less knowledge loss

From an intuitive perspective , the distinction between pretrained knowledge and edited knowledge from existing model editing methods lies in the fact that pretrained knowledge is the combination of pretrained foundation knowledge in W down, whereas existing model editing methods directly manipulate the output of W down. Meanwhile, we know that pretrained models can leverage pretrained knowledge to answer questions requiring multiple answers, but when using multiple edited knowledge, they exhibit significant knowledge loss. Therefore, we intuitively hypothesize that knowledge loss may partly stem from the fact that edited knowledge is not the combination of pretrained foundation knowledge.

From an empirical perspective , we observe that editing W up (i.e., combining pretrained foundation knowledge into edited knowledge) results in less min knowledge loss across 27-31 layers and across baselines compared to editing W down or editing both W up and W down in Tab. 7 while leading to higher SR-S and similar non-compositional performance (Fig. 4).

Table 7: Min knowledge loss across 27-31 layers and across baselines when editing W up, W down, W up and W down.

J Additional Experimental Results

J.1 The effect of the composition number and the number of edits

In Fig. 11, we provide complete experimental results for the effect of different number of edits and the composition number in MAC and MQC on PEAK-CF and PEAK-T. In PEAK-CF dataset, we also conducted additional experiments by sampling 50 questions with 10+ unknown answers each and evaluated the performance with a composition number of 10.

As shown in Tab. 8, the experimental results demonstrate that when a large number of edits need to be applied simultaneously, A 3 E still significantly outperforms the baselines on the SR-1 metric, which means A 3 E retains more edited knowledge with large composition numbers.

120

120

120

120

120

120

120

120

SR-1 (%)

100

80

MELO

Ours

GRACE

GRACE

MELO

Ours

100

80

(%) S-#S

SR-1 (%)

100

80

GRACE

MELO

Ours

GRACE

MELO

100

80

SR-S (%)

SR-1 (%)

100

MELO

GRACE

Ours

GRACE

MELO

Ours

100

80

SR-S (%)

SR-1 (%)

100

GRACE

MELO

Ours

GRACE

MELO

100

80

SR-S (%)

60

60

60

60

60

60

40

40

40

40

40

40

40

40

20

20

20

20

20

20

20

20

0

1

2

3

4

0

0

1

2

3

4

0

0

1

2

3

4

0

0

1

2

3

4

0

Composition number

Composition number

Composition number

Composition number

(a)MAC onPEAK-CF

(b) MQC on PEAK-CF

(c)MAC onPEAK-T

(d) MQC on PEAK-T

120

120

120

120

120

120

120

120

SR-1 (%)

100

80

Ours

MELO

GRACE

MELO

GRACE

Ours

100

80

(%) S-S

(%) -HS

100

00 6040

MELO

GRACE

Ours

GRACE

MELO

sno

100

80

(%) S-#S

(%) [-S

100

00 00 40

GRACE

MELO

Ours

GRACE

MELO

Ours

100

80

(%) S-8S

SR-1 (%)

100

80

GRACE

MELO

Ours

GRACE

MELO

Ours

100

80

(%) S-S

604

6

60

60040

604

40

40

20

20

20

20

20

20

20

20

0

10

Number of edits

100

500

3898

0

0

10

Number of edits

100

500

1948

0

0

10

Number of edits

100

500

1984

0

0

10

Number of edits

100

500

992

0

(e)MAC onPEAK-CF

(f)MQC on PEAK-CF

(g)MAC onPEAK-T

(h) MQC on PEAK-T

J.2 The effect of hyperparameters

The hyperparameters of A 3 E are not sensitive and are easy to adjust. First, as shown in Fig. 12(a)12(d), compared with the performance improvement of A 3 E, its performance is not sensitive to the selection of α , β , γ , and p . Second, the adjustment of hyperparameters is straightforward for two reasons: 1) As shown in Fig. 12(a)-12(e), all hyperparameters exhibit similar behavior across different datasets (i.e., PEAK-CF and PEAK-T), indicating that a one-time hyperparameter search is almost sufficient, and repeated re-tuning can be avoided. 2) As shown in Fig. 12(a)-12(b), there exist correlations among certain hyperparameters, e.g., setting α = 8 β often leads to the best performance. This enables joint tuning, thereby reducing the number of hyperparameters that need to be adjusted.

(a)

(b)

(e)

(d)

(c)

60

58

56

54

52

50

48

46

44

54

52

50

48

46

44

42

40

38

J.3 The effect of different backbones

Table 9 shows the performance with backbone Mistral-7B in four scenarios on two datasets.

J.4 The effect of different prompts

Table 10-11 shows the performance of A 3 E and state-of-the-art baselines WISE, GRACE, MELO with different prompts in four scenarios on two datasets.

J.5 Ablation study

In Table 12, we provide the comprehensive results of ablation study in four scenarios on two datasets.

J.6 Non-compositional ME evaluation on classic datasets for ME

As shown in Tab. 13, the non-compositional performance of A 3 E is on par with baselines.

J.7 Failure cases analysis

We conducted failure case analysis for overlapping, contradictory, and capacity-exceeding edits as follows:

For overlapping and contradictory cases , we conducted statistical analysis on the PEAK-CF dataset under the IMAC scenario with 50 cases of two-edit composition, measuring:

-The overlapping selection of pre-trained foundation knowledge, calculated by the overlapping length between non-zero ∆ a (Sec. 4.1) regions of two edits in adaptive merging (Sec. 4.2)

-The contradictory utilization of pre-trained foundation knowledge, calculated by the length of regions with different signs between ∆ a (Sec. 4.1) of two edits in adaptive merging (Sec. 4.2)

The table below presents how the SR-S metric, which is inversely proportional to the proportion of failure cases, varies across different overlapping and contradictory ranges. As the overlapping length and contradictory length increase, the SR-S of A 3 E gradually decreases, but it remains significantly higher than the baselines.

For capacity-exceeding cases , we found that the failure cases of baselines and A 3 E in both compositional model editing and non-compositional model editing scenarios often involve numerical data in answers. For example,

-Edit 1 : Rheinmetall, which has designed 12 cm leFH 08

Test 1 : Rheinmetall, which has designed 12 cm leFH 8 (failure)

-Edit 2 : Rheinmetall, which has designed 35.5 cm Haubitze M1

Test 2 : Rheinmetall, which has designed 35.5 cm Haubitze M1 (success)

Composition test : Rheinmetall, which has designed 12 cm leFH , 12 cm Haubitze M8 (failure)

This may stem from the model's inherent inability to accurately distinguish numerical data. Therefore, addressing such inherent limitations of the model is a valuable direction for future work in model editing.

We believe these findings can guide future research to further improve edit composability.

J.8 Generalization

The potential generalization limitations are as follows:

Noisy scenarios . In Fig. 8(d), we tested the impact of different few-shot prompts on A 3 E's performance. These varied few-shot prompts not only controlled the model's output but also simulated different noise present in the context. The results show that while A 3 E's performance is affected, it still maintains a leading performance compared to the baselines.

Noisy open-domain scenarios . We humbly acknowledge that in open-domain scenarios, noisy contexts may still impact the retrieval accuracy of the vector database. Since A 3 E primarily focuses on edit training and edit composing, we will explore

-optimizing the hidden state positions for retrieval based on the real data distribution in application scenarios

-employing more powerful embedding models

-training a question-rewriting module to better align with the vector database

for optimal performance in the future work.

Adversarial scenarios . We conducted additional experiments using 2 × 50 successful edits from the PEAK-CF dataset under the IMAC setting by providing the tested question with incorrect answers in the few-shot prompts. The results demonstrate that A 3 E consistently (100%) outputs the edited answers across all test cases, proving its robustness against contextual interference.

Larger models and black-box models . A 3 E is applicable to larger LLMs. Due to limitations in computational resources, we provide experimental results for GRACE, MELO, and A 3 E under the MACscenario using Llama3-70B and the PEAK-CF dataset. As shown in Table 18, A 3 E maintains the leading performance.

However, A 3 E cannot be applied to black-box models. In the field of knowledge editing for large language models, one research direction focuses on editing white-box models, i.e., model editing [22, 21, 20, 19]. The access to model weights is a prerequisite for applying these editing techniques. Another research direction addresses knowledge editing for extremely large-scale models where model editing may be prohibitively expensive or black-box models by performing edits at the text level, including [51, 52, 13, 40, 53, 54, 55, 56, 57, 58, 59, 60]. A 3 E follows the first

direction to edit white-box models and focuses on compositional model editing because text-level knowledge editing is not applicable for the following reason:

-As stated in Sec. 2.1 and illustrated in Fig. 1, we aims to support model editing in a federated manner [61, 62, 63, 64]. Specifically, when a model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.

-In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [49], different deployers are only able to share trained edits rather than the texts.

In a centralized scenario, A 3 E can also be implemented by the owner of black-box models for immediate error correction with much more composable edits. We recommend users to choose methods from either direction based on their needs, and we will explore how to perform compositional knowledge editing at the text level for black-box models in our future work.

K Description of Dataset and Few-shot Prompts

In Table 17 and 19, we provide examples of the simultaneous use of two pieces of knowledge in the MAC and MQC scenarios, respectively. In Table 20, we provide another two prompts to test the effect of different prompts.

L Limitations

Although the proposed method significantly enhances the composability of model editing, it still falls short of achieving fully accurate composition. Future research will continue to explore the factors that influence composability and further improve the effectiveness of compositional model editing. The performance of instruct models under penalty-free decoding condition also needs to be explored. Additionally, as this paper primarily focuses on how to train and compose edits with composability, it does not delve deeply into the organization of vector databases. Future work will aim to continuously

optimize the structure of vector databases. The effectiveness of other PEFT methods [65, 66, 67, 68] need to be explore as well.

M Broader Impact

This paper presents work whose goal is to advance the field of model editing. Model editing is a cutting-edge concept that holds immense significance in a variety of critical and rapidly evolving fields, including but not limited to medical or legal fields. In these scenarios, mistakes of models are found after deployment, thus a timely and effective correction is needed. Our research has taken a step towards more available model editing by allowing multiple edits to be used simultaneously, leading to safer and more ethical applications across a broad spectrum of industries.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Appendix L.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Implementation details are provided in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Release the codes upon acceptance of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Model editing on a specific example is determinative when no additional randomly initialized parameters are introduced. In this case, there is no associated error bar. We provide results in different datasets, models and prompts.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computer resources are provided in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: For the pre-trained models and the datasets used in this paper, we check them manually to ensure the safety.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors cite the original paper that produced the pre-trained models and datasets used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We conduct the experiments of model editing on LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Compositional Model Editing (CME), where multiple edits must be integrated into LLMs. Existing methods fail due to knowledge loss, incorrect precedence, and knowledge sinking. The authors propose A³E, an adaptive editor that combines and merges edits effectively, improving compositional performance by 22.45% without hurting single-edit accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The compositional model editing is a interesting direction and worth to explore.\n- They propose a new benchmark for compositional model editing.\n- They investigate the issues of existing model editing techniques.\n\nWeakness: \n\n- The first thing that comes to mind is **when** we need composition and **when** we don’t. In some cases, composition is necessary; in others, we may need to revert previously edited knowledge. A better approach for compositional model editing might be to inject complete knowledge into the model and erase prior incomplete knowledge. For example:\n  - **Q**: What are the symptoms of COVID-19?  \n    - **Edit 1**: \"Fever\"  \n    - **New correct edit**: \"Fever, loss of taste\" (rather than adding \"loss of taste\" as a separate, independent edit)\n\n- How do you handle continuous editing situations, when previously edited knowledge needs to be changed? For example:\n  - **Q**: What are the symptoms of COVID-19?  \n    - **Edit 1**: \"Fever\"  \n    - Then, we want to update the answer to \"loss of taste\" (not \"Fever, loss of taste\")\n\n- Additionally, I’m not sure how you evaluate the baseline methods, since the details are not included in the paper or appendix. I assume you perform two sequential edits and test whether the model can combine both. If that’s the case, I don’t think it’s a fair comparison, since most baseline methods would treat the latest edit as the new ground truth rather than combining edits. As mentioned in the first point, a better editing strategy would be to erase the previous, incomplete knowledge and then insert the updated, correct information.\n\nQUESTIONS:\nI have covered the questions I want to ask in the weakness section. I would increase my rating if the authors can address the concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Compositional Model Editing (CME), where large language models must integrate multiple knowledge edits to answer complex, real-world questions. Existing model editing methods are shown to struggle in this setting due to issues like knowledge loss and incorrect preceding. To address this, the authors introduce $A^3E$, a two-stage editing framework that improves edit composability through adaptive combination, regularization, and merging techniques. Extensive experiments demonstrate that $A^3E$ significantly outperforms prior methods in compositional settings without reducing standard editing performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper identifies gaps between current model editing and real-world needs by proposing the Compositional Model Editing (CME) task, which is both novel and practically important.\n2. $A^3E$'s architecture thoughtfully aligns with the challenges identified. Each of the three components (adaptive combination, regularization, and merging) corresponds to the failures of prior works.\n3. The method shows large and consistent improvements in compositional settings while preserving performance in non-compositional tasks.\n\nWeaknesses:\n1. While Figure 7 summarizes the core components of $A^3E$, it presents them in a very abstract and schematic way. The figure lacks concrete examples or intuitive annotations, which makes it difficult to understand how each module actually works in practice. Adding a running example would make the method significantly more accessible.\n2. In the introduction, terms like Incorrect Preceding and Knowledge Sinking are introduced as distinct issues but not clearly differentiated. Readers may struggle to understand how they manifest differently in outputs, especially without clearer examples or contrasting definitions.\n3. The approach may not generalize well if the number of required edits for a query becomes very large (e.g., 10+), but this scenario is not deeply explored.\n\nQUESTIONS:\n1. Could you clarify the distinction between Incorrect Preceding and Knowledge Sinking?\n2. How is the adaptive mask in the Adaptive Combination module computed in practice?\n3. You propose setting conflicting $\\delta a$ elements to zero. Could you clarify do you apply this merging at the level of each token, each edit, or batch-wise? I get a little lost in this part.\n\nLIMITATIONS:\nNo.\n\nSuggestions for Improvement:\n1. Discuss generalization limits. It would be helpful to mention how it might perform in more noisy, open-domain, or adversarial scenarios.\n2. The paper lacks a thorough analysis of how the method scales to larger numbers of edits per query (e.g., 10 or more).\n3. The performance is much higher than the baselines. I suggest you check if there are any mistakes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of existing model editing methods for large language models (LLMs), which are typically evaluated and optimized for single, isolated edits, ignoring the real-world need for compositional model editing (CME), i.e., the ability to integrate and utilize multiple edits coherently. The authors define and benchmark the CME task, identifying key issues in existing methods: knowledge loss, incorrect precedence, and knowledge sinking. They further propose an editing framework that adaptively combines and regularizes the pre-trained knowledge to support future composability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The CME task is well-motivated and practically relevant, addressing an important gap in the evaluation of model editing: real-world queries often require combining multiple edited facts.\n2. The work is conceptually clear and innovative, introducing adaptive mechanisms both during training and at test-time to enhance composability.\n3. Extensive experiments on thousands of edits (up to ~4k), different datasets, noise settings, model backbones (Llama3-8B, Mistral-7B), and prompts.\n\nWeaknesses:\n\n1. In the experiments, the authors analyzed the non-compositional ME performance of the proposed framework on PEAKCF. However, such non-compositional ME evaluation should be conducted on other classic datasets for ME.\n2. The considered LLMs only include Llama3-8B and Mistral-7B, which could be insufficient for a comprehensive evaluation, as the model size is not generally large. Furthermore, the authors did not discuss whether the method can be applied to black-box LLMs and how to adapt this method to such LLMs.\n3. The paper does not discuss the failure cases of editing, e.g., if edits are contradictory, overlapping, or beyond the model’s capacity. Such an analysis could be valuable for understanding its robustness.\n\nQUESTIONS:\nHow is the method applicable to larger LLMs and/or black-box LLMs?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for compositional model editing where multiple edits need to be integrated simultaneously to answer complex questions effectively. The proposed method is inspired by their findings that (1) the combination of pre-trained foundation knowledge is enough for model editing while boosting composability (which inspires the method they call *adaptive combination*), (2) the less change in the hidden state of the last answer token, the more context preserving is the model (which inspires the method they call *adaptive regularization*), (3) setting elements with different signs in different edits to zero during edit composing improves composability (which inspires the method they call *adaptive merging*). The paper then conducts empirical evaluation of the proposed method over various datasets, showing improved performance in compositional editing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the paper introduces a new task that is compositional model editing, which evaluates scenarios where multiple edits must be applied to answer a complex question.\n- the paper conducts extensive experiments across multiple datasets \n\nWeakness:\n- the main issue of the paper is its lack of readability. The description of the proposed method is very complex and dense with heavy reliance on prior works and appendices, which makes the paper very hard to read and the contribution hard to appreciate. For example, many of the variable names (W_down, W_up) are assumed to be common knowledge and not defined properly, referred to instead as the 'down projection' and 'up projection' matrices. Variables such as A and B are used instead of the commonly used K and V to refer to the key and value vectors in transformers. Given that new variables are used instead of the commonly used variable names, they should be defined more clearly. \n- in addition, many of the crucial details are missing. For example, the steps for creating the benchmark dataset from existing datasets are unclear. (e.g., which part of the benchmark creation is done manually?). This not only makes the paper hard to read but also reducing its reproducibility. \n- the explanation of the method is not intuitive. To add to the difficulty, the naming of the different parts of the proposed method adds more to the confusion. For example, while the name 'adaptive combination' and 'adaptive merging' refer to different processes in the pipeline of their proposed method, the word 'combination' and 'merging' are similar. Adding more intuitive examples instead of architectural diagram or performance chart will make the proposed concepts more accessible. \n- it is unclear how they identify and select the three critical problems faced by model editing methods: knowledge loss, incorrect preceding, and knowledge sinking. Is it based on prior studies or are they first defined in this work? why the choice of these three problems only? are these commonly used terms/problems in compositional studies? It will be good to motivate this choice for readers unfamiliar with this field of research. \n- the choice of the problems and the solutions seems heavily empirical. For example, how do the authors connect knowledge loss to \"Finding 1\"? Is knowledge loss caused by insufficient combination of the pre-trained foundation knowledge in W_down? How can one come to this conclusion by only looking at performance Figure 4? All I can see from this figure is that updating W_up improves performance. But whether it is because there's less knowledge loss is not clear. It is unclear how the authors connect problems to the cause to the solution. Is there a missing link not detailed here? Because of these gaps in explanation, it is hard to appreciate the contribution. Having more theoretical analysis or even more intuitive motivation that connects problems to observations to the proposed solution can deepen the understanding and appreciation of the proposed method beyond the engineering of the method via empirical trials. \n- the computational overhead is discussed only briefly. Given the complexity of the method proposed, more detailed exploration of the computational and memory cost involved is needed.\n\nQUESTIONS:\n- can you give step-by-step walk through on how the benchmark dataset is constructed?\n- can you improve on the readability of the paper by making the definition of the variables clear (i.e., not assuming that it is self-explanatory) and the paper more self-contained?\n- how do you identify and select these three critical problems faced by model editing methods: knowledge loss, incorrect preceding, and knowledge sinking? Are they based on prior studies or are they first defined in this work? why the choice of these three problems only? are these commonly used terms/problems in compositional studies? \n- how do you make the connection between knowledge loss and \"Finding 1\"? Is knowledge loss caused by insufficient combination of the pre-trained foundation knowledge in W_down? How can one come to this conclusion by only looking at performance Figure 4? \n- can you discuss more on the computational and memory overhead of the proposed method compared to baselines?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}