A$^3$E: Towards Compositional Model Editing
Authors: Hongming Piao
Hao Wang
Dapeng Wu
Ying Wei
Venue: NeurIPS 2025 poster ()
Keywords: Model Editing
Primary area: general_machine_learning

Abstract: Model editing has become a *de-facto* practice to address hallucinations and outdated knowledge of large language models (LLMs). However, existing methods are predominantly evaluated in isolation, i.e., one edit at a time, failing to consider a critical scenario of compositional model editing, where multiple edits must be integrated and jointly utilized to answer real-world multifaceted questions. For instance, in medical domains, if one edit informs LLMs that COVID-19 causes "fever" and another that it causes "loss of taste", a qualified compositional editor should enable LLMs to answer the question "What are the symptoms of COVID-19?" with both "fever" and "loss of taste" (and potentially more). In this work, we define and systematically benchmark this compositional model editing (CME) task, identifying three key undesirable issues that existing methods struggle with: *knowledge loss*, *incorrect preceding* and *knowledge sinking*. To overcome these issues, we propose A$^3$E, a novel compositional editor that (1) ***a**daptively combines and **a**daptively regularizes* pre-trained foundation knowledge in LLMs in the stage of edit training and (2) ***a**daptively merges* multiple edits to better meet compositional needs in the stage of edit composing. Extensive experiments demonstrate that A$^3$E improves the composability by at least 22.45\% without sacrificing the performance of non-compositional model editing.

A E: Towards Compositional Model Editing
Hongming Piao
Hao Wang
City University of Hong Kong hpiao6-comy. cityu. eey..uudu.b... edu. .kk
City University of Hong Kong hao wangōmy cit cityu. edu hk
Dapeng Oliver Wul City University of Hong Kong dapengvu@cityu edu. hk
Ying Wei Zhejiang University ying. wei@zju edu, cn
Abstract
Model editing has become de-facto practice address hallucinations and outdated knowledge of large language models (LLMs). However, existing methods are predominantly evaluated in isolation, i.e., one edit at time, failing to consider critical scenario of compositional model editing, where multiple edits must integrated and jointly utilized to answer real-world multifaceted questions. For instance, in medical domains, if one edit informs LLMs that COVID-19 causes "fever" and another that it causes "loss of taste", a qualified compositional editor should enable LLMs to answer the question "What the symptoms of COVID- with both "fever" and "loss of taste" (and potentially more). In this work, we define and systematically benchmark this compositional model editing (CME) task, identifying three key undesirable issues that existing methods struggle with: knowledge loss, incorrect preceding and knowledge sinking. To overcome these issues, we propose A a novel compositional editor that (1) adaptively combines and adaptively regularizes pre-trained foundation knowledge LLMs in the stage of edit training and (2) adaptively merges multiple edits to better meet compositional needs in the stage of edit composing. Extensive experiments demonstrate that A improves the composability by at least 22.45% without sacrificing the performance of composiisiil model editing. The code is available at /httts:://// thub. cww com/piachongmingg//
1 Introduction
LLMs learn extensive knowledge from massive pre-training corpora and utilize the learned knowledge during inference meet wide range of tasks 2
3 4 Despite their impressive capabilities,
especially as the pre-training data and model size scale 5 LLMs remain prone to factual hallu- cinations (6) and outdated knowledge 77). These errors often emerge gradually after deployment and significantly impairs system reliability. which necessitates timely and effective corrections. Conventional solutions such re-training
not only time-consuming, but fine- fine-tuning without access to pre-training data also poses a high risk of interfering with irrelevant knowledge. Therefore, model editing [8] has emerged as a promising approach that enables targeted updates LLMs in a data-efficient manner (minimal reliance on pre-training data), localized scope (limited impact on irrelevant knowledge), and with high reliability (accurate correction of the targeted knowledge).
Efforts toward model editing center around improving reliability 9 10 generalization 10 and locality 8 [2], with further studies exploring editing ripple effects Progress along with these attempts is
[K] bidirectional generalization
[1], and its impact on downstream tasks 15
16
18
"ooresponniig authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Jack lorged an idemiity dovumnai.. Which article of the Caiiiall Law and Public Security Administration
What are the symptoms COVID-19?
Forging an ideniity documeet the Criminal Law 25- Article 280
COVIn-19 S0r Jever
Punishments Law did he violate?
w/o compusability fever, loss of fever.
we composability
Article 280, Article 28
furging an identity document in the Public ... Law 30- Ao--Aiille 26
COVIN-19 por Joss lasil
w/ composabilliy fever, loss of mese.
w/ compusability
Article 250, Amiile
edit training
edit compasing
edit training
edit composing
(a) Musti-annswr composition (MAC)
(b) Multi-question composition (MQC)
Figure 1: There are two stages in CME: edit training and edit composing. (a) An example for MAC: Multiple symptoms for a disease. (b) An example for MQC: Multiple laws involved in one crime.
unanimously evaluated under a single-edit assumption; that is, LLMs recall each atomic edit in isolation. For example, if an edit previously taught an LLM that COVID-19 causes "loss of taste", evaluations only test whether this LLM can correctly recall this isolated fact, e.g.. "I have loss of taste, do I have COVID-19?" Unfortunately, this evaluation paradigm diverges significantly from world demands. In many practical domains, such as medicine and law (as shown in Fig. D questions likely require access to multiple edits simultaneously. In Fig. [a), for instance, answering multi-symptton diagnostic question requires composing two picces of knowledge (multi-answer composition, MAC); if these facts were introduced through prior edits, LLMs must effectively leverage both. Similarly, in Fig. [b), questions may require reasoning over multiple pieces of edited knowledge to form a unified answer anulti-question composition, MQC). In this paper, we formally define and benchmark tasks that demand such compositional capability as Compositional Model Editing (CME) tasks. The success of CME hinges on two criteria: (1) in the edit training stage, individual edits must be performed in a way that guards their composability for downstream inference; (2) in the edit composing stage, question-related edits must be effectively merged.
As shown in Fig. 2 state-of-the- of-the--rtttttttt- editing methods suffer from knowledge loss, incorrect preceding and knowledge sinking. which comprehensively encompass all the failure cases observed. MEND [ and ROME [ modify the entire matrix in the feed-forward netwok (FFN) through hypernetworks and closed-form solutions, respectively, which yields significant knowledge loss when composing different edits. AlphaEdit [19] exhibits a similar level of knowledge loss, likely because it prevents interference between edits by projecting each update into the null space of unrelated questions. However, in the MAC and MQC settings we target, where a single question depends on multiple edits, this orthogonality collapses and conflicts between edits still persist. WISE (0) alleviates knowledge loss by randomly masking parameter updates of each individual edit, though the likelihood of mask overlap grows significantly as the number of edits increases and thus composability degrades. GRACE [21], MELO (22), and T-patcher (23) replace hidden states or insert new key-value pairs in the FFN layer in a low-rank form. Such low storage overhead for each edit enables them treat all edits as an external vector database attached to the original parameters. Despite reduced knowledge loss, all three methods suffer from the pronounced issue of incorrect preceding. Collectively, these limitations reveal a pressing gap, i.e., how to train and compose different edits to improve edit composability?
Our response to the question is the proposed A Specifically, previous works 24 25
have pointed
out that the FFN layer in Transformer (26 is a key-value neural memory (27. where the down projection matrix stores robust and generalizable foundation knowledge pre-trained on large datasets. The composition of these foundation knowledge pieces forms multiple advanced knowledge needed in the generation process of LLMs. We hypothesize and empirically verify that leveraging the pre-trained foundation knowledge in the down projection matrix is sufficient for not only composing the new knowledge but also boosting composability. Thus, in order to train edits with better composability, we Adaptively combine the pre-trained foundation knowledge in the down projection matrix to reduce knowledge loss. By Adaptively regularizing the use of selected knowledge at the last answer token and non-label logits, we effectively alleviate knowledge sinking and incorreci preceding. In order to compose edits to further enhunce composability, we utilize a vector database [2]] 22 to filter out irrelevant edits a question-wise way at inference and Adaptively merge relevant edits with proper combinations of pre- pre-trained foundation knowledge. In summary, our contributions are four-fold.

---

What are the symploms of COVID-19? fever, loss of taste.
lever, loss of lever.
fever, Stomachache, loss of taste.
lever lever
100
Kooeledga Lass (S4)
a0-
Iooooca Procosing
80
40-
40
20
HERD
ROME
GRAicE
Obers
ROME
Alpher
Knowledge Loss
(b) Incorrect Preceding
(c) Knowledge Sinking
Figure 2: The illustration of knowledge loss, incorrect preceding and knowledge sinking. The text in blue, red, green represents the first correct answer, the wrong answer and the second correct answer. (a) Knowledge loss: Part of answers are not correctly generated. (b) Incorrect preceding: Wrong answers are generated before the correct answers. (c) Knowledge sinking: Context-irrelevant content are generated after prior answers; that is, starting point for generating subsequent answer "loss of taste" is no longer the question itself (e.g., "What are the symptoms of COVID-19?"), but the question combined with the prior answers (e.g.. "What are the symptoms of COVID-19? fever").
We are the first define and benchmark the CME evaluation setup, paving the way for future model editors to support multi-edit reasoning that is crucial for real-world use cases.
We conduct a systematic analysis of existing methods under CME, revealing three pivotal failure modes in composability, including knowledge loss, incorrect preceding and knowledge sinking.
We develop dual-stage framework, edit training (adaptive combining and regularizing) and edit composing (adaptive merging), that preserves the effectiveness of individual edits while enabling their synergistic integration during inference.
Through extensive experiments on two datasets and four CME tasks, we improves the composability by at least 22.45% without sacrificing the performance non-compositional model editing.
2 Preliminaries
In this section, we first provide formal definition of compositional model editing, and then demon- strate how to perform edit training and edit composing in FFN. Please refer to Tab. for the definition of notations.
2.1 Compositional Model Editing (CME)
In model editing (ME), let fe X + Y, parameterized by O, represents model mapping the input x to the output fe()). Given a model O, before a single edit (æe. ye). ). let I(xe) denote the in-scope input with the same semantics the edit question. Then the objective of ME is:
fe. ME (Jer Fes He), s.t. Se. (x)
V (z)
if I(xe). if & I(re).
(1)
Compositional model editing (CME) contains two stages: edit training and edit composing. For the edit training stage, an edit is trained for (Ie, ye). For the edit composing stage, the model needs to compose the edit for (It, ye) with other edits at inference time answer single question. Let C(X.) represent the question that contains the semantics of multiple edit questions X. = {æe, w 22 ..., _x--1 including Te. where is the composition number. Let C():) represent the target output that contains corresponding multiple answers Ve- {ye, Ve. v....., Ve including ye-
The shared objective for the stages becomes
C(Y.)
if E I(C(VX.)). fe, (z) else
fe. CME (Jeo, Tes Ye), S.C. (o)
worth noting that, first, the edits involved in edit composing are not necessarily all contained within (i)), which introduces interference from edits useless to the current problem. Additionally, within our definition, all edits may appear asynchronously over time and be distributed spatially. Therefore, our definition is applicable to both lifelong model editing and federated model editing.
3

---

Appatad
Pro-rained Weight Nawr
Edit AM
Bdi aW
Fre tra-traned Woights Nawe
Edli ailana
Pre-isited Weazhte Wines d
Protiisoal Waust Waswe
Protussed Weezhes Maxes @
Sillay
Au As
da
4B
Pre-maieed Wigght
Litel AWg
Fdlt
Pr-tesinel Wicglts
Prestainal Wagee
Pre-kaneed Wengher
Prettainal mangtts
Wh
Wap *
Wap
Wep
The visualization of different baselines, where MEND, ROME, AlphaEddit and WISE edit either W down or both Wup and W down with or without mask. In contrast, GRACE, T-patcher, and MELO edit the W down or both W up and W down in different low-rank forms.
Figure 3:
2.2 Edit Training and Edit Composing in FFN
Based on the results in Fig. 2 we categorize the factors influencing the composability of edits into Composable and context-preserving. Whether the edits are composable refers to whether there significant interference between the trained edits, which impacts knowledge loss and incorrect preceding, Contex(- pentexx-preserving. on the other hand, refers to whether the edits affect the model's ability follow the context generate other answers, thereby influencing knowledge sinking. In this section, we discuss whether edit training stage of existing methods effectively have these properties and whether their edit composing preserves these properties shown in Table I
Most existing model editing methods insert new knowledge by modifying the parameters or outputs of the FFN layer during the forward process of different tokens. The FFN operates key-value neuron memories, where the values (rows) in the down projection layer, store pre-trained foundation knowledge obtained from vast amounts of data (4 (5)). During the forward process, it retrieves values from the down projection matrix W down E Raxm by matching the keys in the up projection
matrix W E R with the input hin Rlxm.
hout = (h)( = a down + bdown: where a Act hm W up
Dup)
Here, Act is the activation function (e.g., Relu (28. Gelu 29 and SwiGLU (30) ). a is the vector of activation values and serves the weights to retrieve different pre-trained foundation knowledge in W down- Djown& E Ixxm and Dup E R 1lxn are bias vectors.
We summarize edit training and edit composing in FFN for existing methods in Fig. B Among them, GRACE only edits the forward process of the last input token while achieving context- preserving. The other methods modify the for- ward process of all tokens but lack context- preserving. MEND, ROME, AlphaEditt and WISE modify the entire W down- or W up and W Waown to insert new knowledge in edit training, thus different A Wap and AW down are added together to utilize multiple pieces of knowledge in edit composing. AlphaEdii only edits W down and projects AW down onto the null space of the preserved knowledge including other edits be- fore applying it to W down avoid interference between edits. However, in the CME settings we focus on, where single question depends on multiple edits, this orthogonality becomes ineffective and conflicts between edits still persist. WISE learns more composable edits by masking part of Wup and W down in edit training. but the mask on the whole matrix with no carefully designed compos- ing algorithm struggles to keep a large number of edits composable in edit composing. T-patcher concatenates multiple keys A E gmxr and corresponding values B E gg to Wap and Wdown respectively. For GRACE, the output hout are replaced with the sum of edit vectors Ahout E R 1xm for multiple pieces of knowledge. MELO trains a LoRA 3I (A, B) for each edit, where A E Baxr and B E Rr The combination of different knowledge is achieved by concatenating A and B of different LoRAs, respectively. Thanks to the low storage overhead of their low-rank form, GRACE and MELO employ vector database to store all edits and filter out useless ones during inference,
Table 1: Summary of model editing methods under composable and context- presseriing in edit training and if edit composing preserves these properties.
Edit Training
Method
Edit Composing
Composable
Context-preservigg
MEND ROME AlphaFdit WISE T-patcher GRACE MELO A
X
X
X
x
X
X
K
X

---

which achieves more composable edit composing with large number of edits than others but the results contain more incorrect preceding and are far from ideal.
3 Rethinking the Composability in Model Editing
This section examines existing model editing methods explore three key questions: How to train composable edits? How to train context-preserving edits? How to compose edits to further enhance composability? Our answers are Finding Finding 2 and Finding 3, respectively.
Experimental settings. We conducted analytical experiments using the Llama3-8B model 32 on the PEAK-CE dataset [3]] with 50 2-edit composition samples, During inference, tokens for each question, which is enough to output all edited answers. We analyze the performance of existing model editing methods with the metric SR-S Pi. Eler Puull I(Va E V/, rank(a) < (Di)). which is the rate of question & in test dataset Diest that all edited answers Vi output before irrelevant content, where rank(-) represents the rank of an answer in all answers in the outpul and [ represents the number of edited answers. Please refer to Appendix for example.
generate 30
Finding 1. The combination of pre-trained foundation knowledge is enough for ME while boosting the composability. As introduced in Sec. W down stores pre-trained foundation knowledge obtained from vast amounts of data. Our hypothesis that the ability of LLMs to combine different advanced knowledge stems from these well- foundation knowledge elements. These elements boosts the composability with a more robust and generalizable knowledge representation from large-scale pre-training. We validate and in low-rank form when editing Wap or Waown at different layers. As shown in Fig. a) and Fig. ()), editing W up is sufficient to guarantee the performance compared with editing W down as well as editing both Wap up and Wdown- This means the pre-trained foundation knowledge in W down sufficient to express new knowledge. The generally higher SR-S of editing W up in different
hypothesis by the average performance of methods editing full matrix
layers shown in Fig. ([ confirms our hypothesis. For more comprehensive discussion about the sufficiency of editing Wap up for higher SR-S and less knowledge loss, please refer Appendix I
30-
Down projection
Doan projection
Dean seojection
projection
Up smoecion
Up proection
Up Down onjection
Up Down projectin
Up & Down project or
27
28
29
31
30
27
28
29
30
31
Layer (6)
(a)
(b)
Figure 4: (a) Performance of non-compositional ME measured by SR-7 when editing W down: W up
both of them. (b) Performance of non-cone concooppoiiio ME measured by SR-1 when editing W dowa- W up or both of them with pseudo samples to simulate totally unseen knowledge (Appendix lb. (c)
Performance of CME measured by SR-S when editing W
Wdowns W up or both of them.
Finding 2. Less change of the hid- den state of the last answer token [la],
Commett-prreeeeein rate (86
GRAcE
Durs
60:
more context context-preserving preserving. We demon- strate in Fig. 5- 5[a) the relationship be- tween the context-preserving rate and the change of the hidden state at the
30 $
purs
purs f
196)
ROME
MEND
5R-5
W1SE
patcher MELO
edited FFN layer when inferring with
MELO
pha RUME
GRACE
Troatcher
MEND
[la] (c.g. "What are the symptoms of COVID-19? fever"). The results show evident inverse relationship between these two factors in the base- lines, indicating that smaller change in the hidden state of [la] can achieve better pontext-preserving. The finding also answers why GRACE achieves context-following because it has no parametric interference on the forward process of [la] by editing the last input token only.
60
The change of hidden state
Context preseevveeiing rate (%)
Figure 5:
The relevance (a) between the change of hid- den state at [la] and the context-presene context-preserving rate (b) between context-preserving and SR-S.
5

---

edit training
Aasi
edit composing
Aust
positive negative
Values W Cowa
Values in Sown
Adapiiee conbination
Retrieve vallas DG
ooo0
Aa DDO
ooo0
Adaptive
Adaptive merging
tegularzzatiot
Aa 000g 8
Aag o
Ag
A
Edit
Edit
Edit
Keys Wap
Figure 7: Our proposed E including adaptive combination (Sec. 4.1 adaptive regularization (Sec. and adaptive merging (Sec. 4.2)
Finding 3. Setting the elements with different signs in differ- ent edits to zero (Adaptive Merging) during edit composing enhances the composability. The elements with opposite signs between different edits may contain conflicting knowledge. In Fig. 6 inspired by [44, we validate that setting these opposing elements in different edits to zero better balances the know wedge forgetting the edits and their negative interference on each other. As can be seen, the mean performance of methods edit- ing the full matrix and in low-rank form with Adaptive Merging outperforms directly composing different edits.
Figure 6: Performance of CME measured by SR-S with or without Adaptive Merging.
4 The Proposed Method
Although the three findings mentioned above contributes the composability of model editing, there still lacks of a unified framework to fully integrate their advantages. How to train composable and context-preserving edits? Based Finding 1, we adaptively select the pre-trained foundation knowledge using a mask and in a low-rank form to utilize the vector database, namely Adaptive Combination of pre-trained foundation knowledge. We propose two Adaptive Regularization to regularize the use of pre- pre-trained foundation knowledge inspired by Finding while further alleviating incorrect preceding in low-rank form. How to compose edits to further enhance composability? We utilize a vector database to filter out useless edits following [21 22) while proposing Adaptive Merging for different combinations of pre-trained foundation knowledge inspired by Finding 3. The method is shown in Fig. 7 Please refer to Appendix D for the vector database and Alg. for the complete editing progress.
4.1 How to train composable and context-preserving edits?
Adaptive Combination. Inspired by Finding 1, we perform model editing only Wap in a low-rank form similar to LoRA but incorporate an adaptive mask on B. The adaptive mask aims to only preserve the most related pre-trained foundation knowledge achieve better trade-off between the composability and learning ability. Specifically, an (A, B) pair is trained for each edit following (22 The forward process of the edited FFN is calculated as:
Nom = FdN (hin, A, B, = a W down + Ddownn
where a = Act(hin W... up
+ Tii A (B x M) + bbp)
(4)
Aa

---

Algorithm The Edit Training Stage
1: Input The initial LLM fo. the edit dataset Dedn the initialized vector database Sab.
2: Output The vector database Sub
3: for each edit (Zee Be) E Dedi do
4: Compute mask M
5: Replace hown (F6(()))) (Bqq. 3 with hour FFFNedi (hing Ae, Bey M.) (Eq. 4 selected
layers.
6: Update (Ac, with (Eq. 7
7: Save K. into Sab as the key of vector database and save (Ae: Be) as the corresponding value.
8:
end for
A E Ruxr, B E gr*" are the components of LoRA while him and A are normalized hin and A avoid the weights of using different B have significant differences and lead to knowledge loss. M E Rrxx is the mask to select top & related values in W down. where the correlation is calculated by the dot product between values in W down and the answer tokens' embedding:
Adaptive Regularization, Based on Finding 2, we design a regularization f. to restrict the additional of pre-trained foundation knowledge by the last answer token [la], thereby enabling the context- preserving to generate the next answer: Specifically:
C.
=
|| in A
12,
(5)
where An represents the normalized input hidden state of the FFN layer at [la]. L. achieves context-preserving by encouraging a low Aa in Eq. 4
In order to further alleviate incorrect preceding in low-rank form, we propose another regularization Co to restrict the logits of tokens [a] except labels by:
C. - Σ JEW! M
where [a] represents the positions of an edit sample with labels, Way represents the set of all token IDs except the label of the position [a]. During the edit training stage, we optimize (A, B) with the loss:
L= L. ale + BL., where Ce are cross-entropy loss, a and 3 are hyperparameters.
4.2 How to compose edits?
Adaptive Merging, During the inference time, after filtering out useless edits by the vector database Sa (Appendix D). we need to compose different combinations of pre-trained foundation knowledge and different edits may conflict in the utilization of pre-trained foundation knowledge. Specifically, some edits aim to leverage the j-th value in W down to compose the target answer, so [Aa], tends to be positive. In contrast, other edits may want erase the knowledge in the j-th to compose the target answer, so [ tends to be negative. Inspired by Finding 3, when such conflicts occur, we set [ of all used edits 0 to avoid the overly strong impact each other. For simplicity, represent this merging process of different Aa as Σ and the Aa preserved useful edits by vector database S. Thus the forward process during inference changes to:
foundaatnn
Non
FFN (Nin S) aW down + bawwns
where a = Am W + Σ AnES
(8)
Aa + bup).
Experiment
5.1 Evaluation Benchmarks
Dataset. We utilize the PEAK-CF and PEAK-T datasets 33 which contain large number of
questions with multiple answers. We keep the questions for which Llama3-8B [32] and Mistral-7B
7

---

Algorithm 2 The Edit Composing Stage
1: Input The initial LLM fo, the vector database Sub, the test dataset Dieat
2: Output The answers of the queries in Dust-
3: for each query X₂ E Diss do
4: Retrieve edits and calculate the Aa set and S, related to A3 with its representation hab
(Eq.
5: Generate the output with replacing hom = FFN(( (Eq. with hout
FFN (Mtnn S.)
(Eq.
6: end for
models have more than four unknown answers, with 1,949 instances in PEAK-CF and 922 instances in PEAK-T. We directly use the rephrase instances and locality instances from PEAK-CF and PEAK-T to test the generalization and locality. Please refer to Appendix G for the specific construction process.
35
Evaluation metrics. Inspired by R3 we evaluate baselines and our methods with SR-S
Paal 1(Pml I(va E ran < (Da)), which represents the success rate with all edited an- swers the output without incorrect preceding. In Appendix H we provide examples about SR-S while discussing its effectiveness for evaluation and the results of its variants; SR-/ = d Prrll Enex, Iaa Di)), which represents the success rate that edited answers
Vr are in the output answer set Dr. e represents the composition number, which means pieces of edited knowledge are needed for question; GSR [ (Padl II(Va E 3',, (a) < [J41), but
the questions are changed into rephrased ones with the same semantic to evaluate the generalization;
LSR |Deel E(Pnal Rouge-L(S)( _) which represents the fl score of Rouge-L [36 between
the edited outputs Dooe for locality questions and origin output Vo Please refer to Appendix for examples of the input.
Baselines. We compared our method with FT, ROME, KE, MEND, AlphaEdit, WISE, T-patcher, GRACE, and MELO. Please refer to Appendix Gor more details.
5.2 Experimental Results
Different datasets and CME tasks. To evaluate the effectiveness of the proposed A E we conduct large-scale experiments with all instances of the PEAK-CF and PEAK-T datasets, covering four CME tasks: independent multi-answer composition (IMAC), independent multi-question composition (IMQC), multi-answer composition (MAC) and multi-question composition (MQC). independent scenarios, we manually filter out useless edits for each question directly verify the effectiveness of adaptive combination, regularization and merging without the effect of vector database. In MAC and MQC, the vector database is used to evaluate in real-world scenarios with large number of edits asynchronously and distributedly. We set the composition number to 2, which means two pieces of edited knowledge needed for question. The results shown in Table 2 indicate that A consistently performs better on four CME tasks across two datasets. For IMAC on PEAK-CE, E surpasses all baselines by at least 22.45% in SR-S, least 13.63% in SR-/ and at least 18.36% in GSR. Even on challenging MAC, A consistently achieves the best editing performance and generalization while maintaining similar level locality. The lead of A on PEAK-T is even more obvious, with at least 67.16%. 16% and 406.73% in SR-S for IMQC and MQC respectively.
Different number of edits. To evaluate the performance of A throughout the lifelong usage process, we compare the performance among A A GRACE and MELO, which have relatively strong MAC and MQC capability. The results of MAC on PEAK-CF, shown in Fig. (a), indicate that A maintains the most outstanding performance until 3898 edits. Please refer to Appendix. J.1 for more results.
Different composition number. So far, we have conducted experiments with a composition number of 2. To demonstrate that A still has superiority when it needs to simultaneously utilize more edited knowledge, we compared the performance of A3E, GRACE, and MELO with composition numbers ranging from I to 4, each containing 50 (50x 1), 100 (50x2), 150 (50x3) 3) and 200 (50x4)

---

Table 2: Performance measured by metrics in Sec. 5.1 in IMAC (3898 edits) and MAC (3898
edits) on PEAK-CF. 2) IMAC (1984 edits) and MAC (I984 edits) on PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) on PEAK-CF, 4) IMQC (992 edits) and MQC (992 edits) PEAK-T.
PEAK-CF
PEAK-T
Method
IMAC
MAC
IMAC
MAC
St
SR-11 11
GSR
SR-S1
SR-11
GSR1
LSR1
SR-St
SR-11
GSR1
SR-S1
SR-11
GSR1
LSR1
FT
0.31
12.01
0.36
0.00
0.00
0.00
5.25
0.10
1.92
0.10
0.00
0.00
0.00
3.79
ROME
0.92
16.30
0.82
0.00
0.62
0.00
12.25
0.30
18.24
0.60
0.00
0.00
0.00
6.22
KE
0.92
10.31
0.72
0.00
0.00
0.00
10.51
0.10
5.04
0.10
0.00
0,00
0.00
4,56
MEND
2.36
23.09
1.90
0.00
0.56
0.00
12.35
1.01
12.40
0.81
0.00
0.00
0.00
5.41
AlphaEdit
5.44
31.01
4.57
6.98
33.24
6.16
15.17
2.02
21.78
2.12
0.50
16.28
0.41
8.45
WISE
39.99
65.20
38.60
0.10
5.65
0.10
23.08
54.64
21.37
8.40
T-patcher
26.64
52.54
12.83
0.00
0.59
0.00
18.15
0.67
GRACE
15.20
57.44
14.53
12.99
50.16
9.75
6.05
33.07
37.54
MELO
19.46
44.33
18.47
18.22
38.51
16.68
24.13
9.48
28:94
9.07
6.96
22.94
6.56
35.98
A'E
48.97
75.49
45.69
43.38
70.72
39.48
25.76
41.63
51.62
36.90
38.00
49.60
35.89
40.97
PEAK-CF
PEAK-T
Method
IMQC
MQC
IMQC
MQC
SR-$1
SR-11 1
GSR
SR-51
SR-11
GSR1
1.SR1
SR-51
SR-11
GSR1
SR-51
SR-11
GSR1
1.SR1
FT
5.64
21.65
4.82
0.00
0.00
0.00
2.09
0.00
3.23
0.40
0.00
0.00
0.00
2.12
ROME
23.49
47.08
19.38
0.00
0.41
0.00
4.52
19.15
21.92
14.52
0.00
0.00
0.00
3.63
KE
1.03
15.40
0.77
0.00
0.00
0.00
4.23
3.13
0.00
2.98
MEND
5.23
22.38
5.05
0.00
0.00
0.00
3.77
8.77
0.00
4.02
AlphaFdit
7.69
28.57
5.64
5.33
24.92
WISE
46.56
70.11
44.51
0.0
3.23
0.0
7,28
27.62
32.77
24.60
0.20
3,33
0.20
4.40
T-ptchher
33.23
60.41
31.30
0.00
0.00
0.00
0.45
26.81
31.68
25.20
0.00
0.00
0.00
0.32
GRACE
6.56
39.02
5.23
3.59
34.61
2.87
38.71
6.05
32.06
5.54
4.84
30.55
3.63
38.59
MELO
20.31
44.16
18:47
9.13
36.16
8.21
43.22
6.25
23.49
5.85
2.82
19.86
2.02
36.25
A'E
52.51
73.49
52.41
26.46
53.44
20.31
45.73
46.17
68.85
38.31
28.63
53.73
21.77
41.37
edits, respectively. As shown in Fig. ((b). although the SR-S gradually decreases with increasing composition number, A always maintains significantly better performance. Please refer to Appendix. [or more results.
Non-compositional ME. A stronger composability should come at the expense worse non-compositional ME perfor- mance. Therefore, we compare the average non-compositional ME performance of A and baselines. The results, shown in Fig. c).. demonstrate that our method is par with baselines in SR-1, GSR and LSR. We provide non-compositional results of other datasets in Appendix J.6
120
120
120
120
100
MILL
120
100
120
10
102
550
3998
Number wdi.
Compectionn number
(4)
(b)
5023 (4)
055 176
(6)
(d)
Different backbones and prompts. To investigate the impact of different backbones (Llama3-88 8B and Mistral-7BB and few-shot prompts (please refer to Appendix K on the proposed method, we conduct experiments in MAC setting with 3898 edits.
Figure 8: (a) Performance change of A and baselines in MAC until 3898 edits PEAK-CF, (b) The effect of composition number from to 4 in MAC on PEAK-CF, (c) The comparison between A and baselines in non-compositional ME on PEAK- CF. (d) The effect of different backbones and different prompts in MAC on PEAK-CF.

---

The results in Fig. 8 d) show that: (1) A has leading SR-S when using different backbones and different prompts. (2) A E and baselines generally have better editing performance on Llama3-8B. (3) When using different few-shot prompts to combine multiple questions, there is certain performance difference in A This may be due to the fact that, although we manually ensure that few-shot prompts do not contain answers and content obviously related to answers, there may still be some implicit associations between them and the answers. However, this does not affect the significant improvement of A the composability of model editing. Please refer to Appendix. J.3 and Appendix. [.4 for more comprchensive results. We also provide more discussions on failure cases in Appendix J.7 and generalization in Appendix J.8
5.3 Ablation Study
As shown in Fig. 9 a). we conduct experiments in MAC with 3898 edits to evaluate seven ablation versions removing the proposed Co. Le, adaptive combination, adaptive merging, norm and editing only W up. respectively. All the components contribute to the composability of A A Please refer to Appendix. J.5 for more results. We further demonstrate three advantages of the components.
P0
wio
35
Randam mark model merging Randim mack
30
model
448 Unimakeed suM
10)
(b)
Figure 9: (a) Performance of A'E and seven ablation versions in MAC on PEAK-CF. (b) The robustness of edited knowledge when randomly masking B in Wup editing, W down editing and Wap& & W down editing in non-compositional ME on PEAK-CF. (c) Performance improved by adaptive merging with random masks from different seeds or adaptive combination in MAC on PEAK-CF.
Advantage 1. Adaptive combination achieves performance beyond the trade-off between composabil- ity and learning ability. As can be seen from Fig. ()) although adding random mask can achieve good composability, the larger standard deviation indicates that is significantly affected by the random seed. Meanwhile, as the unmask size increases, the learning ability gradually improves with the increasing overlap between edits. We overcome this trade-off by using adaptive combination to select more useful pre-trained foundation knowledge.
Ad Advantage 2. Le makes beller contexi-preservingg See Fig. a]) by incorporating L., our method significantly reduces the change in the hidden state at the last answer token compared to our methods without Le. thus improving the peatext-preservigg capability and achieving a level comparable to GRACE. Mcanwhile, as shown in Fig. b), our stronger context patexxtpressrving capability successfully translates into a higher SR-S.
Advantage 3. Adaptive merging and adaptive combination mutually reinforce each other. In Fig. Mc,, compare the impact of test-time random masks B for editing Waown down and W up in the non- compositional ME scenario, where the mask size is set to the mean size of the parts that require adaptive merging in the CME scenario. It shows that editing W., up is less likely to suffer from forgetting caused by the presence of adaptive merging.
Conclusion
In this paper, we first define and benchmark the CME task, which aims to evaluate the ability of model editing methods to simultaneously utilize multiple edited knowledge. By revisiting existing model editing methods, we point out their lack of composability manifests as knowledge loss, incorrect preceding and knowledge sinking. To mitigate them, we propose A to train more composable and context-preservee edits and compose edits with enhanced composability. Extensive experiments under various scenarios, backbones, and prompts demonstrate the effectiveness of the proposed A "E.
10

---

Acknowledgements
We thank all the anonymous reviewers for their constructive suggestions on improving this paper. This paper is partially supported by Hong Kong Innovation and Technology Fund (ITF) grant MHP/034/22 and National Natural Science Foundation of China (NSFC) grant 62441236.
References
[1] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1rXiv:1909.01066,222 2019.
[2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint 200.::0002.09100, 2020.
[5] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:200.08361,, 2020.
[6] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[7] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint arXiv:2 2021.
(8] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372. 2022.
[9] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2/ 13099, 2021.
[10] Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma, and Jie Yu. Pmet: Precise model editing in transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18564-18572, 2024.
[11] Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang. Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.001286 2024.
[12] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng. Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305. 13172, 2023.
[13] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283-298, 2024.
[14] Jun Yu Ma, Jia-Chen Gu, Zhen- Hua Ling, Quan Liu, and Cong Liu. Untying the reversal curse via bidirectional language model editing. arXiv preprint arXiv:2310. 10320, 2023.
[15] Jia- Ha-Chhn Gu, Hao- Xiang Xu, Jun-Yo Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arXiv preprint arXiv:2401.04700, 2024.
11

---

[16] Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07553, 2024.
[17] Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wen- peng Yin, and Lifu Huang. Navigating the dual facets: A comprehensive evaluation of sequential memory editing in large language models. arXiv preprint #rXi::2402.11122, 2024.
[18] Jun-Yu Yu Ma, Hong Wang. Hao-Xiang Xu,, Zhen-Hua Ling, and Jia-Chen Gu. Perturbation- restrained sequential model editing. arXiv preprint arXiv:2405. 16821, 2024.
[19] Junfeng Fang, Houcheng Jiang, Kun Wang. Yunshan Ma, Xiang Wang. Xiangnan He, and Tat-seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355, 2024.
[20] Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405. 1114405.14766 14768, 2024.
[21] Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
[22] Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron- indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19449-19457, 2024.
[23] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785, 2023.
[24] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012. 14913, 2020.
[25] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint (04i::2004008666, 2021.
[26] Vaswani. Attention is all you need. Advances Neural Information Processing Systems, 2017.
[27] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks, Advances in neural information processing systems, 28, 2015.
[28] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807-814 14, 2010.
[29] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 160ivv1606.000 2016.
[30] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
[31] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00685, 2021.
[32] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[33] Jun-Yu Ma, Zhen-Hus Ling, Ningyu Zhang, and Jia-Chen Gu. Neighboring perturbations of knowledge editing on large language models. arXiv preprint arXiv:2401.17625, 2024.
[34] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[35] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
[36] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[37] Chenmien Tan, Ge Zhang, and Jie Fu, Massive editing for large language models via meta learning, arXiv preprint arXiv:2311.0466) 2023.
[38] Kevin Meng, Arnb Sen Sharma, Alex Andonian. Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. arXiv preprint arXlv:2210.07229, 2022.
[39] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022.
[40] Zexuan Zhong, Zhengxuan Wu, Christopher 1) Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint vrXiv:2305.14799 2023.
[41] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks, arXiv preprint arXiv:2110.06296, 2021.
[42] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
[43] Xisen Jin, Xiang Ren, Daniel Preotiuc- Piettr, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:22/2.09849, 2022.
[44] Michael Matena and Colin Raffel. Merging models with weighte averaging.
Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
[45] Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character, 222(594-6004)339933666 1922.
[46] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13)33521-333 2017.
et al.
[47] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
[48] Shachar Don- Yeh-ye, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022.
[49] Protection Regulation. https://eur-lex. europa. eudti/neg20166679////ccccesss 10 September 2023), 2016.
Eu general data protection regulation. available at, available at:
[50] Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv: /706.04 15, 2017.
[51] Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai-Wei Chang. Deepedit: Knowledge editing as decoding with constraints. arXiv preprint arXiv:2401. 10471, 2024.
[52] Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.127400 2023.
13

---

[53] Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, and Wei Han. Retrieval meets reasoning: Dynamic in-context editing for long-text understanding. arXiv preprini arXiv:2406. / ,
[54] Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, et al. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492, 2024.
[55] Yucheng Shi, Qiaoyu Tan, Xuansheng Wu, Shaochen Zhong, Kaixiong Zhou, and Ninghao Liu. Retrieval-enhanced knowledge editing in language models for multi-hop question answering. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 2056-2066, 2024.
[56] Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, and Weiran Xu. Assessing and post-processing black box large language models for knowledge editing. In Proceedings of the ACM on Web Conference 2025, pages 1716-1732, 2025.
[57] Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models. arXiv preprint wrXiv:2401.06885 2024.
[58] Zihao Zhao, Yuchen Yang, Yijiang Li, and Yinzhi Cao. Ripplecot: Amplifying ripple effect of knowledge editing in language models via thain-of-thought in-context learning. arXiv preprint arXiv:2410.03122. 2024.
[59] Changyue Wang, Weihang Su, Qingyao Ai, Yujia Zhou, and Yiqun Liu. Decoupling reasoning and knowledge injection for in-context knowledge editing, arXiv preprint 2506600566, 2025.
[60] Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. Atm: Adversarial tuning multi- agent system makes robust retrieval-augmenter generator. arXiv preprint arXiv:2405. / 2024.
[61] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 2019.
[62] Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Distributed pruning towards tiny neural networks in federated learning. In 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS), pages 190-201. IEEE, 2023.
[63] Hong Huang, Weiming Zhuang, Chen Chen, and Lingjuan Lyu. Fedmef: Towards memory- efficient federated dynamic pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 27548-27555 2024.
[64] Hong Huang, Hai Yang, Yuan Chen, Jiaxun Ye, and Dapeng Wu. Fedrts: Federated robust pruning via combinatorial thompson sampling. arXiv preprint arXiv:250.119122 2025.
[65] Hong Huang and Dapeng Wu. Quaff: Quantized parameter-efficient fine-tuning under outlier spatial stability hypothesis. arXiv preprint arXiv:2505. /4742, 2025.
[66] Junda Zhu, Jun Ai, Yujun Li, Yichun Yin, Yasheng Wang, Lifeng Shang, and Qun Liu. RidgeloRA: Matrix ridge enhanced low-rank adaptation of large language models. In The Thitt Annual Conference on Neural Information Processing Systems, 2025.
[67] Yichen Wu, Hongming Piao, Laig-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. arXiv preprint arXiv:2501. 138, 2025.
[68] Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei. Federated continual learning via prompt-based based dual knowledge transfer. In Forty-first International Conference Machine Learning, 2024.
14

---

Appendices Contents
Notation
16
Related Work
16
Compositional vs. Collect Edit
17
D The Design of Vector Database
18
Handling Continuous Editing Situations
18
Computational and Memory Overhead
18
Experiment Details
19
H Discussions of SR-S
20
The Sufficiency of Editing Wep
For higher SR-S
I.2 For less knowledge loss
21
J
Additional Experimental Results
J.1
The effect of the composition number and the number of edits
21
The effect of hyperparameters
22
The effect of different backhones
22
J.4
The effect of different prompts
23
J.5
Ablation study
25
................................
0.6
Non-compositional ME evaluation on classic datasets for ME
................................
25
0.7
Failure cases analysis
25
J.8
Generalization
26
Description Dataset and Few-shot Prompts
28
I Limitations
28
M Broader Impact
29
15

---

A
Notation
Table 3: The definition of the notations.
Notation
Definition
m
Input & output dimension of feod-forward network (FFN)
n
Hidden dimension FFN
d
Vocabulary size
W
Parameters of the first layer of FEN; Wap E GR
W down
Parameters the second layer of FFN; Waoen Ruxm
Dup
Bias vector of the first layer of FFN; Dup Bgix
blown
Bias vector of the second layer of FFN; biown E R Xm
haa
Input hidden state of FFN: hip E R 1m
how
Output hidden state of FFN; hox E 1
Activation vector after the first layer of FFN; R
a
(A, B)
Two learnable low-rank matrices A E (R""((""X)) and E "=*"(""*XX constrain the parameter updates to a low-rank subspace following [1.2]
M
The mask to select pre- pre-trained foundation knowledge in W Wawni M E R
hugn
Output logits of I.IM
W
The set all token IDs
S
The preserved edits for inference by the vector database
a
Generated answer of LLM
y
The set of generated answers of LLM
y
The set of edited answers
B Related Work
Model Editing. Model editing has become a popular technique to correct LLMs without the costly re-training. Existing model editing methods mainly divided into constrained fine-tuning meta- learning, santting-andedittin and memory-basee methods. Meta-lcarning methods such as MEND [9] and KE ( train hypernetwork generate model updates, with MALMEN 37 addressing the cancellation effects. Locating-and Locating-and-eddingg and-editing methods like ROME g identify factual associations the feed-forward layer and update them. Building this, MEMIT [8 extends it to batch edit setting, while AlphaEdit [19 projects perturbation onto the null space of the preserved knowledge to achieve better lifelong model editing. To avoid conflicts among numerous edits and the impact original knowledge, memory-based methods such as SERAC (39], T-patcher 223 GRACE (21,, MELO 22 and WISE [20 construct extra working memory in different knowledge preservation forms, from which useless knowledge is filtered out during inference.
Model Editing Evaluation. From single edit to batch edit and then to lifelong edit, the evaluation of model editing methods is becoming increasingly close to real-world applications and more challenging
in terms of the ability to handle interference between different edits and with the original knowledge. In addition to the classic assessments of reliability 9 10, generalization [1] and locality 8 [2].
Cohen et al. [13] proposed evaluating the ripple effect of model editing, which means series of
knowledge that needs to be changed along with the new knowledge. Ma et al. [1] evaluated the
knowledge inserted into the model bidirectionally, while these works [5] 16. 17 assessed the impact of model editing methods on general downstream tasks. Unlike previous works, we are the first to point out that model editing methods should possess composability, namely enabling the model to use multiple edited knowledge simultancously. On the one hand, our work extends locality to scenarios where multiple edits are needed for the same input (multi-question composition & multi-answer composition), which needs to avoid interference between useful edits. On the other hand, our work expands generalization two ways. From the input perspective, we explore how to make an edit generalize to scenarios where only part of the input requires the edit (multi-questio) composition). From the output perspective, we explore how to make an edit generalize to scenarios when there is some content that corresponds to multiple other edits in the output (nulti-question composition & multi-answer composition). This work takes model editing step further towards more complete answers and more complex questions. It is noteworthy that seemingly related
16

---

work 40 considers the success of a single edit in a multi-hop question with reasoning, instead of compositional question/answers that require multiple edits in our focus.
Model Merging. Model merging aims to merge the weights of different models while retaining their respective knowledge. However, due to permutation symmetry [41] and disjoint optimization trajectories [42], simply averaging two models can lead to a catastrophic drop in their performance. RegMean [43 proposed a closed-form solution by solving local linear regression problem for each linear layer, but it requires additional data transmission and inference overhead. Fisher merging A4] weighs the parameters in each model with the Fisher Information Matrix 45 memory overhead and conflicts during the merging of multiple models. Recently, since different fine- tuned models initialized from the same pre-trainet model effectively share a part of the optimization trajectory, model merging methods based on pre- pre-trained models, such as TIES-M TIES-Merging [34,, task arithmetic in tangent space [47), and ColD Fusion (48), have achieved leading results. Our method draws inspiration from these works but differs in that our proposed compositional model editing aims to enable the model to utilize model merging to use multiple edited knowledge simultaneously during the inference.
46 but suffers from
C Compositional vs. Collect & Edit
compositional model editing addresses a different scenario where: during the edit training for "fever" and the edit training for "loss of taste", they are unaware of each other. This means cannot collect all of the knowledge to be edited and then directly compose them at text level. We will explain why this scenario is meaningful below.
As stated in Sec. 2.1 (cf. Lines 97-99) and illustrated in Fig. 1, we believe that it is necessary to support model editing in a federated manner. Specifically, when an open-source model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.
In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR 44), different deployers are only able to share trained edits. In such cases, when a deployer receives trained edits from other deployers, it does not know the questions of the edits. is impossible to get complete answers during edit training because:
1. The deployer has to iterate through all previous edit questions to determine which previous edits need to be composed with the newly received edit to form a complete answer.
2. However, storing all previous edit questions is impossible in lifelong model editing.
When exchanging raw data is allowed, local deployers still have to perform immediate model editing mitigate potential adverse efffecs without waiting to receive errors discovered by other deployers. In this scenario, assuming that each of n deployers updates an answer to the same question and exchanges these n different answers with each other.
Continuously injecting complete knowledge would require times edit training for each deployer at most.
Exchanging independently trained edits would only require time edit training for each deployer.
Training edits independently while ensuring their composability supports model editing in federated manner much more efficiently. This is especially important in the field of model editing, which emphasizes immediate correction of errors.
Meanwhile, for multi-question composition, compositional model editing is clearly meaningful because.
Edits for any two different questions may need to be composed during testing.
Different enrors to be edited may be discovered at different time.
Therefore, is impossible to exhaustively cover all possible composition cases for complete knowl- edge during the edit training process.
17

---

D The Design of Vector Database
To enhance the edit composing stage, we adapt the vector database in 21 22 a simple but effective way. We retrieve useful edited knowledge with a threshold y to ensure locality and a top-p selection to balance the completeness of knowledge and interference from useless edits:
S = (AA((A,))))))) ( A, B) E Sab and Kas and h K Etop-p(ha,K )),
where S represents the retrieved edits from the vector database Sab with K E g1xm xm as the key and (A, B) the value. K is saved from selected hidden state of the edit question. hab E R lxm represents the selected hidden state the test question, which are hyperparameters. We use hap to retrieve edits by matching K. Because of the mask M. we only need to save the unmasked B and the unmasked position. Please refer to Alg. and Alg. for the construction and utilization of Sab- Please also refer Alg. for the complete test procedure.
E Handling Continuous Editing Situations
When adding edit for "loss of taste," if recognize that the existing edit for "fever" is outdated, our method can easily remove the outdated edit with the following locate-and-delete algorithm:
1. Iterate and mask each edit (A, B) in the retrieved part & of the vector database during the generation process locate the edit that leads to the outdated answer. 2. Delete the located edit.
We consider a three-answer composition scenario with x 50 successful edits to test the effectiveness of the deetleeandddelete algorithm above, where each question receives correct edits and one outdated edit. The accuracy of deleting the outdated edits is 100 %. We encourage the construction of outdated edits feedback and avoidance mechanisms when deploying A
Computational and Memory Overhead
In Tab. and Tab. we provide compared to baselines with Llama3-8B and Llama3-70B from four aspects: storage overhead, trainable parameters, extra FLOPs for inference (infer.), extra FLOPs for training (train.). We only provide the extra FLOPs for training of T-patcher, GRACE, MELO and A because other methods that update the whole matrix train a hypernewoork have significantly larger training overhead. Specifically,
detailed analysis of the computational and memory overhead
Storage overhead refers to the additional parameters that need to be stored compared to the pre-trained model for 1000 edits.
Trainable parameters denote the parameters that are updated during training for each edit. Infer. denotes extra FLOPs compared to the pre-trained model for the inference tokens with v input tokens.
Train. denotes extra FLOPs compared to the vanilla forward-backward progress with cross-entropy loss for w training steps of edit with z-token answer.
For Llama3-8B.
also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:
Adaptive merging of A 1.64u x 10-5 GFLOPs
Adaptive combination of A AE: 0.03z GFLOPs
Adaptive regularization of A::: 1.28wz 10-4 1.64 10-5 GFLOPs
In A&E, is set to 50 and < for instance. The extra FLOPs for training and inference are very small compared to the generation of tokens with v input tokens with an 8B model, which about 16u + 16v GFLOPs.
18

---

Table 4: Computational and Memory Overhead in Llama3-8B.
Method
Storage Overhead
Trainable Parameters
infer. (GFLOPs)
trinn (GFLOPs)
FT
0
0,54GB
ROME
L64GB
0,54GB
KE
0.72GB
0.72GB
MEND
1.14GB
1,14GB
AlphaEdit
1.6GBB
0,54GB
WISH
0.11GB
0.11GB
0.27p 0.27a
T-patcher
0.07GB
32.77KB
0.02a
0.01ws
GRACE
0.07GB
32.77KB
0.02
0
MELO
0.13GB
98,30KB
0.02
0
A
0.07GB
39,94KB
0.02 4.92u 10-6 0.03: + 1.28 x 10 + 1.64 x 10
Table 5: Computational Overhead in Llama3-70B.
Method
Trainable Parameters
infer. (GFLOPs)
train. (GFLOPS)
FT
1.88BB
ROME
1.88GB
KE
2.52GB
MEND
2,09GB
0
AlphaFdit
1.88GB
WISE
0,38GB
0.94e + 0.94u
T-patcher
65.54KB
0.03w
0.02mz
GRACE
65.54KB
0.03
0
MELO
180,22KB
0.03
0
A
72.71KB
0.03 + 1. 1.19g x 10-4
0.002 + 1.28azz x 10-
+ 2.87 x 10-5
For Llama3-70B,
We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:
Adaptive merging of A 1.19u x 10-4 GFLOPs
Adaptive combination of A A 0.09z GFLOPss
Adaptive regularization of A 1.28wz x 10 + 2.87w x 10-5 GFLOPs
In A is set to 50 and < for a instance. The extra FLOPs for training and inference are very small compared to the generation of u tokens with v input tokens with an 70B model, which about 140u + 140v GFLOPs.
G Experiment Details
For the baselines, we follow the reproduction code and hyperparameters described in [I]. For A we set the unmasked size k to 896 for PEAK-CF and to 448 for PEAK-T. We set the loss weight and (3 to 1 to balance their utilities. We store the output of the 5-th down projection layer the last subject token as K and hab- We set the edited FFN layer to 31, the learning rate to 0.01, and train each edit for 50 epochs. To better evaluate the priority of different answers within the model, for all baselines and E we assign a penalty of 10 to the answers that have already been generated. How to achieve effective compositional model editing under both instruction- inference settings remains an area requiring urgent exploration. For GRACE, MELO and A&E, we use the vector database in Appendix D with the same p = and = 0.5. The experiments are conducted on server with 8 NVIDIA RTX 5880 Ada GPUs.
models and conventional
For the benchmark construction with PEAK-CF as an example: (1) The original PEAK-CF is model editing dataset containing questions with multiple answers. (2) We queried Llama3 8B and Mistral-7B, selecting questions from PEAK-CE CF where both models missed four or more answers, along with their corresponding missing answers in PEAK-CF. (3) For multi-answer composition, the retained questions and their randomly selected missing answers form a test instance, where e is the composition number. (See Tab. 17 for an example) (4) For multi-question composition, we randomly selected retained questions without repetition and one randomly chosen missing answer per question to form a test instance. (See Tab. 19 for an example)
19

---

H Discussions of SR-S
Fig. shows a correct example and false example of SR-S. SR-S aims accurately measure the interference between different edits to evaluate the composability without considering the case of outputting unrelated answers after all correct answers. This is mitigate the confounding effect of conventional model editing, where "outputting unrelated answers after correct answers" is also a problem, and warrant fairness in evaluating composability. We report the metric SR-AS when "outputting unrelated answers after all correct answers" is considered as error in Table 6 A maintains clearly superior performance.
What the symptoms of COVII-119 loss taste.
fever, loss of taste, Stomachache.
fever, Stemachachee, loss of taste.
(a) SR-S
What are the symptoms of COVID-19? fever, loss of tasto.
lover, loss of uste..
Tover, loss of taste, Stomachache.
fever, Stomachache, loss of taste.
(b) SR-AS
Figure 10:
The illustration (a) SR-S and (b) SR-AS. The text in blue, red, green represents the first correct answer, the wrong answer and the second correct answer, respectively.
Table Performance measured by SR-S and SR-AS in IMAC (3898 edits) and MAC (3898 edits) on PEAK-CF, 2) IMAC (1984 edits) and MAC (1984 edits) PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) on PEAK-CF: 4) IMQC (992 edits) and MQC (992 edits) on PEAK-T.
PEAK-CF
PEAK-T
Method
IMAC
MAC
IMAC
MAC
SR-St
SR-ASt
SR-St
SR-AS1
SR-S?
SR-AS?
SR-S7
SR-ASt
0.31
0.26
0.00
0.00
0.10
0.00
0.00
0.00
ROME
0,92
0.62
0,00
0.00
0.30
0.05
0.00
0,90
KE
0.92
0.41
0.00
0.00
0.10
0.00
0.00
0.00
MEND
2.36
1.95
0.00
0.00
1.01
0.05
0.00
0.00
AlphaEdit
5.44
4.67
6.98
5.23
2.02
1.64
0.50
0.30
WISE
39.99
30.12
0.10
0.00
23.08
17.97
0.20
0.10
T-pateher
26.64
18.98
0.00
0.00
20.77
16.74
0.00
12.99
18.22
5.54
A
48.97
40.69
43.38
38.92
41.63
33.68
38.00
30.90
PEAK-CF
PEAK-T
Method
IMQC
MQC
IMQC
MQC
SR-St
SR-AS1
SR-St
SR-A51
SR-S1
SR-AS?
SR-ASt
5.64
3.70
0.00
0.00
0.00
0.00
0.00
0.00
23.49
19.15
15.12
0.00
MEND
5.23
3.59
0.00
0.00
14.11
10.28
0.00
0.00
AlphaEdit
7.69
6.16
5.33
4.31
16.33
12.70
5.65
4.54
WISE
46.56
37.68
0.00
0.00
27.62
21.67
0.20
0.00
T T-putcher
33.23
26.39
0.00
0.00
26.81
20.26
0.00
0.00
GRACE
6.56
4.93
3.59
2.77
6.05
4.54
4,84
3.93
MELOO
20.31
15.61
9.13
7.39
6.25
5.04
2.82
2.28
A
52.51
43,02
26.46
22.07
46.17
37.40
28.63
24.19
20

---

The Sufficiency of Editing up
L.1 For higher SR-S
From an intuitive perspective, new information does not appear out thin air, it will be related to past information to some extent.
From an empirical perspective, we have shown in Fig. aa that using existing columns form new knowledge (edit Wap) can achieve results comparable to or even slightly better than editing Wdown alone, as well as editing both Wup and W down simultaneously. In Fig- 4b), we conduct the experiment above with fake relation in the input (e.g. share border with -> share boodee with) and fake answer (e.g. Syria -> Sykie) to simulate unseen knowledge. It can be seen that the performance does not change a lot. These experiments indicate that editing Wup is enough to guarantee the performance.
From mathematical perspective, Woown is often of full rank, which to some extent guarantees its expressive power in its output space.
1.2 For less knowledge loss
From intuitive perspective, the distinction between pretrained knowledge and edited knowledge from existing model editing methods lies in the fact that pretrained knowledge is the combination of pretrained foundation knowledge in W down. whereas existing model editing methods directly manipulate the output of W Wowwn Mcanwhile, we know that pretrained models can leverage pretrained knowledge to answer questions requiring multiple answers, but when using multiple edited knowledge, they exhibit significant knowledge loss. Therefore, we intuitively hypothesize that knowledge loss may partly stem from the fact that edited knowledge is not the combination of pretrained foundation knowledge.
From empirical perspective, we observe that editing W up (i.e., combining pretrained foundation knowledge into edited knowledge) results in less min knowledge loss across 27-31 layers and across baselines compared editing W down editing both W up and W down in Tab. 7 while leading to higher SR-S and similar non con-cooposiiionall performance (Fig. 4p.
Table 7: Min knowledge loss across 27-31 layers and across baselines when editing W Wap W Wown. W up and W down-
Method
Min knowledge loss (%)
W up
60.6
W down
71.4
W up and W
69.7
Additional Experimental Results
J.1 The effeec of the composition number and the number of edits
In Fig. III provide complete experimental results for the effect of different number of edits and the composition number MAC and MQC on PEAK-CF and PEAK-T. In PEAK-CF dataset, we also conducted additional experiments by sampling 50 questions with 10+ unknown answers each and evaluated the performance with a composition number of 10.
Table 8: Experimental results PEAK-CF dataset with composition number 10.
Method
ROME
AlphaEdit
T-patcher
GRACE
MELO
A'E
SR-1
2.40
4.60
6.60
8.60
12.20
17.20
As shown in Tab. 8 the experimental results demonstrate that when large number of edits need to be applied simufraneously, A still significantly outperforms the baselines on the SR-/ metric, which means A retains more edited knowledge with large composition numbers.
21

---

100
10C
130
330
100
130
Compostiie sumberr MAC on PEAK-CF
Compectior numterr MQC on PEAK-CF
Compestior number (c) MAC on PBAK-T
Compestion ramber MQC on PEAK-T
100
H83
130
100
130
180 Th
Murost of 28 (9 MCQC on PEAK-CF
Nurbict 00
(5) MAC PEAR-CC
(g) MAC on PEAK-T
(1) MQC on PEAK-T
Figure 11: (a)-d) Performance with different composition number measured by SR-S and SR-/ in MAC and MQC on PEAK-CF and PEAK-T. (e)-(h) Performance change with different number of edits measured by SR-S and SR-/ in MAC and MQC on PEAK-CF and PEAK-T.
J.2 The effect of hyperparameters
The hyperparameters of A E are not sensitive and easy adjust. First, as shown in Fig. 12 a)-
d), compared with the performance improvement of A its performance is not sensitive to the selection of a, _3, 7. and p. Second, the adjustment of hyperparameters straightforward for two reasons: 1) As shown in Fig. 12 a) [2]e), e), all hyperparameters exhibit similar behavior across different datasets (i.e., PEAK-CF and PEAK-T), indicating that a one-time hyperparameter search is almost sufficient, and repeated re- re-tuning can be avoided. 2) As shown in Fig. 12(a) ()) 12),), there exist correlations among certain hyperparameters. c.g., setting a 83 often leads the best performance.
This enables joint tuning, thereby reducing the number of peprraarameters that need to be adjusted.
44
50
52
54
54
52
34
65
40
59
8.0425
56
56
50
50
6.229
52
50
56
0
4
20
42
4
26
52
0.25
46
52
50
48
54
56
56
50
60
60
48
42
52
50
52
54
42
45
0.5
(b)
56
59-5 (5)
FEAK PEANT
PEAK
59-5
52
PEAK
PEART
50
0.3
3.5
0.6
3.8
112
224
448
896
1792
81
(d)
Figure 12: The effect of (a) 3 on the performance measured by SR-S in MAC PEAK-CF, (b) on the performance measured by SR-S in MAC on PEAK-T. The effect of (c) y, (d) p and (e) on the performance measured by SR-S in MAC on PEAK-CF and PEAK-T.
JJ3 The effect of different backbones
Table shows the performance with backbone Mistral-7B in four scenarios two datasets.
22

---

Table 9: Performance with Mistral-7B measured by metrics in Sec. 5.1 in IMAC (3898 edits) and MAC (3898 edits) on PEAK-CF. 2) IMAC (1984 edits) and MAC (1984 edits) on PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) PEAK-CF, 4) IMQC (992 edits) and MQC (992 edits)
PEAK-T.
PEAK-T
METHOD
IMAC
MAC
IMAC
MAC
SR-S1
SR-11
GSR1
SR-S1
SR-11
GSR1
LSR1
SR-S1
SR-11
GSR1
SR-S*
SR-11
GSR1
LSR1
FT
0.26
10.19
0.31
0.00
0.00
0.00
4.93
0.00
2.03
0.32
0.00
0.00
0.00
2.10
ROME
0.77
13.83
0.71
0.00
0.52
0.00
11.51
14.44
17.61
11.83
0.00
0.00
0.00
3.59
KE
0.76
8.74
0.73
0.00
0.00
0.00
9.88
3.19
5.08
2.55
0.00
0.00
0.00
3.85
MEND
1.97
19.58
1.66
0.00
0.00
11.61
10.64
11.06
7.14
0.00
3.98
ALPHARDIT
26.30
15.06
WISE
33.91
58.70
14.98
19.25
8.02
T-PATCHER
22.22
44.56
11.19
0.00
0.95
20.21
31.68
20.53
0.00
0.00
0.00
0.32
GRACE
12.68
48.72
12.68
10.70
42.24
8.22
23.55
4.56
20.22
4.51
5.37
23.61
5.00
40.14
MELO
15.24
35.25
14.52
13.80
32.99
13.23
23.11
7.86
23.99
7.60
5.75
18.95
5.24
34.76
A"E
40.84
64.03
39.86
35.73
59.55
33.30
24.21
34.80
43.43
31.21
31.75
41.53
30.04
40.88
PEAK-CF
PEAK-T
METμOD
IMQC
MQC
IMQC
MQC
SR-St
SR-17 17
GSRT
SR-ST
SR-If
GSRT
LSRF
SR-ST
SR-17
GSRT
SR-S7
SR-If
GSRT
LSKT
FT
0.28
9.79
0.31
0.00
0.00
0.00
5.24
0.00
2.71
0.00
0.00
0.00
0.00
2.06
ROME
0.82
13.28
0.79
0.00
0.40
0.00
12.55
15.89
18.35
12.15
0,00
0.00
0.00
3.53
KE
0.82
8.40
0.69
0.00
0.00
0.00
12.49
3.51
6.75
2.62
0.00
0,00
0.00
2.90
MEND
2.12
18.82
1.82
0.00
0.36
0.00
12.73
11.71
14.69
7.34
0.00
0.00
0.00
3.92
ALPHARDIT
4.88
25.26
4.38
3.60
21.03
2.82
15.70
13.55
15.44
9.28
4.74
21.01
4.44
4.00
WISE
38.96
58.73
37.27
0.00
2.71
0.00
7.32
23.39
27.42
20.56
0.00
2.82
0.00
4,20
T-PATCHER
23.90
42.80
12.28
0.00
0.37
1.90
22.25
26.53
21.09
0.00
46.80
13.91
31.74
5.02
26.85
25.56
37.57
36.96
7.60
A
43.94
61.50
43.74
22.33
44.76
18.06
45.33
38.31
57.66
32.06
19.15
40.27
J.4
The effect of different prompts
Table 10
shows the performance of A A and state-of-the-ar baselines WISE, GRACE, MELO with different prompts in four scenarios two datasets.
23

---

Table 10: Performance with prompt 2 measured by metries in Sec. 5.1 in 1) IMAC (3898 edits) and MAC (3898 edits) on PEAK-CF. 2) IMAC (1984 edits) and MAC (1984 edits) on PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) on PEAK-CF. IMQC (992 edits) and MQC (992 edits) on
PEAK-T.
PEAK-CF
PEAK-T
Method
IMAC
MAC
IMAC
MAC
SR-5
SR-11
GSR1
SR-51
SR-17
GSR1
1.RR
SR-S7
SR-11
GSR1
SR-51
SR-17
GSR1
1.RR1
0.26
9.97
0.30
0.00
0.00
0.00
5.36
0.09
1.59
0.09
0.00
0.00
0.00
3.15
0.76
0.00
0.00
0.00
0.09
0.00
MEND
1.92
18.80
1.55
0.00
0.45
0.00
12.06
0.82
10.00
0.66
0.00
0.00
0.00
5.37
AlphaIdit
4.60
26.21
3,86
5,90
28.10
5.21
15.82
1.71
18.41
1,80
0.42
13.76
0.34
8.14
WISE
32.81
53.54
31.67
0.13
4.59
0.12
15.46
18.88
44.78
17.72
0.27
2.51
0.27
8.04
T-parcher
22.43
44,30
10:77
0.08
0.44
0.04
1.81
17.46
36.71
15.49
0.00
0.00
0.00
0.70
GRACE
12.66
48.04
12.10
10.88
41.96
8.21
22.73
5.39
34.98
5.22
4.62
27.81
4.32
37.57
MELO
17.75
40.52
16.84
16. 68
35,20
15.31
25.05
8.58
26.35
8:45
6.26
21.11
6.11
35.07
A-E
47.88
73.86
44.68
42.48
69.20
38.70
25.88
40.66
50.40
36.25
37.13
48.68
35.26
40.03
PEAK-CF
PEAK-T
Method
IMQC
MQC
IMQC
MQC
SR-S-
SR-11
GSR
SR-S1
SR-17
GSR1
LSR1
SR-S1
SR-11
GSR1
SR-S1
SR-17
GSR1
LSR1
FT
4.68
17.98
4.00
0.00
0.00
0.00
2.73
0.01
2.68
0.34
0.00
0.00
0.00
2.76
ROME
19.51
39.10
16.09
0.00
0.34
0.00
4.75
15.92
18.20
12.07
0.00
0.00
0.00
3.02
KF
0.85
12.79
0.64
0.00
0.00
0.00
3.25
3.49
6.63
2.58
0.00
0.00
0.00
2.45
MEND
4.21
18.05
4.07
0.00
0.00
0.00
3.04
11.5
14.28
7.15
0.00
0.00
0.00)
4.28
AlpbaEdit
6.50
24.15
4.76
4.51
21.06
3.56
6.85
13.81
15.58
9.38
4.77
21.23
4.26
4.48
18.14
57.51
36.46
0.09
22.51
26.66
2.96
Typatcher
27.95
50.93
26.32
0.09
21.58
GRACE
5.35
32.55
4.25
3.10
88
2.48
4.87
4,94
3.83
25.8
3.24
38.58
MILO
18.45
40.30
16.78
8.45
32.99
7.60
39.48
5.65
21.22
5.53
2.36
18.40
2.05
36.45
A'E
51.28
71.83
51.19
26.01
52.23
19.97
44.69
45.02
67.14
37.81
27.82
52.84
21.52
40.77
Table 11: Performance with prompt 3 measured by metries in Sec. 5. in 1) IMAC (3898 edits) and
MAC (3898 edits) on PEAK-CF 2) IMAC (1984 edits) and MAC (1984 edits) on PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) on PEAK-CF. IMQC (992 edits) and MQC (992 edits) on
PEAK-T.
PEAK-CF
PEAK-T
Method
IMAC
MAC
IMAC
MAC
SR-11
SR-51
GSR1
LSR1
SR-S1
SR-11
GSR1
SR-51
SR-11
GSR1
LSR
0.28
9.33
0.24
0.00
0.00
0.00
5.08
0.08
1.49
0.08
0.00
0.00
0.00
3.95
ROME
0.72
12.78
0.64
0.00
0.48
0.00
12.61
0.47
14.31
0.24
0.00
0.00
0.00
0.72
8.09
0.56
0.00
0.00
0.00
0.10
3.95
0.08
0.00
0.00
0.00
4.58
MEND
1.83
17.92
1.47
0.00
0.43
0.00
12.58
0.80
9.81
0.65
0.00
0.00
0.00
5.28
AlphaEdit
4.34
24.78
3.65
5.58
26.56
4.92
15.12
1.62
17.40
1.62
0.40
13.01
0.33
8.75
WISE
31.58
51.53
30.48
0.13
4.42
0.12
15.99
18.17
43,10
17.06
0.26
2.42
0.26
8.78
T-paccher
20.97
41.43
10.07
0.05
0.41
0.04
1.75
16.34
34.36
14.51
0.00
0.00
0.00
0.66
GRACE
12.05
45.75
11.52
10.41
39.95
7.82
27.64
5.13
33.30
4.97
4.40
26.49
4.12
37.07
MELO
13.36
30.52
12.68
12.57
26.51
11.55
24.03
6.44
19.82
6.41
4.69
15.94
4.63
34.95
A'E
46.38
71.54
43.28
41.15
67.03
37.49
25.39
39.25
48.64
35.03
35.80
46.99
34.02
40.86
PEAK-CF
PEAK-T
Method
IMQC
MOC
IMQC
MQC
SR-S-
SR-11
GSR*
SR-St
SR-1"
GSRT
LSR1
SR-S7
SR-11
GSR?
SR-St
SR-17
GSRT
LSR1
FT
4.38
16.81
3.74
0.00
0.00
0.00
2.62
0.01
2.50
0.32
0.00
0.00
0.00
2.65
ROME
18.42
36.93
15:20
0.00
0.32
0.00
4.54
15.03
17.19
11.40
0.00
0.00
0.00
3.85
KE
0.80
12.08
0.60
0.00
0.00
0.00
3.07
3.33
6.32
2.46
0.00
0.00
0.00
2.34
MEND
4.13
17.71
3.99
0.00
0.00
0.00
3.98
10.96
13.61
6.81
0.00
0.00
0.00
4.12
AlphaEdit
6.14
22.83
4.26
19.91
6.53
13.06
14.73
8.87
4.51
20.07
4.02
36.71
55.35
35.09
0.08
21.66
25.65
19.77
2.86
0.20
T-patcher
26.16
47.66
0.00
0.00
20.98
20.20
0.00
0.00
0.00
GRACE
5.09
30.99
4.04
2.96
27.49
2.37
38.78
4.63
25.28
4.72
3.64
24.59
3.10
38.04
MELO
13.87
30.34
12.61
6.39
24.84
5.74
39.73
4.26
16.13
4.11
1.89
13.76
1.45
37.08
A'E
49.49
69.33
49.40
25.10
50.40
19.28
45.13
43,60
65.03
36.64
26.94
51.19
20.85
41.51
24

---

J.5
Ablation study
In Table 12
we provide the comprehensive results of ablation study in four scenarios on two datasets.
Table 12: Ablation study measured by metrics in Sec. 5.1 in 1) IMAC (3898 edits) and MAC (3898 edits) on PEAK-CF. 2) IMAC (1984 edits) and MAC (T984 edits) on PEAK-T. 3) IMQC (1948 edits) and MQC (1948 edits) PEAK-CH 4) IMQC (992 edits) and MQC (992 edits) PEAK-T.
PEAK-CF
PEAK-T
Method
IMAC
MAC
IMAC
MAC
SR-ST
SR-17
GSR
SR-S7
SR-11
GSR
LSRT
SR-ST
SR-17
GSR
SR-ST
SR-17
GSRT
LSR
A L.
44.07
67.94
41.12
39.04
63.65
35.53
23.19
37.47
46.46
33.21
34,20
44.64
32.30
40.97
A w/o
39.18
60.39
36.55
34,70
56.58
31.59
23.61
33.31
41.30
29.52
30.40
39.68
28.71
40.77
A comhin.
37.54
57.88
35.03
33.26
54.22
30.27
23.15
32.13
39.83
28.47
29.32
38.28
27.70
40.61
A merg.
38.60
59.51
36.02
34,20
55.75
31.12
24.31
32.47
40.26
28.78
24.64
38.69
28.00
40.95
A²E wro norm
38.85
59.89
36.25
56.10
31.32
31.85
39.49
28. 23
29.07
37.95
41.34
A'E
37.30
53.86
27.67
E up&dowwa
36.93
A
48.97
75.49
45.69
43.38
70.72
39.48
25.76
41.63
51.62
36,90
38,00
49.60
35.89
40.97
PEAK-CF
PEAK-T
Method
IMQC
MQC
IMQC
MQC
SR-St
SR-17
GSR
SR-11
SR-S1
SR-11
48.09
A'E
A combin.
40.45
41.23
15.67
44.29
35.39
52.78
29.38
39.17
"E wro mery.
40.96
57.32
40.88
20.64
41.68
15.84
44.67
36.40
54.28
30.20
22.57
42.36
17.17
39.61
A'E w/o
40.17
56.22
40.10
20.24
40.88
15.53
44.99
36.63
54.62
30.39
22.71
42.63
17.28
40.82
A w/ down
39.38
55.12
39.31
19.84
40.08
15,23
44.30
35.
52.44
29.18
21.80
40.93
16.59
38.51
A w/ updkdown
41.04
57.45
40.97
20.68
41.77
15.87
44.75
37.32
55.65
30.97
23.14
43.44
17.60
38.33
A
52,51
73.49
52.41
26.46
53.44
20.31
45.73
46.17
68.85
38.31
28.63
53.73
21.77
41.37
J.6
Non-compositional ME evaluation classic datasets for ME
As shown in Tab. 13
the non-compositional performance A is on par with baselines.
Table 13: Non-compositional results classic ME dataset ZsRE 50 and Counterfact [8] ing on Llama3-8B.
follow-
Method
ZsRE
Counterfact
SR-1f
GSR
LSR
SR-11
GSR
LSR
FT
47.95
52.32
73.08
51.53
48. 14
63.95
ROME
99.53
97.52
95.57
100.00
92.77
77.40
MEND
92.24
90.70
96.23
79.10
58.76
91.19
AlphaEdit
98.00
96.54
95.57
99.55
95.03
79.21
WiSE
93.39
92.53
100.00
98.59
93.50
99.32
T-pachher
95.28
94.24
92.62
95.03
93.79
89.38
GRACE
96.69
94.00
100.00
99.77
94.12
100.00
MELO
95.28
94.08
100.00
98.53
92.88
100.00
ABE
97.00
96.77
100.00
99.77
93.79
100.00
J.7
Failure cases analysis
We conducted failure case analysis for overlapping, contradictory, and capacity-exceeding edits follows:
For overlapping and contradictory cases, we conducted statistical analysis on the PEAK-CF
dataset under the IMAC scenario with 50 cases of two-edit composition, measuring:
The overlapping selection of pre-trained foundation knowledge, calculated by the overlapping
length between non-zero Aa (Sec. [.] regions of two edits in adaptive merging (Sec. 4.2)
The contradictory utilization of pre-trained foundation knowledge, calculated by the length of regions with different signs between Aa (Sec. 4.1 of two edits in adaptive merging (Sec. 42
25

---

The table below presents how the SR-S metric, which is inversely proportional to the proportion of failure cases, varies across different overlapping and contradictory ranges. As the overlapping length and contradictory length increase, the SR-S of A gradually decreases, but it remains significantly higher than the baselines.
Table 14: SR-S across different overlapping ranges.
Overlapping
[0, 500)
500, 1000)
[1000. 1500)
[1500, +
SR-S
75.00
66.67
45.45
50.00
Table SR-S across different contradictory ranges.
Contradictory (0,2
[200, 400;
[400, 600)
[600, +
SR-S
66.67
68.75
57.14
46.67
Table 16: SR-S comparison with baselines.
Baselines
MEND
ROME
AlphaHddir
WISE:
T-patcher
GRACE
MHIO
SR-S
4.00
4.00
32.00
20.00
20.00
For capacity-exceedingg cases, found that the failure cases of baselines and A in both compo- sitional model editing and non-compositional model editing scenarios often involve numerical data in answers. For example,
Edit Rheinmetall, which has designed 12 em leFH 08
Test I: Rheinmetall, which has designed 12 em leFH 8 (failure)
Edit 2: Rheinmetall, which has designed 35.5 em Haubitze M1
Test 2: Rheinmetall, which has designed 35.5 em Haubitze M1 (success)
Composition test: Rheinmetall, which has designed 12 cm leFH, 12 em Haubitze M8 (failure)
This may stem from the model's inherent inability to accurately distinguish numerical data. There- fore, addressing such inherent limitations the model a valuable direction for future work in model editing.
We believe these findings can guide future research to further improve edit composability.
J.8 Generalization
The potential generalization limitations are as follows:
Noisy scenarios. In Fig. R(d), we tested the impact of different few-shot prompts A A performance. These varied few-shot prompts not only controlled the model's output but also simulated different noise present in the context. The results show that while A performance is affected, it still maintains a leading performance compared to the baselines.
Noisy open-domain scenarios. We humbly acknowledge that in open-domain scenarios, noisy contexts may still impact the retrieval accuracy of the vector database. Since A primarily focuses on edit training and edit composing, we will explore
optimizing the hidden state positions for retrieval based on the real data distribution in application scenarios
employing more powerful embedding models
training a question-rewrittng module to better align with the vector database
for optimal performance the future work.
Adversarial scenarios. We conducted additional experiments using x 50 successful edits from the PEAK-CF dataset under the IMAC setting by providing the tested question with incorrect answers the few-shot prompts. The results demonstrate that A& consistently (100%) outputs the edited answers across all test cases, proving its robustness against contextual interference.
26

---

Table 17: An example of multi-an -nswer composition containing two pieces of edited knowledge, where the blue text indicates the question and the rephrased question.
Question
Bertrand Russell is the author of
Answer
Power: A New Social Analysis, A History of Western Philosoohy
Edit 1 x
Bertrand Russell is the author of
Edit 1
Power: A New Social Analysis
Edit 2 at
Bertrand Russell is the author of
Edit 2
A History of Westemn Philosophy
Test Ag
Tim Dorsey, who has written the Cadillac Beach, Nuclear Jellyfish, Triggerfish Twist, Hammer- head Ranch Motel, The Big Bamboo, Orange Crush (novel), Hurricane Punch, The Stingray Shufile, Atomic Lobster, Torpodo Juice (novel), Florida Roadkill \n Jerusalem. which is the partner town of NYC, New York, Praha, New York City, United States, Rio de Janeiro, NY, New York, NY, Prague, New York City, Tehran, Buenos Aires, Moscow, Manhattan. \n Pushkin is the author The Fountain of Bakhchisaray, Eugene Onegin, The Tale of the Fisherman and the Fish, Poltava (poem), The Tale of the Golden Cockerel, Dubrovsky (novel), The Belkin Tales, Onegin, The Stone Guest (play). The Bronze Horseman (poem), The Queen of Spades (story), The Tale of the Priest and of His Workman Balda, The Gypsies, The Blizzard, Tatiana Larina, The Tale of the Dead Princess and the Seven Knights. \n WWE is the owner of WWE Classics on Demand, FCW Florida Heegh Championship, WWE Studios, WWE Films, WWE Network, NXT, FCW, WWE Classics On Demand, FCW Southern Heavyweight Championship, WCW. World Championship Wrestting, WCW, Inc., Florida Championship Wrestling, NXT Wrestling, Universal Wrestling Corporation, WWE NXT.An Bertrand Russell is the author of
Test Di
Power: A New Social Analysis, A History of Western Philosophy
Rephrased test
Tim Dorsey, who has written the Cadillac Beach, Nuclear Jellyfish, Triggerfish Twist, Hammer- head Ranch Motel. The Big Bamboo, Orange Crush (novel), Hurricane Punch, The Stingray Shuffle, Atomic Lobster, Torpedo Juice (novel), Florida Roadkill. An Jerusalem, which the partner town of NYC, New York. Praha, New York City, United States, Rio Janciro. NY. New York, NY, Prague, New York City, Tehran, Buenos Aires, Moscow, Manhattan. An Pushkin the author of The Fountain of Bakhchisaray, Eugene Onegin, The Tale of the Fisherman and the Fish, Poltava (poem). The Tale of the Golden Cockerel. Dubrovsky (novel). The Belkin Tales, Onegin, The Stone Guest (play). The Bronze Horseman (poem), The Queen of Spades (story), The Tale of the Pricst and of His Workman Balda, The Gypsics, The Blizzard, Tatiana Larina. The Tale of the Dead Princess and the Seven Knights \n WWE is the owner of WWE Classics on Demand, FCW Florida Heavyweight Championship. WWF Studios, WWF Films, WWF Network. NXT, FCW. WWE Classics On Demand, FCW Southern Heavyweight Championship. WCW, World Championship Wrestling, WCW, Inc., Florida Championship Wrestling, NXT Wrestling, Universal Wrestling Corporation. WWE NXT An Bertrand Russell is the writer of
Locality test
Philosophy, which is played by
Larger models and black-hox models. A is applicable larger LLMss Due to limitations in computational resources, we provide experimental results for GRACE, MELO, and A&E under the MAC scenario using Llama3-70B and the PEAK-CE dataset. As shown in Table 18 A maintains the leading performance.
Table 18: MAC and MQC performance on PEAK-CF with Llama3-70B.
MAC
MQC
Method
SR-St
SR-11 11
GSRt
LSR
SR-St
SR-11 11
GSR1
LSRt
GRACE
16.32
42.48
11.24
24.02
3.59
37.28
3.62
38.16
MELO
28.12
48.64
26.58
23.18
12.13
39.27
11.33
41.41
A
50.69
69.47
47.10
25.10
33.12
55.65
32.98
44,26
However, A cannot be applied to black-box models. In the field of knowledge editing for large language models, one research direction focuses on editing white-box box models, i.e., model
editing 21 20 19] The access to model weights is a prerequisite for applying these editing techniques. Another research direction addresses knowledge editing for extremely large-scale models where model editing may be prohibitively expensive or black-box models by performing
edits at the text level, including [51 52 13
40
53
54
55
56
57
58
59 60).
A A follows the first
27

---

direction to edit white-box models and focuses compositional model editing because text-level knowledge editing not applicable for the following reason:
As stated in Sec. 2.1
and illustrated in Fig. I we aims to support model editing in a federated
manner [61
62
63 64]. Specifically, when model is deployed globally, numerous distinet
errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other.
In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [49] different deployers are only able to share trained edits rather than the texts.
In a centralized scenario, A can also be implemented by the owner of black-box models for imme- diate error correction with much more composable edits. We recommend users to choose methods from either direction based on their needs, and we will explore how to perform compositional knowledge editing at the text level for black-box models our future work.
K Description of Dataset and Few-shot Prompts
In Table 17 and T9 we provide examples of the simultaneous use of two pieces of knowledge in the MAC and MQC scenarios, respectively. In Table 20 provide another prompts test the effect of different prompts.
Table 19:
An example of multi-question composition containing two edited pieces knowledge, where the blue text indicates the question and the rephrased question.
Edit 1
Carnarthenshire shares border with
Edit 1
Ceredigion
Edit 2 at
Turkey shares border with
Edit 2
Syria
Test 2g
The answers of the questions "Daimler has made the". "Saxony adjacent to" are Mercedes- Benz, Hamburg respectively. An The answers of the questions "Karlheinz Stockhausen is the composer of", "Daimler the parent organization of" are Originale, Car2go respectively. An The answers of the questions "Kttheinne Roberts is the writer of". "Florida International University has the employer" are The Colossus Crisis, Carlos Alvarez respectively, An The answers of the questions "contraception has subclass of", "Kering has subsidiary" are Intrauterine device, Volcom respectively. An The answers the questions "Carmarthenshire shares border with". "Turkey shares border with"
Test
Ceredigion. Syria
Rephrased test
The answers of the questions "Daimler made the". "Saxony adjacent to" Mereedes- Benz, Hambbur respectively.'y An The answers of the questions "Karlheinz Stockhausen the composer of", "Daimler the parent organization of Originale, Car2go respectively, An The answers of the questions "Katherine Roberts is the writer of". "Florida International University has the employer" are The Colossus Crisis, Carlos Alvarez respectively. An The answers of the questions "contraccption has subclass of", "Kering has subsidiary" are Intrauterine device, Volcom respectively.N. An The answers of the questions "Carmarthenshire is adjacent to", "Turkey is adjacent to" are
Locality test
Vasif Talibov a citizen of
L Limitations
Although the proposed method significantly enhances the composability of model editing, it still falls short of achieving fully accurate composition. Future research will continue to explore the factors that influence composability and further improve the effectiveness of compositional model editing. The performance of instruct models under penalty-free decoding condition also needs to be explored. Additionally, as this paper primarily focuses on how to train and compose edits with composability, does not delve deeply into the organization of vector databases. Future work will aim to continuously
28

---

Table 20: Different few-shot prompts.
Prompt X:
Dorsey, who has written the Cadillac Beach, Nuclear Jellyfish, Triggerfish Twist, Hammer- head Ranch Motel, The Big Bamboo, Orange Crush (novel), Hurricane Punch, The Stingray Shufflc, Atomic Lobster. Torpedo Juice (novel), Florida Roadkill. \n Jerusalem. which is the partner lown of NYC, New York. Praha, New York City, United States, Rio de Janeiro. NY, New York, NY, Prague, New York City. Tehran, Buenos Aires, Moscow, Manhattan. An Pushkin is the author of The Fountain of Bakhchisaray. Eugene Onegin, The Tale of the Fisherman and the Fish, Poltava (poem), The Tale of the Golden Cockerel, Dubrovsky (novel), The Belkin Tales, Onegin, The Stone Guest (play). The Bronze Horseman (pocm), The Queen of Spades (story), The Tale of the Priest and of His Workman Balda, The Gypsies. The Blizzard, Tatiana Larina, The Tale of the Dead Princess and Seven Knights. An WWE is the owner of WWE Classics on Demand, FCW Florida Heaavyeighh Championship. WWE Studios, WWE Films, WWE Network, NXT, FCW, WWE Classics On Demand, FCW Southern Heavyweight Championship. WCW. World Championship Wrestiigg. WCW, Inc., Florida Championship Wrestling. NXT Wrestling, Universal Wrestling Corporation, WWE NXT.An
Prompt X;
Saxony adjacent to Hamburg, Nordsachsen, Liberec Region, Sachsen, Saxony-Anhall Bavaria, Sachsen-Anhalt Thuringen, North Rhine- Westphalia, Thuringia, Saxony Anhalt, Lower Saxony, Brandenburg, An Florida International University has the employer Carlos Alvarez, Les Standi- fond, Elizabeth Price Foley, Stanley Fish, Leonard Strickman, Barhara Walsh, Jerry Markham. \n Contraccption has subclass of Intrauterine device, withdrawal method, Emergency contracep- tion, Coitus interruptus, Condom, intrauterine device, morning-aie maater pill. An Kering has subsidiary Volcom, Saint Laurent Paris. Gucei Group. Christopher Kane, Puma, Boucheron, Gucci, Saint Laurent, Bottega Veneta, Balenciaga, Sergio Rossi.A
Prompt X;
Westchester County adjacent to Bronx County, The Bronx, south BrOXx, Putnam County, Rockland, Bronx. An Margery Allingham the writer The Casc the Late Pig. Hide My Eyes, The Mind Readers, Cargo of Eagles, The China Governess, Sweet Danger, Coroner's Pidgin. An The position Philippine President is held by Fidel Ramos, Corazon Aquino, Fmilio Aguinaldo, Macapagal, Joseph Ejercito Estrada, Roxas, Quezon, Benigno Aquino III. Diosdado Macupagal, Ferdinand Marcos, Manuel Quezon. An cytokine has a subclass of Monokine, Interferon, lym- phokine, interleukins, Chemokine.VV
optimize the structure of vector databases. The effectiveness of other PEFT methods 65 need to be explore as well.
66
68
M Broader Impact
This paper presents work whose goal is to advance the field of model editing. Model editing is cutting-edge concept that holds immense significance in a variety of critical and rapidly evolving fields, including but not limited to medical or legal fields. In these scenarios, mistakes of models are found after deployment. thus timely and effective correction is needed. Our research has taken a step towards more available model editing by allowing multiple edits be used simultaneously. leading to safer and more ethical applications across broad spectrum of industries.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix L
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [No]
Justification: Release the codes upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Subbmiiiioooiiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Model editing on specific example determinative when additional randomly initialized parameters are introduced. In this case, there is no associated error bar. We provide results in different datasets. models and prompts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources are provided in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pwp....ee.....o/oooooooaa
Answer: [Yes]
Justification: The authors conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix M Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: For the pre-trate treitrained models and the datasets used in this paper, we check them manually to ensure the safety.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors cite the original paper that produced the pre-trained models and datasets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
35

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We conduct the experiments of model editing on LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
36


Reviews:
[{"review_id": "tDeHyLIVZJ", "reviewer": "Reviewer_qR3R", "summary": "The paper proposes a method for compositional model editing where multiple edits need to be integrated simultaneously to answer complex questions effectively. The proposed method is inspired by their findings that (1) the combination of pre-trained foundation knowledge is enough for model editing while boosting composability (which inspires the method they call *adaptive combination*), (2) the less change in the hidden state of the last answer token, the more context preserving is the model (which inspires the method they call *adaptive regularization*), (3) setting elements with different signs in different edits to zero during edit composing improves composability (which inspires the method they call *adaptive merging*). The paper then conducts empirical evaluation of the proposed method over various datasets, showing improved performance in compositional editing.", "questions": "- can you give step-by-step walk through on how the benchmark dataset is constructed?\n- can you improve on the readability of the paper by making the definition of the variables clear (i.e., not assuming that it is self-explanatory) and the paper more self-contained?\n- how do you identify and select these three critical problems faced by model editing methods: knowledge loss, incorrect preceding, and knowledge sinking? Are they based on prior studies or are they first defined in this work? why the choice of these three problems only? are these commonly used terms/problems in compositional studies? \n- how do you make the connection between knowledge loss and \"Finding 1\"? Is knowledge loss caused by insufficient combination of the pre-trained foundation knowledge in W_down? How can one come to this conclusion by only looking at performance Figure 4? \n- can you discuss more on the computational and memory overhead of the proposed method compared to baselines?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the paper introduces a new task that is compositional model editing, which evaluates scenarios where multiple edits must be applied to answer a complex question.\n- the paper conducts extensive experiments across multiple datasets \n\nWeakness:\n- the main issue of the paper is its lack of readability. The description of the proposed method is very complex and dense with heavy reliance on prior works and appendices, which makes the paper very hard to read and the contribution hard to appreciate. For example, many of the variable names (W_down, W_up) are assumed to be common knowledge and not defined properly, referred to instead as the 'down projection' and 'up projection' matrices. Variables such as A and B are used instead of the commonly used K and V to refer to the key and value vectors in transformers. Given that new variables are used instead of the commonly used variable names, they should be defined more clearly. \n- in addition, many of the crucial details are missing. For example, the steps for creating the benchmark dataset from existing datasets are unclear. (e.g., which part of the benchmark creation is done manually?). This not only makes the paper hard to read but also reducing its reproducibility. \n- the explanation of the method is not intuitive. To add to the difficulty, the naming of the different parts of the proposed method adds more to the confusion. For example, while the name 'adaptive combination' and 'adaptive merging' refer to different processes in the pipeline of their proposed method, the word 'combination' and 'merging' are similar. Adding more intuitive examples instead of architectural diagram or performance chart will make the proposed concepts more accessible. \n- it is unclear how they identify and select the three critical problems faced by model editing methods: knowledge loss, incorrect preceding, and knowledge sinking. Is it based on prior studies or are they first defined in this work? why the choice of these three problems only? are these commonly used terms/problems in compositional studies? It will be good to motivate this choice for readers unfamiliar with this field of research. \n- the choice of the problems and the solutions seems heavily empirical. For example, how do the authors connect knowledge loss to \"Finding 1\"? Is knowledge loss caused by insufficient combination of the pre-trained foundation knowledge in W_down? How can one come to this conclusion by only looking at performance Figure 4? All I can see from this figure is that updating W_up improves performance. But whether it is because there's less knowledge loss is not clear. It is unclear how the authors connect problems to the cause to the solution. Is there a missing link not detailed here? Because of these gaps in explanation, it is hard to appreciate the contribution. Having more theoretical analysis or even more intuitive motivation that connects problems to observations to the proposed solution can deepen the understanding and appreciation of the proposed method beyond the engineering of the method via empirical trials. \n- the computational overhead is discussed only briefly. Given the complexity of the method proposed, more detailed exploration of the computational and memory cost involved is needed.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer qR3R for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.\n\n**W1&Q2**. Lack of readability caused by variable names without definition.\n> We sincerely appreciate the reviewer's suggestions for improving the readability of the paper. We promise to include the following notation table in the revised version, with each notation clearly defined.\n> \n>| Notation| Definition|\n>| ----------- | ---------- |\n>| $\\mathbf{m}$| Input & output dimension of feed-forward network (FFN)|\n>| $\\mathbf{n}$| Hidden dimension of FFN|\n>| $\\mathbf{d}$| Vocabulary size|\n>| $\\mathbf{W}_{\\text{up}}$   | Parameters of the first layer of FFN; $\\mathbf{W}_{\\text{up}} \\in \\mathbb{R}^{m\\times n}$|\n>| $\\mathbf{W}_{\\text{down}}$ | Parameters of the second layer of FFN; $\\mathbf{W}_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$|\n>| $\\mathbf{b}_{\\text{up}}$   | Bias vector of the first layer of FFN; $\\mathbf{b}_{\\text{up}} \\in \\mathbb{R}^{1\\times n}$|\n>| $\\mathbf{b}_{\\text{down}}$ | Bias vector of the second layer of FFN; $\\mathbf{b}_{\\text{down}} \\in \\mathbb{R}^{1\\times m}$|\n>| $h_{\\text{in}}$ | Input hidden state of FFN; $h_{\\text{in}} \\in \\mathbb{R}^{1\\times m}$|\n>| $h_{\\text{out}}$| Output hidden state of FFN; $h_{\\text{out}} \\in \\mathbb{R}^{1\\times m}$|\n>| $\\mathbf{a}$| Activation vector after the first layer of FFN; $\\mathbf{a} \\in \\mathbb{R}^{1\\times n}$|\n>| $(\\mathbf{A}, \\mathbf{B})$ | Two learnable low-rank matrices $\\mathbf{A} \\in \\mathbb{R}^{n\\times r} (\\mathbb{R}^{m\\times r})$ and $\\mathbf{B} \\in \\mathbb{R}^{r\\times m} (\\mathbb{R}^{r\\times n})$ to constrain the parameter updates to lie in a low-rank subspace following [1,2] |\n>| $\\mathbf{M}$               | The mask to select pre-trained foundation knowledge in $\\mathbf{W}_{\\text{down}}$; $\\mathbf{M}  \\in \\mathbb{R}^{r\\times n}$|\n>| $h_{\\text{logit}}$         | Output logits of LLM |\n>| $\\mathcal{W}$              | The set of all token IDs|\n>| $\\mathcal{S}$              | The preserved edits for inference by the vector database|\n>| $a$                        | Generated answer of LLM|\n>| $\\hat{\\mathcal{Y}}$        | The set of generated answers of LLM|\n>| $\\mathcal{Y}$              | The set of edited answers|\n>  \n>  [1] Lora: Low-rank adaptation of large language models. (ICLR 2022)\n>\n>  [2] Melo: Enhancing model editing with neuron-indexed dynamic lora (AAAI 2024)\n\n**W2&Q1**. Benchmark dataset construction process\n> We sincerely appreciate the reviewer’s suggestions for improving the reproducibility of our paper. We promise to provide a complete benchmark dataset construction process in the revised version as follows (using PEAK-CF as an example):\n> 1. The original PEAK-CF [1] is a model editing dataset containing questions with multiple answers.\n> 2. We queried Llama3-8B and Mistral-7B, selecting questions from PEAK-CF where both models missed four or more answers, along with their corresponding missing answers in PEAK-CF.\n> 3. For multi-answer composition, the retained questions and their randomly selected $c$ missing answers form a test instance, where $c$ is the composition number. (See Tab. 9 for an example)\n> 4. For multi-question composition, we randomly selected $c$ retained questions without repetition and one randomly chosen missing answer per question to form a test instance. (See Tab. 10 for an example)\n>\n> [1] Neighboring Perturbations of Knowledge Editing on Large Language Models (ICML 2024)\n\n**W3**. Distinguishing naming of the different parts of the proposed method and intuitive explanation of the method\n> We sincerely appreciate the reviewer’s suggestions for improving the explanation of the method.\n>\n> **Distinguishing naming**\n> + We have revised **adaptive combination** to the more detailed **adaptively combine pre-trained foundation knowledge** and modified **adaptive merging** to **adaptively merge different combinations of pre-trained foundation knowledge.**\n> \n> **Intuitive explanation**\n> + We commit to updating Fig. 7 in the revised version to include an intuitive running example consisting of two stages: **edit training** and **edit composing**. Due to image upload limitations, we summarize the content of each stage here:\n>\n>   **Edit training**\n>   - How to adaptively combine pre-trained foundation knowledge into new knowledge.\n>   - How to adaptively regularize and an intuitive example of the effects brought by the regularization.\n>   \n>   **Edit composing**\n>   - How to use the vector database to filter out irrelevant edits during inference.\n>   - How to adaptively merge different combinations of pre-trained foundation knowledge to leverage them simultaneously during inference.\n\n**W4&Q3**. Three critical problems faced by compositional model editing methods\n> + We humbly clarify that these three problems are not derived from prior studies or commonly used in compositional studies. \n> + Instead, they were identified and defined for the first time by summarizing failure cases in existing methods under compositional model editing, making them critical problems specific to this field.\n> + The reason we focus solely on these three problems is that they comprehensively encompass all the failure cases we observed.\n> + We promise to add the above motivations for the three critical problems in the revised version.\n\n**W5&Q4**. The connection between knowledge loss and Finding 1\n> We sincerely appreciate the reviewer’s suggestion to strengthen the connections among problems, observations, and solutions in the paper. We have reorganized the logic in Finding 1 as follows:\n> + Motivation:\n>   - The distinction between pre-trained knowledge and edited knowledge from existing model editing methods lies in the fact that pre-trained knowledge is the combination of pre-trained foundation knowledge in the down projection matrix, whereas existing model editing methods directly manipulate the output of the down projection layer. \n>   - Meanwhile, we know that pre-trained models can leverage pre-trained knowledge to answer questions requiring multiple answers, but when using multiple edited knowledge, they exhibit significant knowledge loss. \n>   - Therefore, we intuitively hypothesize that knowledge loss may partly stem from the fact that edited knowledge is not the combination of pre-trained foundation knowledge.\n> + Observations:\n>   - We observe that editing $\\mathbf{W}\\_{\\text{up}}$ (i.e., combining pre-trained foundation knowledge into edited knowledge) results in less **min knowledge loss** across 27-31 layers and across baselines compared to editing $\\mathbf{W}\\_{\\text{down}}$ or editing both $\\mathbf{W}\\_{\\text{up}}$ & $\\mathbf{W}\\_{\\text{down}}$ in the following table while leading to higher SR-S and similar non-compositional performance (Fig. 4).\n>   \n>      | Method  | Min knowledge loss (%) |\n>      | ------- | :-----------: |\n>      | Up      | 60.6   |\n>      | Down    | 71.4 |\n>      | Up&Down | 69.7 |\n> + Solutions:\n>   - We opt to edit only $\\mathbf{W}_{\\text{up}}$.\n\n**W6&Q5**. More discussions on the computational and memory overhead compared to baselines\n> We greatly appreciate the reviewer for the suggestions. In the table below, we provide a more detailed analysis of the computational and memory overhead compared to baselines with Llama3-8B from four aspects: **storage overhead**, **trainable parameters**, **extra FLOPs for inference (infer.)**, **extra FLOPs for training (train.)**. We only provide the extra FLOPs for training of T-patcher, GRACE, MELO and A$^3$E because other methods that update the whole matrix or train a hypernetwork have significantly larger training overhead. Specifically,\n> + **Storage overhead** refers to the additional parameters that need to be stored compared to the pre-trained model for 1000 edits.\n> + **Trainable parameters** denote the parameters that are updated during training for each edit.\n> + **Infer.** denotes extra FLOPs compared to the pre-trained model for the inference of $u$ tokens with $v$ input tokens.\n> + **Train.** denotes extra FLOPs compared to the vanilla forward-backward progress with cross-entropy loss for $w$ training steps of an edit with $z$-token answer.\n>   \n>| Method    | Storage Overhead | Trainable Parameters | infer. (GFLOPs)            | train. (GFLOPs) |\n>| --------- | ---------------- | -------------------- | -------------------------- | --------------- |\n>| FT        | 0                | 0.54GB       | 0          | -               |\n>| ROME      | 1.64GB           | 0.54GB               | 0                          | -               |\n>| KE        | 0.72G            | 0.72G                | 0                          | -               |\n>| MEND      | 1.14GB           | 1.14GB               | 0                  | -               |\n>| AlphaEdit | 1.64GB           | 0.54GB               | 0                          | -               |\n>| WISE      | 0.11GB           | 0.11GB               | $0.27v+0.27u$              | -               |\n>| T-patcher | 0.07GB           | 32.77KB              | $0.02u$                    | $0.01wz$        |\n>| GRACE     | 0.07GB           | 32.77KB              | $0.02$                     | 0               |\n>| MELO      | 0.13GB           | 98.30KB              | $0.02$                     | 0               |\n>| A$^3$E    | 0.07GB           | 39.94KB              | $0.02+4.92u\\times 10^{-5}$ | $0.03z+1.28wz\\times 10^{-4}+1.64w\\times 10^{-5}$         |\n>\n> We also provide the extra FLOPs of the proposed adaptive combination, adaptive regularization and adaptive merging for inference and training:\n>    - Adaptive merging of A$^3$E: $1.64u\\times 10^{-5}$ GFLOPs\n>    - Adaptive combination of A$^3$E: $0.03z$ GFLOPs\n>    - Adaptive regularization of A$^3$E: $1.28wz\\times 10^{-4}+1.64w\\times 10^{-5}$ GFLOPs\n> \n> The extra FLOPs for training and inference are very small compared to the generation of $u$ tokens with $v$ input tokens with an 8B model, which is about $16u+16v$ GFLOPs."}, {"review_id": "tfyo0nSsa0", "reviewer": "Reviewer_e3yR", "summary": "This paper addresses the limitations of existing model editing methods for large language models (LLMs), which are typically evaluated and optimized for single, isolated edits, ignoring the real-world need for compositional model editing (CME), i.e., the ability to integrate and utilize multiple edits coherently. The authors define and benchmark the CME task, identifying key issues in existing methods: knowledge loss, incorrect precedence, and knowledge sinking. They further propose an editing framework that adaptively combines and regularizes the pre-trained knowledge to support future composability.", "questions": "How is the method applicable to larger LLMs and/or black-box LLMs?", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The CME task is well-motivated and practically relevant, addressing an important gap in the evaluation of model editing: real-world queries often require combining multiple edited facts.\n2. The work is conceptually clear and innovative, introducing adaptive mechanisms both during training and at test-time to enhance composability.\n3. Extensive experiments on thousands of edits (up to ~4k), different datasets, noise settings, model backbones (Llama3-8B, Mistral-7B), and prompts.\n\nWeaknesses:\n\n1. In the experiments, the authors analyzed the non-compositional ME performance of the proposed framework on PEAKCF. However, such non-compositional ME evaluation should be conducted on other classic datasets for ME.\n2. The considered LLMs only include Llama3-8B and Mistral-7B, which could be insufficient for a comprehensive evaluation, as the model size is not generally large. Furthermore, the authors did not discuss whether the method can be applied to black-box LLMs and how to adapt this method to such LLMs.\n3. The paper does not discuss the failure cases of editing, e.g., if edits are contradictory, overlapping, or beyond the model’s capacity. Such an analysis could be valuable for understanding its robustness.", "was_revised": true, "final_justification": "The rebuttal has addresed most of my concerns.", "rebuttal": "We sincerely thank Reviewer e3yR for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.\n\n**W1**. Non-compositional ME evaluation should be conducted on other classic datasets for ME.\n> We greatly appreciate the reviewer for drawing our attention to this point, which indeed benefits from more solid experiments. In response to the reviewer's suggestion, we provide additional non-compositional results of state-of-the-art baselines on classic ME dataset ZsRE [1] and Counterfact [2] following [3] on Llama3-8B, which demonstrate that **our method is on par with baselines**.\n>\n> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**ZsRE**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Counterfact**\n>  | Method    | SR-1  | GSR   | LSR    | SR-1   | GSR   | LSR    |\n>  | --------- | :-----: | :-----: | :------: | :------: | :-----: | :------: |\n>  | FT        | 47.95 | 52.32 | 73.08  | 51.53  | 48.14 | 63.95  |\n>  | ROME      | 99.53 | 97.52 | 95.57  | 100.00 | 92.77 | 77.40  |\n>  | MEND      | 92.24 | 90.70 | 96.23  | 79.10  | 58.76 | 91.19  |\n>  | AlphaEdit | 98.00 | 96.54 | 95.57  | 99.55  | 95.03 | 79.21  |\n>  | WISE      | 93.39 | 92.53 | 100.00 | 98.59  | 93.50 | 99.32  |\n>  | T-patcher | 95.28 | 94.24 | 92.62  | 95.03  | 93.79 | 89.38  |\n>  | GRACE     | 96.69 | 94.00 | 100.00 | 99.77  | 94.12 | 100.00 |\n>  | MELO      | 95.28 | 94.08 | 100.00 | 98.53  | 92.88 | 100.00 |\n>  | A$^3$E    | 97.00 | 96.77 | 100.00 | 99.77  | 93.79 | 100.00 |\n>\n> [1] Zero-shot relation extraction via reading comprehension (ACL 2017)\n>\n> [2] Locating and editing factual associations in GPT (Neurips 2022)\n>\n> [3] A Comprehensive Study of Knowledge Editing for Large Language Models\n\n**W2&Q1**. How is the method applicable to larger LLMs and/or black-box LLMs?\n> **A$^3$E is applicable to larger LLMs.**\n> + We humbly acknowledge the value of testing on **larger LLMs**. Due to limitations in computational resources, we provide experimental results for GRACE, MELO, and A$^3$E under the MAC scenario (Sec. 5.2, cf. Lines 267-274) using Llama3-70B and the PEAK-CF dataset. As shown in the following table, **A$^3$E maintains the leading performance**. We promise to provide complete experimental results in the revised version.\n> \n>   | Method | SR-S  | SR-1  | GSR   | LSR   |\n>   | ------ | :-----: | :-----: | :-----: | :-----: |\n>   | GRACE  | 16.32 | 42.48 | 11.24 | 24.02 |\n>   | MELO   | 28.12 | 48.64 | 26.58 | 23.18 |\n>   | A$^3$E | 50.69 | 69.47 | 47.10 | 25.10 |\n> \n> We sincerely appreciate the reviewer's suggestion regarding the discussion on the applicability of A$^3$E to **black-box models**. We promise to include the following discussion in the revised version:\n> + In the field of knowledge editing for large language models, \n>   - One research direction focuses on editing white-box models, i.e., model editing [1-4]. The access to model weights is a prerequisite for applying these editing techniques.\n>   - Another research direction addresses knowledge editing for extremely large-scale models where model editing may be prohibitively expensive or black-box models by performing edits at the text level, including [5-15]\n>   \n> + **A$^3$E follows the first direction to edit white-box models** and focuses on **compositional model editing** because **text-level knowledge editing is not applicable** for the following reason:\n>   - **As stated in Sec. 2.1 (cf. Lines 97–99) and illustrated in Fig. 1, we aims to support model editing in a federated manner**. Specifically, when a model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other. \n>   - In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [16], **different deployers are only able to share trained edits rather than the texts**.\n> + In a centralized scenario, **A$^3$E can also be implemented by the owner of black-box models** for immediate error correction with much more composable edits.\n> + We recommend users to choose methods from either direction based on their needs, and we will explore how to perform **compositional knowledge editing at the text level** for black-box models in our future work.\n>\n> [1] Alphaedit: Null-space constrained knowledge editing for language models (ICLR 2025 outstanding paper)\n>\n> [2] MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA (AAAI 2024)\n>\n> [3] WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (Neurips 2024)\n>\n> [4] Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (Neurips 2023)\n>\n> [5] DeepEdit: Knowledge Editing as Decoding with Constraints\n>\n> [6] Can we edit factual knowledge by in-context learning?\n>\n> [7] Evaluating the ripple effects of knowledge editing in language models. (TACL 2024)\n>\n> [8] Mquake: Assessing knowledge editing in language models via multi-hop questions. (EMNLP 2023)\n>\n> [9] Retrieval meets reasoning: Dynamic in-context editing for long-text understanding\n>\n> [10] Multi-hop question answering under temporal knowledge editing (COLM 2024)\n>\n> [11] Retrieval-enhanced knowledge editing in language models for multi-hop question answering (CIKM 2024)\n>\n> [12] Assessing and post-processing black box large language models for knowledge editing (WWW 2025)\n>\n> [13] Fine-grained Hallucination Detection and Editing for Language Models (COLM 2024)\n>\n> [14] Ripplecot: Amplifying ripple effect of knowledge editing in language models via chain-of-thought in-context learning (EMNLP 2024)\n>\n> [15] Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing\n>\n> [16] The eu general data protection regulation (gdpr)\n\n\n**W3**. Failure cases of editing\n> We greatly appreciate the reviewer for taking this point into consideration. We conducted **failure case analysis for overlapping, contradictory, and capacity-exceeding edits** as follows:\n> + For **overlapping and contradictory cases**, we conducted statistical analysis on the PEAK-CF dataset under the IMAC scenario with 50 cases of two-edit composition, measuring:\n>   - The overlapping selection of pre-trained foundation knowledge, calculated by the overlapping length between non-zero $\\Delta a$ (Sec. 4.1) regions of two edits in adaptive merging (Sec. 4.2)\n>   - The contradictory utilization of pre-trained foundation knowledge, calculated by the length of regions with different signs between $\\Delta a$ (Sec. 4.1) of two edits in adaptive merging (Sec. 4.2)\n> \n>   The table below presents how the SR-S metric, which is inversely proportional to the proportion of failure cases, varies across different overlapping and contradictory ranges. As the overlapping length and contradictory length increase, the SR-S of A$^3$E gradually decreases, but it remains significantly higher than the baselines.\n>   \n>   | Overlapping | \\[0,500) | \\[500,1000) | \\[1000,1500) | \\[1500,$+\\infty$) |\n>   | ----------- | :---------: | :-----------: | :------------: | :-----------------: |\n>   | SR-S     |     75.00    |     66.67      |       45.45      |     50.00             |\n>\n>   | Contradictory | \\[0,200) | \\[200,400) | \\[400,600) | \\[600,$+\\infty$) |\n>   | ------------- | :--------: | :----------: | :----------: | :----------------: |\n>   | SR-S          | 66.67    | 68.75      | 57.14      | 46.67            |\n>\n>   | Baselines | MEND | ROME | AlphaEdit | WISE  | T-patcher | GRACE | MELO  |\n>   | --------- | :----: | :----: | :---------: | :-----: | :---------: | :-----: | :-----: |\n>   | SR-S      | 4.00 | 4.00 | 8.00      | 32.00 | 20.00     | 14.00 | 20.00 |\n>\n>\n> + For **capacity-exceeding cases**, we found that the failure cases of baselines and A$^3$E in both compositional model editing and non-compositional model editing scenarios often involve numerical data in answers. For example,\n>   - **Edit 1**: Rheinmetall, which has designed **12 cm leFH 08**\n>\n>     **Test 1**: Rheinmetall, which has designed **12 cm leFH 8** (failure)\n>   - **Edit 2**: Rheinmetall, which has designed **35.5 cm Haubitze M1**\n>\n>     **Test 2**: Rheinmetall, which has designed **35.5 cm Haubitze M1** (success)\n>   - **Composition test**: Rheinmetall, which has designed **12 cm leFH**, **12 cm Haubitze M8** (failure)\n>   \n>\n>   This may stem from the model's inherent inability to accurately distinguish numerical data. Therefore, addressing such inherent limitations of the model is a valuable direction for future work in model editing. \n>   \n> We believe these findings can guide future research to further improve edit composability."}, {"review_id": "2mEdzyrnLn", "reviewer": "Reviewer_cjhv", "summary": "This paper proposes Compositional Model Editing (CME), where large language models must integrate multiple knowledge edits to answer complex, real-world questions. Existing model editing methods are shown to struggle in this setting due to issues like knowledge loss and incorrect preceding. To address this, the authors introduce $A^3E$, a two-stage editing framework that improves edit composability through adaptive combination, regularization, and merging techniques. Extensive experiments demonstrate that $A^3E$ significantly outperforms prior methods in compositional settings without reducing standard editing performance.", "questions": "1. Could you clarify the distinction between Incorrect Preceding and Knowledge Sinking?\n2. How is the adaptive mask in the Adaptive Combination module computed in practice?\n3. You propose setting conflicting $\\delta a$ elements to zero. Could you clarify do you apply this merging at the level of each token, each edit, or batch-wise? I get a little lost in this part.", "limitations": "No.\n\nSuggestions for Improvement:\n1. Discuss generalization limits. It would be helpful to mention how it might perform in more noisy, open-domain, or adversarial scenarios.\n2. The paper lacks a thorough analysis of how the method scales to larger numbers of edits per query (e.g., 10 or more).\n3. The performance is much higher than the baselines. I suggest you check if there are any mistakes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper identifies gaps between current model editing and real-world needs by proposing the Compositional Model Editing (CME) task, which is both novel and practically important.\n2. $A^3E$'s architecture thoughtfully aligns with the challenges identified. Each of the three components (adaptive combination, regularization, and merging) corresponds to the failures of prior works.\n3. The method shows large and consistent improvements in compositional settings while preserving performance in non-compositional tasks.\n\nWeaknesses:\n1. While Figure 7 summarizes the core components of $A^3E$, it presents them in a very abstract and schematic way. The figure lacks concrete examples or intuitive annotations, which makes it difficult to understand how each module actually works in practice. Adding a running example would make the method significantly more accessible.\n2. In the introduction, terms like Incorrect Preceding and Knowledge Sinking are introduced as distinct issues but not clearly differentiated. Readers may struggle to understand how they manifest differently in outputs, especially without clearer examples or contrasting definitions.\n3. The approach may not generalize well if the number of required edits for a query becomes very large (e.g., 10+), but this scenario is not deeply explored.", "was_revised": true, "final_justification": "I appreciate the clarifications and additional experiments and hope the authors can incorporate these helpful clarifications and improvements in the final version of the paper. I'll keep my scores.", "rebuttal": "We sincerely thank Reviewer cjhv for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.\n\n**W1**. Adding a running example for the method\n> We sincerely appreciate the reviewer’s suggestions for improving the explanation of the method.\n> + We commit to updating Fig. 7 in the revised version to include an intuitive running example consisting of two stages: **edit training** and **edit composing**. Due to image upload limitations, we summarize the content of each stage here:\n>\n>   **Edit training**\n>   - How to adaptively combine pre-trained foundation knowledge into new knowledge.\n>   - How to adaptively regularize and an intuitive example of the effects brought by the regularization.\n>   \n>   **Edit composing**\n>   - How to use the vector database to filter out irrelevant edits during inference.\n>   - How to adaptively merge different combinations of pre-trained foundation knowledge to leverage them simultaneously during inference.\n\n**W2&Q1**. The distinction between Incorrect Preceding and Knowledge Sinking\n> We greatly appreciate the reviewer's suggestion for further distinguishing between Incorrect Preceding and Knowledge Sinking. Referring to the examples and definitions provided in Fig. 2, we offer the following contrasting definitions:\n> + Knowledge Loss refers to **some parts of the edited answers were perturbed or missing**.\n>   - fever, **loss of taste** -> fever, **loss of fever**\n> + Incorrect Preceding refers to cases that **the model knows all edited answer but outputs the wrong answers first**.\n>   - fever, **stomachache**, loss of taste\n> + Knowledge Sinking refers to cases that **the model potentially knows some edited answers but being unaware that it should generate them**, instead producing only some of the edited answers before generating non-answer content.\n>   - fever **fever is a …**\n\n**W3&S2**. Performance with large composition number\n> We sincerely appreciate the reviewer for raising this point. In PEAK-CF dataset, we conducted additional experiments by sampling 50 questions with 10+ unknown answers each and evaluated the performance with a composition number of 10. \n>\n>| Method | ROME | AlphaEdit | WISE | T-patcher | GRACE | MELO  | A$^3$E |\n>| ------ | :----: | :---------: | :----: | :---------: | :-----: | :-----: | :------: |\n>| SR-1   | 2.40 | 4.60      | 9.40 | 6.60      | 8.60  | 12.20 | **17.20**  |\n>\n> The experimental results demonstrate that when a large number of edits need to be applied simultaneously, \n> + A$^3$E still significantly outperforms the baselines on the SR-1 metric, which means A$^3$E retains more edited knowledge with large composition numbers.\n\n**Q2**. How is the adaptive mask in the Adaptive Combination module computed in practice?\n> In practice, $\\mathbf{M} \\in \\mathbb{R}^{r\\times n}$ is the mask to select top $k$ related values (rows) in $\\mathbf{W}\\_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$ for each answer token, where the correlation is calculated by the dot product between values (rows) in $\\mathbf{W}\\_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$ and the answer tokens' embedding. The specific calculation process is as follows:\n> 1. Calculate the correlation $\\mathbf{V}\\_s \\in \\mathbb{R}^{1\\times n}$ between values (rows) in $\\mathbf{W}\\_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$ and the embedding of each answer token $s$ in the token set of an answer $\\mathcal{A}\\_s$:\n> \\begin{align}\n>  \\mathbf{V}\\_s=[\\mathbf{W}\\_{\\text{unemb}}]\\_s\\mathbf{W}\\_{\\text{down}}^{\\top}\n>  \\end{align} \n>  $\\mathbf{W}\\_{\\text{unemb}}  \\in \\mathbb{R}^{d\\times m}$ is the unembedding matrix. $d$ is the vocabulary size.\n>\n> 2. Select top $k$ related values (rows) in $\\mathbf{W}\\_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$ for each answer token $s$:\n> \\begin{align}\n> \\\\{v|[\\mathbf{V}\\_s]\\_v \\in \\text{top}\\_k(\\mathbf{V}\\_s)\\\\}\n> \\end{align}\n> $v$ is the index of the selected values (rows) in $\\mathbf{W}\\_{\\text{down}} \\in \\mathbb{R}^{n\\times m}$.\n> 3. Get the union of $v$ selected by all answer tokens in $\\mathcal{A}\\_s$:\n> \\begin{align}\n>    \\mathcal{K}=\\bigcup\\nolimits\\_{s\\in \\mathcal{A}\\_s}\\\\{v|[\\mathbf{V}\\_s]\\_v \\in \\text{top}\\_k(\\mathbf{V}\\_s)\\\\}\n>  \\end{align}\n> 4. Calculate the adaptive mask:\n> \\begin{align}\n> [\\mathbf{M}]\\_{:,i}=&\\left\\\\{\n> \\begin{array}{rcl}\n> \\textbf{0}     &      & i \\notin \\mathcal{K} \\\\\\\\\n> \\textbf{1}       &      & i \\in \\mathcal{K}\n> \\end{array} \\right. \n> \\end{align}\n\n**Q3**. How to apply adaptive merging?\n> Our proposed adaptive merging operates **at the token level** to enable token-specific adaptations on the utilization of pre-trained foundation knowledge. Specifically, for $n$ edits that need to be applied simultaneously, each token's $h\\_{\\text{in}}$ computes $n$ $\\Delta a$ in Eq. (4). We then perform adaptive merging on these $n$ $\\Delta a$ values in Eq. (8).\n\n**S1**. Discuss generalization limits.\n> We sincerely appreciate the reviewer's suggestion regarding the potential generalization limitations of A$^3$E in various scenarios. \n> + **Noisy scenarios**. In Fig. 8(d), we tested the impact of different few-shot prompts on A$^3$E's performance. These varied few-shot prompts not only controlled the model's output but also simulated different noise present in the context. The results show that while A$^3$E's performance is affected, it still maintains a leading performance compared to the baselines.\n> + **Noisy open-domain scenarios**. We humbly acknowledge that in open-domain scenarios, noisy contexts may still impact the retrieval accuracy of the vector database. Since A$^3$E primarily focuses on edit training and edit composing, we will explore \n>   - optimizing the hidden state positions for retrievement based on the real data distribution in application scenarios \n>   - employing more powerful embedding models \n>   - training a question-rewriting module to better align with the vector database\n> \n>   for optimal performance in the future work.\n> + **Adversarial scenarios**. We conducted additional experiments using $2\\times50$ successful edits from the PEAK-CF dataset under the IMAC (cf. Lines 269-274) setting by providing the tested question with incorrect answers in the few-shot prompts. The results demonstrate that A$^3$E consistently (100%) outputs the edited answers across all test cases, proving its robustness against contextual adversarial interference.\n\n**S3**. Check if there are any mistakes.\n> We can guarantee the correctness of our implementation. The poor performance of existing baselines primarily stems from the fact that **none of them consider the scenario where multiple edits are applied simultaneously, nor do they address the resulting conflicts between these edits**. We promise to release the codes upon acceptance of the paper."}, {"review_id": "y59mWkRW9c", "reviewer": "Reviewer_dHwZ", "summary": "The paper introduces Compositional Model Editing (CME), where multiple edits must be integrated into LLMs. Existing methods fail due to knowledge loss, incorrect precedence, and knowledge sinking. The authors propose A³E, an adaptive editor that combines and merges edits effectively, improving compositional performance by 22.45% without hurting single-edit accuracy.", "questions": "I have covered the questions I want to ask in the weakness section. I would increase my rating if the authors can address the concerns.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The compositional model editing is a interesting direction and worth to explore.\n- They propose a new benchmark for compositional model editing.\n- They investigate the issues of existing model editing techniques.\n\nWeakness: \n\n- The first thing that comes to mind is **when** we need composition and **when** we don’t. In some cases, composition is necessary; in others, we may need to revert previously edited knowledge. A better approach for compositional model editing might be to inject complete knowledge into the model and erase prior incomplete knowledge. For example:\n  - **Q**: What are the symptoms of COVID-19?  \n    - **Edit 1**: \"Fever\"  \n    - **New correct edit**: \"Fever, loss of taste\" (rather than adding \"loss of taste\" as a separate, independent edit)\n\n- How do you handle continuous editing situations, when previously edited knowledge needs to be changed? For example:\n  - **Q**: What are the symptoms of COVID-19?  \n    - **Edit 1**: \"Fever\"  \n    - Then, we want to update the answer to \"loss of taste\" (not \"Fever, loss of taste\")\n\n- Additionally, I’m not sure how you evaluate the baseline methods, since the details are not included in the paper or appendix. I assume you perform two sequential edits and test whether the model can combine both. If that’s the case, I don’t think it’s a fair comparison, since most baseline methods would treat the latest edit as the new ground truth rather than combining edits. As mentioned in the first point, a better editing strategy would be to erase the previous, incomplete knowledge and then insert the updated, correct information.", "was_revised": true, "final_justification": "Thanks to the authors for their detailed responses. My main concern about the motivation of compositional editing has been addressed, so I would like to increase my rating.", "rebuttal": "We sincerely thank Reviewer dHwZ for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.\n\n**W1**. The first thing that comes to mind is when we need composition and when we don’t. In some cases, composition is necessary; in others, we may need to revert previously edited knowledge. A better approach for compositional model editing might be to inject complete knowledge into the model and erase prior incomplete knowledge.\n> We greatly appreciate the reviewer for taking this point into consideration, which means that the reviewer has given in-depth thought to the significance of the compositional model editing scenario. In fact, we also took this into account in our research but finally chose to perform each edit independently for the following reasons, which we promise to discuss in the revised version: \n> \n> + The reviewer's example actually falls under standard **non-compositional model editing**, not **compositional model editing**. This is because its assumption is that when editing the question \"What are the symptoms of COVID-19?\", we already know the standard answer is \"fever, loss of taste.\" The example provided by the reviewer can be resolved by A$^3$E in two steps:\n>   1. Locate and delete the outdated edit \"fever\" (refer to W2 for details).\n>   2. Update the answer to \"fever, loss of taste\" with non-compositional model editing.\n>   \n>   We demonstrated in Fig. 8\\(c\\) that the performance of A$^3$E in non-compositional model editing is on par with baselines.\n> + However, **compositional model editing addresses a different scenario** where: during the edit training for \"fever\" and the edit training for \"loss of taste\", **they are unaware of each other**. We will explain why this scenario is meaningful below.\n> + **As stated in Sec. 2.1 (cf. Lines 97–99) and illustrated in Fig. 1, we believe that it is necessary to support model editing in a federated manner.** Specifically, when an open-source model is deployed globally, numerous distinct errors may arise simultaneously across different deployed regions. We want these deployers can benefit from the edits of each other. \n>   - In such cases, similar to the assumption that raw data cannot be shared in federated learning due to regulations like GDPR [1], **different deployers are only able to share trained edits**. In such cases, when a deployer receives trained edits from other deployers, it does not know the questions of the edits. **It is impossible to get complete answers during edit training** because:\n>     1. The deployer has to iterate through all previous edit questions to determine which previous edits need to be composed with the newly received edit to form a complete answer.\n>     2. However, storing all previous edit questions is **impossible in lifelong model editing**.\n>   - When exchanging raw data is allowed, **local deployers still have to perform immediate model editing to mitigate potential adverse effects without waiting to receive errors discovered by other deployers.** In this scenario, assuming that each of $n$ deployers updates an answer to the same question and exchanges these $n$ different answers with each other. \n>     - Continuously injecting complete knowledge would require **$n$ times edit training for each deployer at most**.\n>     - Exchanging independently trained edits would only require **$1$ time edit training for each deployer**. \n>    \n>     Training edits independently while ensuring their composability supports model editing in a federated manner **much more efficiently**. This is especially important in the field of model editing, which emphasizes **immediate correction of errors**.\n> \n> + Meanwhile, for **multi-question composition**, compositional model editing is clearly meaningful because,\n>   - Edits for any two different questions may need to be composed during testing.\n>   - Different errors to be edited may be discovered at different time.\n>    \n>   Therefore,\n>   - It is impossible to exhaustively cover all possible composition cases for complete knowledge during the edit training process.\n>\n> [1] The eu general data protection regulation (gdpr)\n\n**W2**. How do you handle continuous editing situations, when previously edited knowledge needs to be changed?\n> We greatly appreciate the reviewer for drawing our attention to this point, which indeed increases the practicality of A$^3$E. When adding the edit for \"loss of taste,\" if we recognize that the existing edit for \"fever\" is outdated, **our method can easily remove the outdated edit** with the following locate-and-delete algorithm:\n>  \n> 1. Iterate and mask each edit $(\\mathbf{A},\\mathbf{B})$ in the retrieved part $\\mathcal{S}$ of the vector database during the generation process -> locate the edit that leads to the outdated answer. \n> 2. Delete the located edit.\n> \n> We consider a three-answer composition scenario with $3\\times50$ successful edits to test the effectiveness of the locate-and-delete algorithm above, where each question receives two correct edits and one outdated edit. **The accuracy of deleting the outdated edits is 100%**. We will encourage the construction of outdated edits feedback and avoidance mechanisms when deploying A$^3$E.\n\n**W3**. The concern about unfair comparison with baselines.\n> We would like to humbly clarify that **our evaluation is fair for evaluating the composability**.\n> + **We adapt all baselines to make it applicable to multi-answer composition without treating the latest edit as the new ground truth.** As shown in Fig. 1, our evaluation of all baselines follows the protocol of **training multiple edits for one question independently and then composing them to ensure a fair comparison, focusing on evaluating the composability**. Please refer to Sec. 2.2 for the edit composing of the baselines. Specifically,\n>   - Each edit is trained without knowledge of other edits for the same question. In IMAC (cf. Lines 269-274), to accurately evaluate the improvement in composability brought by A$^3$E, edits for the same question are trained on top of **the pre-trained model** and then composed. In MAC (cf. Lines 269-274), to verify that we also significantly improve composability in lifelong scenarios, edits for the same question are trained on top of **previous edits for different questions** and then composed. In this way, we adapt the baselines to compose edits. We elaborate on the necessity of this evaluation scenario in W1. \n> + Regarding the reviewer’s suggestion to “erase the previous, incomplete knowledge and then insert the updated, correct information,” we provide a detailed discussion in W1 on why this approach is infeasible for compositional model editing."}]

Author rebuttal:

Meta-review: **Summary**

This paper addresses a critical limitation of existing large language model (LLM) editing methods—their focus on isolated, single edits—by defining and benchmarking Compositional Model Editing (CME), a task requiring LLMs to integrate multiple independent edits to answer real-world multifaceted questions (e.g., combining "COVID-19 causes fever" and "COVID-19 causes loss of taste" to list both symptoms). The authors identify three CME-specific failures in existing methods: knowledge loss (perturbed/missing edited answers), incorrect preceding (correct answers but wrong output order), and knowledge sinking (generating only partial answers before non-answer content). To solve these, they propose A³E, a framework with two core stages: (1) adaptive combination/regularization of pre-trained foundation knowledge during edit training, and (2) adaptive merging of multiple edits during inference. Key findings include large improvement in composability over baselines, preserved non-compositional editing performance, and negligible computational overhead, with validation on Llama3-8B, Mistral-7B, and (post-rebuttal) Llama3-70B.

**Strengths**

1. CME directly addresses a real-world gap ignored by prior work, where real queries often require integrating multiple edited facts (e.g., medical symptom lists). All reviewers acknowledged this as a pressing research need, with Reviewer e3yR noting it is "practically relevant" and Reviewer cjhv labeling it "novel and practically important."

2. A³E’s three adaptive components (combination, regularization, merging) are tightly linked to the identified CME failures. For example, adaptive combination targets knowledge loss by integrating pre-trained foundation knowledge into edits, while adaptive merging resolves conflicts between multiple edits. Reviewer qR3R highlighted the method’s "conceptually clear adaptive mechanisms," and Reviewer cjhv emphasized that each component "corresponds to the failures of prior works."

3. The paper initially evaluates across datasets (PEAK-CF), models, and noise settings. Post-rebuttal, it expands to classic non-compositional datasets (ZsRE, Counterfact) to confirm performance preservation, Llama3-70B to validate scalability, and composition numbers up to 10 to test robustness. Reviewer e3yR praised the "extensive experiments on thousands of edits," and Reviewer dHwZ noted the value of the new CME benchmark.

4. A³E maintains leading performance while introducing negligible computational/memory overhead. Detailed post-rebuttal analysis (FLOPs, trainable parameters for Llama3-8B/70B) shows its overhead is orders of magnitude smaller than baselines (e.g., 39.94KB trainable parameters per edit for Llama3-8B vs. 0.54GB for ROME). This addresses Reviewer qR3R’s concern about computational cost and ensures real-world usability.

**Weaknesses**

1. The original submission suffered from unclear variable definitions and non-standard notation (A/B instead of transformer K/V), making the method hard to follow. Reviewer qR3R rated clarity as "poor" and noted reliance on "prior works and appendices." While the authors added a notation table and revised naming (e.g., "adaptive combination" → "adaptively combine pre-trained foundation knowledge") in rebuttal, the initial lack of accessibility delayed understanding.

2.  The paper originally tested only small models (Llama3-8B, Mistral-7B) and lacked analysis of failure cases (overlapping/contradictory edits) and large composition numbers (≥10). Reviewer e3yR criticized the "insufficient" model size coverage, and Reviewer cjhv questioned generalization to "10+ edits." Though these gaps were filled in rebuttal (e.g., Llama3-70B results, 10-edit experiments), they should have been part of the initial submission.

3. A³E focuses on white-box models (requiring weight access) with limited initial discussion of black-box scenarios that is critical for extremely large or proprietary LLMs. The rebuttal clarifies that A³E follows white-box editing conventions and outlines future text-level adaptations for black boxes, but does not resolve the current limitation. Reviewer e3yR’s question on "applicability to black-box LLMs" remains partially unanswered.

4. The links between CME failures and A³E’s solutions are primarily empirical. For example, the connection between knowledge loss and editing W_up (instead of W_down) is supported by performance figures but lacks theoretical justification. Reviewer qR3R noted the "heavily empirical" nature of problem-solution links and called for more "theoretical analysis," which the rebuttal did not fully address.

**Discussions during Rebuttal Period**

During the rebuttal period, reviewers raised four categories of concerns, all responded with concrete actions by the authors:

1. About readability and notation. Reviewers (e.g., qR3R) criticized undefined variables and ambiguous method naming. The authors responded by adding a detailed notation table (defining W_up, W_down, etc.), revising method names to be more descriptive (e.g., "adaptive merging" → "adaptively merge different combinations of pre-trained foundation knowledge"), and promising a running example in revised Fig. 7. This resolved Reviewer qR3R’s "poor" clarity rating, as they confirmed their concerns were addressed.

2. About evaluation gaps: (a) Non-compositional performance (Reviewer e3yR): Authors added results on ZsRE/Counterfact, showing A³E matches baselines (e.g., 97.00 SR-1 on ZsRE vs. 96.69 for GRACE). (b) Large models (Reviewer e3yR): Added Llama3-70B experiments, where A³E achieved 50.69 SR-S vs. 28.12 for MELO. (c) Failure cases/large composition numbers (Reviewers e3yR, cjhv): Conducted analysis of overlapping/contradictory edits (A³E outperforms baselines even with high overlap) and 10-edit composition (17.20 SR-1 vs. 12.20 for MELO). These additions validated A³E’s robustness and scalability.

3. About computational overhead. Reviewer qR3R requested detailed cost analysis. The authors provided tables for Llama3-8B/70B, quantifying trainable parameters (39.94KB for A³E vs. 0.54GB for ROME) and extra FLOPs (negligible vs. 16u+16v GFLOPs for 8B model inference). This confirmed A³E’s efficiency, addressing the concern.

4. About method clarity and fairness: (a) Distinguishing "incorrect preceding" and "knowledge sinking" (Reviewer cjhv): Authors provided contrasting examples (e.g., "fever, stomachache, loss of taste" = incorrect preceding; "fever → fever is a…" = knowledge sinking). (b) Baseline fairness (Reviewer dHwZ): Authors clarified baselines were adapted to CME (training edits independently, no latest-edit bias) and justified federated editing scenarios (GDPR-compliant edit sharing). Reviewer dHwZ acknowledged this resolved their "fair comparison" concern and raised their rating.

The authors proactively resolved reviewer concerns with concrete additions (e.g., notation tables, failure case analysis, running examples), strengthening the paper’s rigor and readability. All reviewers acknowledged the response.

**Decision Justification**

A³E fills a critical, understudied gap in model editing, with strong empirical support and practical utility. Unlike incremental improvements to single-edit methods, CME addresses a real-world limitation of LLMs (multifaceted query answering) that prior work ignored. This aligns with NeurIPS’ focus on impactful, real-world AI research. Meanwhile, the authors comprehensively addressed all major reviewer concerns: readability (notation tables, revised naming), evaluation (large models, classic datasets, failure cases), and computational cost (detailed FLOPs analysis). These additions transformed a borderline submission into a rigorous one.
Decision: Accept (poster)