MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMs
Authors: Yupu Gu
Rongzhe Wei
Andy Zhu
Pan Li
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Editing; Mixture-of-Experts; Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Knowledge editing (KE) is crucial for making precise modifications to factual knowledge within large language models (LLMs). Existing KE methods, however, are primarily designed for dense architectures, limiting their applicability to the increasingly popular sparse Mixture-of-Experts (MoE) models that power modern scalable LLMs. While MoEs offer remarkable efficiency and capacity scaling, their unique structure introduces new challenges for KE. Naively adapting dense-model editors to MoEs is not only computationally expensive but also induces routing distribution shifts that degrade model stability and consistency. To address these challenges, we introduce MoEEdit, the first systematic framework for routing-stable knowledge editing in MoE LLMs. Our approach reparameterizes expert updates through per-expert null-space projections, ensuring router inputs remain invariant to suppress these shifts, and solves the resulting block-structured optimization with an efficient block coordinate descent (BCD) solver. Experiments demonstrate that MoEEdit achieves state-of-the-art efficacy and generalization, while maintaining high specificity, routing stability, and superior computational and memory efficiency. Our work establishes a robust foundation for scalable and precise knowledge editing in modern sparse LLMs by highlighting the necessity of routing-stable interventions.

Published as a conference paper at ICLR 2026
MOEEDIT:
EFFICIENT AND ROUTING-STABLE EDITING FOR MIXTURE-OF-EXPERTS
KNOWLEDGE LLMS
Yupu Gu', Rongzhe Wei", Andy Zhu", Pan Li² TTiinghua University, 2Georgia Institute of Technology guyp228mails. tsinghua. edu.cn, cn, {rongzhe. wei, azhu311, panl i eeattegggtthheeeee. edu
ABSTRACT
Knowledge editing (KE) enables precise modifications to factual content in large language models (LLMs). Existing KE methods are largely designed for dense ar- chitectures, limiting their applicability the increasingly prevalent sparse Mixture- of-Experts (MoE) models that underpin modern scalable LLMs.. Although MoEs offer strong efficiency and capacity scaling, naively adapting dense-model editors is both computationally costly and prone to routing distribution shifts that undermine stability and consistency. To address these challenges, we introduce MoEEdit, the first routing-stable stable framework for parameter- parameter-modifying knowledge editing in MoE LLMs. Our method reparameterizes expert updates via per-expert null-space pro- jections that keep router inputs invariant and thereby suppress routing shifts. The resulting block-structured optimization is solved efficiently with a block coordinate descent (BCD) solver. Experiments show that MoEEdit attains state-of offthhe-rr art efficacy and generalization while preserving high specificity and routing stability, with superior compute and memory efficiency. These results establish robust foundation for scalable, precise knowledge editing in sparse LLMs and underscore the importance of nouting-stable interventions. The code for this work is available at https: thub. com http//TeeeeneGG//MEeaa.
INTRODUCTION
LLMs can store and retrieve substantial factual knowledge (Petroni et al., 2019; Sun et al., 2024), yet they sometimes produce incorrect outdated statements. For instance, asserting that the capital of France is Berlin or misreporting the CEO of a major company. Such errors undermine user trust and constrain deployment in knowledge-sensitive applications (Zhang et al., 2024c; Zhong al., 2023; Wei et al., 2024; 2025a;b). Fully retraining these models or performing broad fine-tuning is computationally prohibitive and can induce catastrophic forgetting of unrelated capabilities (Luo et al., 2025). These limitations motivate knowledge editing. which aims to revise specific facts while preserving the model's 's general behavior (Meng al., 2022; 2023; Mitchell al., 2022a;b).
Most KE methods have been designed for dense Transformer architectures, where all parameters are active for each input (Wang et al., 2024). Broadly, KE falls into two families: prrperter-rreeervin approaches leave base weights unchanged and attach auxiliary mechanisms that conditionally override outputs (e.g., SERAC with an external edit memory and routing module (Mitchell et al., 2022b)), and parameter-modiffing approaches aim to directly update model weights responsible for factual recall. Many methods follow a scate-then-edii paradigm: they identify mediating parameters, often mid-layer feed-forward MLP modules (Geva al., 2021; Dai al., 2022), using causal analyses, and then apply structured weight updates. Representative methods include ROME (Meng et al., 2022), MEMIT (Meng et al., 2023), and PMET (Li et al., 2024). Recent work improves locality by projecting edits into the null space of a preservation set, which reduces interference with unrelated behaviors (Fang et al., 2025).
Work was done when Yupu Gu and Andy Zhu did research internship at Georgia Tech.

---

Published as a conference paper at ICLR 2026
ofate-of-the-att LLMs increasingly adopt Mixture-of of--xperts (MoE) architectures to enlarge parame- ter capacity while maintaining nearly constant computational throughput (FLOPs) (Shazeer et al., 2017). In an MoE layer, trainable router activates small subset of experts for each token (for instance, 8 of 128 in Qwen3-30B-An -33),, yielding sparse, input-dependent computation and enabling marked expert specialization (Lepikhin et al., 2021; Du et al., 2022). However, this sparse, modular design introduces a tripartite challenge for knowledge editing that is absent in dense models.
The first and most direct challenge is compulational complexity. Naively applying dense-model editing techniques would require updating all experts, multiplying the cost by their total number (e.g., 128x) and thus rendering the process computationally prohibitive. This necessitates a targeted approach, but editing only a subset of experts introduces further complications. Second, because the layer's output is gate-weighted combination of multiple expert outputs, any edit must contend with inter-expert coupling. A modification to single expert's 's parameters can be diluted or cause unintended side effects when combined with others, demanding a principled allocation of the update across the appropriate specialists. Third, and most subtly, edits risk inducing routing distribution shifts in subsequent layers. Parameter perturbations in one MoE layer alter the input manifold for downstream layers, causing their routers to select different experts. This cascading effect disrupts the model's learned routing patterns and specialized knowledge pathways, jeopardizing both edit locality and overall model stability. Collectively, these intertwined issues of computational cost, expert coupling, and routing stability make successful and localized knowledge editing in MoEs substantially more difficult than in their dense counterparts.
To address this tripartite challenge, we introduce MoEEdit, an expert-aware editor that reframes MoE knowledge editing as a principled, block-structured optimization problem where each expert constitutes a block. We are the first to formally identify routing-induced instability as a central obstacle successful editing in MoEs. To solve this, we develop a novel per-expert null-space projection that constrains parameter updates to preserve the input to subsequent routers, thereby safeguarding model stability. This technique is paired with a highly efficient randomized block coordinate descent (BCD) solver that tackles computational complexity and inter-expert coupling by strategically updating only the most relevant experts for a given edit. This decoupling ensures our method's is cost scales linearly with the expert hidden size, not the total number of experts, making it highly scalable. Our integrated approach sets a new state-of-the-re on standard benchmarks (COUNTERFACT, zsRE), decisively outperforming cense-model editors adapted for MoEs. This result underscores the necessity of expert-aware routing-stabe interventions tailored to the unique architectural properties of MoE models.
2 RELATED WORK
KE in dense Transformers. KE seeks to revise specific factual associations in LLMs while preserv- ing general capabilities (Zhang et al., 2024c; Zhong et al., 2023). Methods for dense Transformers fall into two families: parameter-moddifing and parameter-preeseriing Within the former, locate- then-edit approaches such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023) directly modify a small set of FFN dowa-projection weights identified via causal analysis, enabling single-fact and batched edits, respectively. Gradient--asee editors such as MEND (Mitchell et al., 2022a) learn hypernetworks that transform fine-tuning gradients into localized weight updates. To improve locality and robustness under sequential edits, AlphaEdit (Fang et al., 2025) projects updates into the null space of a preservation set. Other strands explore instruction-based editing (Zhang et al., 2024a), hybrid ceural-symboll mechanisms (Zhang et al., 2024b), and in-context editing (Zheng et al., or cemi-paraeetric methods (e.g.. SERAC (Mitchell et al., 2022b)) store edits in external memories and perform inference-time routing, trading parametric locality for reversibility and capacity. Notably, LEMoE (Wang & Li, 2024) introduces a Mfxture-of-Experts (MoE) architecture within the adaptor itself to manage lifelong editing. However, LEMoE functions as a parameter-preseee preserving framework that attaches external modules to frozen backbone (typically dense). It addresses routing consistency within the added adaptor rather than the routing distribution shift of the base model itself.
2023). Complementing these, parameter-presennve
MoE architectures. MoE layers scale capacity by activating only a sparse subset of experts per token through a learned router, thereby increasing representational power while keeping FLOPs nearly constant (Shazeer et al., 2017; Lepikhin et al., 2021; Du et al., 2022). Each expert is gated
2

---

Published as a conference paper at ICLR 2026
Challenge
Routing Distribution Shift
Pen-expert Null-space Projection
ourport
p
ourgut
O
Expert Pnterdd
Expert N
Rum a Ay
MoE Layer
TO
North Layer
Editing
inputs invanant
Wi
WX
We
PiKiO
PoK--
Attention
Randomized BCD Solver
Reating shert
North Layer
Router
Roater
Piik en expert
Get on saport- specific updiste,
Loper 100-
Challenge 2: Computational cost
Challenge 3: Expert Coupling
Cleee--fee Onc shet Copplcatty vs.
1000 4010
E:
E4
Optimiae the expert weights
foxed target cutpert indepert
Decoupting
gi
9+
Editing tne effect
E4
Rounter
40
(a) Challenge
(b) Method
Figure I: Overview of knowledge editing in cexture-of-Experee (MoE) LLMs. (a) Challenges: MoE editing hindered by routing distribution shift, high computational cost, and expert coupling. (b) Method: MoEEdit mitigates these issues using per-expert null-space projection to stabilize routing and randomized block coordinate descent (BCD) solver for efficient expert updates.
feed-forward network with its own parameters; the router selects the top-K experts using input- dependent logits and produces gate-weighter sum of their outputs. This conditional computation yields strong expert specialization but also creates editing complications: edits must respect the router's distribution, multiple experts jointly determine the output, and perturbations at one layer can alter downstream routing.
Knowledge editing for MoE LLMs. However, KE for MoE architectures remains largely unex- plored. While methods like LEMoE (Wang & Li, 2024) utilize MoE structures externally, they do not tackle the challenge of modifying intrinsic MoE parameters. Existing techniques, which assume fully active parameters, ill-suited for the conditional computation in MoEs and present intractable trade-off: updating all experts is computationally prohibitive, while updating a subset is unreliable due to stochastic routing. This leaves critical gap for editing framework explicitly designed for the sparse and modular nature of MoE models.
3 PRELIMINARIES
Locate-then-Eate Paradigm (Dense Models). An autoregressive LLM updates the layer-/ hidden state as h' h1 a' v'. where a' and ot are the outputs of the attention and feed-forward (FFN) blocks at layer I, respectively. The FFN output can be written as
v' W out (W' (h + a'))
W out k.
"keys""
out can be viewed as linear associative memory that maps post-gate features ("keys": k a (W!! 7(h'11 + a'))) to outputs ("values": e = v²). factual knowledge is formalized as triples (s, r, o) (subject, relation, object), one can view & as encoding (x, r) and as encoding (Meng et al., 2022; Dai et al., 2022). We adopt a locate-then-ede formulation: given keys K₁ = [k1 [kg] ... k for the new associations and targets V1 [v, |22] ... v,,,, find a small perturbation A single FFN projection W... such that (W... a. + A) z V1 while preserving behavior on a preservation set with keys Ko. This yields the regularized least-squares objective
with layer norm y(-) and nonlinearity (-). Following Geva et al. (2021), W
A =arg min [| + ÁJKi 1ill + || 2 + + A12
(2)
3

---

Published as a conference paper at ICLR 2026
where A2 0 controls locality and conditioning. For clarity, layer indices are omitted below since the formulation applies identically to each layer.
KE in MoE LLMs. Modern LLMs increasingly adopt MoE layers to scale capacity with near- constant FLOPs (Shazeer et al., 2017; Lepikhin et al., 2021). Given an input representation u, trainable router produces logits 8n u en (e,, is the routing embedding for expert n) and selects a small set = TopK(s K) (TopK selects biggest element of $1:N, and typically K < N for MoE models). Let In. denote the router weight for expert n. The MoE block output is router-weighted mixture
N
(exp(8n)/ Ejes exp(8)); n E S 0,
In (u),
In
(3)
otherwise,
where each expert a gated FFN
Eu)u)
w(o down
(Wir)u) u) o
WWii (n) x)
Thus, every expert realizes its own key-value memory: the post-gate feature kn = (W(p)) u) gate of ((Win)) u) serves as key and the linear map Win)) returns a value Un w(())
down kns which the
router aggregates into v En. In on-
Extending Eqn. to MoE, we edit expert-specific projections (W J. (For brevity the editing
down). For edit request i, let k;,n RRde be the post-gate key of expert n, Di.n 20 its router weight, and v; the target output. We consider disjoint sets: edit set & where remain stable. The MoE KE objective seeks small
objective, we denote W, = W
outputs should change and a preservation perturbations (An} that (i) match targets on and (ii) preserve behavior on P:
[An] = arg min
Din Wn + ki vi
gin
kinn
+ A
A
12. (5)
(An)
Compared with the dense case, the router weights {gr,.} couple experts: each example influences up to K experts (Top-K gating), and a naive closed-form solve would require inverting a (Nd,) ) (Ndc) system, computationally prohibitive for large N. This coupling and scale motivate the expert-aware optimization strategy developed in Section 4.
METHOD
In this section, we present MoEEdit, a purpose-built, expert-aware knowledge editor for MoE models. Our approach tackles the unique challenges of MoE editing head-on: (i) it mitigates the routing distribution shift: a key instability when naively apply KE to MoE. models by reparameterizing expert updates through per-expert null-space projections, and (ii) solves the resulting objective efficiently with randomized block coordinate descent (BCD) procedure that scales with the expert hidden size, making large-scale MoB editing tractable.
4.1 ROUTING DISTRIBUTION SHIFT IN MoE EDITING
Editing expert parameters changes the MoE block outputs and, after subsequent normalization and attention. perturbs the input u to the router in the following MoE layers. Following the definition in Section 3, let E, = .ej.... exx] collect the router embeddings at layer f. The router computes logits and mixture weights as = softmax(s:)) 2, where 8t = E u A perturbation applied in layer (l - 1) changes us by du,, thus 8, by òs: E, du, and produces new routing weights gi softmax(s, (s; + ōst). We define the routing distribution shift ògt gí ge- Its magnitude can be quantified over set of prompts using the Kullback-Leibe (KL) divergence or the Routing
For clarity of analysis, use the full softmax distribution and omit the Top- K selection, so that ge remains differentiable for the baseed first-order analysis.
4

---

Published as a conference paper at ICLR 2026
Similarity (RS), which is defined the Jaccard similarity between the pre- and post-edit T expert sets:
s n sportt
RS RS
KL(g: (g))).))))
(6)
| U USppar
where Sp and Spaat denote the expert sets before and after editing, respectively. To characterize öge analytically, we linearize the softmax around 8y:
ge ~J gr + (8t)08:
ögt ~ (8t (8:) Eg due.
(7)
Where ze denotes a first-order Taylor approximation of the softmax function around 8g, when the perturbation ōs; is small, and (s) and sm is the softmax function. This relation highlights a crucial observation: only the component of ou, that lies in the span of Eg influences the routing probabilities, and the Jacobian can amplify such components, potentially destabilizing expert selection. This insight motivates our design, suppressing the projection of perturbations onto span Ex) is key to preventing routing drift.
sm((s sm(s) T is the Jacobian of softmax
4.2 PER-EXPERT NULL- SPACE PROJECTION REPARAMETERIZATION
As established in Section 4.1, suppressing routing drift amounts to ensuring that dus = O on preservation set. To achieve this by construction, we reparameterize each expert update so that its effect vanishes along directions spanned by preservation features. Inspired by the null-space constrained approach of ALPIIADDIT for dense models, we generalize the idea to the MoE setting by computing per-expert projector that filters out harmful update components.
Concretely, recall from Eqn. that expert n produces output W,k,,,, for cost-activation key k Let P denote the set of preservation prompts, and collect their features for expert into the matrix KD = [kim]IeP E [Rdxx|P! The covariance K K captures the subspace of activations wish to keep invariant. We compute its eigendecomposition, K Ko K A,A,, U, n. and select
indices To {p Xn.p < - corresponding to (near- (ncar-)null eigenvalues under small threshold T 0. Let T = Un To] and define the orthogonal projector onto the complement of span ( by P. UHU Intuitively, P. preserves only those directions orthogonal to all preservation features, any update projected by P.. is guaranteed not to alter expert outputs on P.
We then reparameterize the expert update as An A..P. where Á, is the free variable to be optimized. Because P,ki,, = for all i (up to numerical error), the preservation outputs are unaffected: A, P,,,n = O. Consequently, for every E P we have du, (i) = O, and by Eqn. 7 the induced routing shift satisfies (g)(i) O, minimizing Eqn. the preservation set.
Projected editing objective. Let ki,, = P. ki denote the projected key. Substituting A, = Ån Pr into the MoF. objective (Eqn. 5) yields
{ n N arg min (A.)
Σ Ii,n WW,,i, + Ankki,)) Us
+
A
113
(8)
3Eg
where x >> 0 controls the update magnitude and improves locality. No separate preservation term is needed, as P. removes all preservation components by construction.
4.3 RANDOMIZED BLOCKK COORDINATE DESCENT SOLVER
naive step is to solve the projected objective in Eqn. 8 in one shot, just like what we do in dense model KE (Meng al., 2022; 2023; Fang al., 2025). In this subsection, (i) expose the structure of the global elosed-form solution, (ii) explain why the direct route is computationally impractical in MoE, and (iii) arrive at an efficient randomized block coordinate descent (BCD) procedure that scales with the expert hidden size.
The global closed-form (one shot). Let Á, E Rdd. xdy, be the projected free variable for expert and stack all expert updates horizontally as Å = [ ÂN) E Rd x (Ndd), For edit example i, define the base residual (excluding any edits) r; 02
ΣN Di, W. W,kins and the design vector
5

---

Published as a conference paper at ICLR 2026
Φ 1 91,N N E Nd,, where kn,n P Then Equ. 8 becomes regularized
multi- linear regression: min Eieg A rill
> | Vectorizing with
= () E Nds and using vec( A Aeb)) = (2b) @ Id 0, the normal equations take the compact form
Σ(biab) @ Lam + Nd IEE
Nd 0 -=E(th @ Jra
(9)
SEE
with unique minimizer
0 -Matol bglub glob
and Â unvec(@"),
(10)
where Myloh E, @ Id... XI and bglob Appendix B.2.
Σ. (3be @ Ia )r A proof is provided in
Although Eqn. 9 elegant, it is not practical editing primitive at MoE scale. First, even exploiting the Kronecker structure, the system decomposes into dm independent problems of size (Nd2) x (Ndg) each. For typical MoE layers, can be 8-128 and dę in the thousands, so factorizing dm such systems (and re- re-tatcori as & changes) is prohibitively expensive in both time O (d [Ndx)²) and memory (d (NV (2)) Second, while Top-K routing makes each vb, K-block sparse, the accumulated Gram matrix E. sheeb? quickly densifies, yielding substantial fill-in under Cholesky/LDL.. These realities make the one-shot solve not suitable for fast, iterative MoE editing.
From global to block: randomized BCD. Since the insight that every expert is natural chunk. The structure of Eqn. 8 suggests block strategy: treat each expert as block and optimize one block while holding the rest fixed. This reduces the problem to sequence of well-conditioned, small ridge least-squares solves of size dç x dç.
Fix {Ae)en and define the residual that excludes expert nt:
(-n) U,
Σ
(Wekre Bcks.). E/n
(11)
The subproblem in Ån becomes the ridge-regularized least squares
min A
||r(--)
Di,n ki Al An.
(12)
3E9
Its normal equations
An
Σ
Fi, kinnk + AI)
Dint
(nn) ( k
(13)
iEE
ieg
M,, Xdd
B ER
admit the closed-form update
A B,M
(-n) E Rirry Efunr
T.T (Σ
Fo.n k. + +
(14)
sEE
We then write to parameters via the projection A A.P. and move to the next block.
Practicalities and complexity. We traverse experts in randomized order and update only those active in the current minibatch, which further reduces cost. For each updated expert, forming M,, costs ((E)))) and inverting it costs Odd)). typically modest since dy & dm. We stream-accumulate B, and M,, cache ki... and use Cholesky with diagonal loading for numerical stability. Because Eqn. 8 is strictly convex quadratic in { { ). (randomized) BCD with exact block solves converges globally under standard conditions (Tseng, 2001; Richtárik Takáč, 2014). Empirically we see fast decrease within a few passes (S 10).

---

Published as a conference paper at ICLR 2026
Table 1: Sequential knowledge editing on MoE LLMs. Eff.. Gen., Spe. denote Efficacy, Generalization, Specificity; Uti. is their mean (higher is better f). Best in bold, second-best underlined.
COUNTERFACT
ZsRE
Method
Model
Eff.
Gen.
Spe.7
Uti.
EMT.
Gen.t
Spe.1
Uti
Pre-edited
13.30.0.44
15.101031
84,45.0..4
37.62
41.30.0.99
40.50
40.91
40.90
FT
Qwend: 300 A3D
80.70
63,9510.43
41,44
62.03
6,44.0..4
6.13
2.15
4.91
FT-L
82.40
22.7510.33
11.28.10.5
58,88
44. 19.09
42.46.029
41.92-127
42.86
AdaLoRA
51,90
49.75..0.00
18.101...6
49.92
3.66.10.9
3.60
4,68
3.98
UnKE
89.30
82.85
18.15
73.43
31.43.028
29.78.021
25.30
28.84
MoBEdii
20.00
94.10.0..0
80.97
91.46
84.47
42.82
68.43
Pre-odited
11.80
14.70=0.31
84,53++0.44
37.01
33.20=a01
32. 22
28.02=
31.12
FT
83,40.00.7
58,40.1042
55.72.00.3
65, 65,84
25.57
23.41:1026
17.61
22.20
FT-L
73,80 1.044
43.10..0.4
59.75.10.3
58.88
32.75.029
33.09
30.06
31.97
AdaLoRA
62, 02.40
55,0010.4
43.65100.4
53.68
43.4610.30
42.9610.30
33.60
40.01
UnKE
78.00=0.1
44.40=0042
73.91
65.44
46.58=a31
43.99=031
31,40=0226
40.66
MoHEdit
05.90
44. 10
81.09+0.25
73.70
81.68=025
68.44=034
32.55=002
60.89
5 EXPERIMENTS
5.1 BASELINES, DATASETS, AND METRICS
We evaluate two modern MoE LLMs on standard factual-editing benchmarks: Qwen3-30B-
A3B (Yang et al., 2025) (128 experts; top-8 per token) and GPT-OSS-20B (Agarwal et al., 2025)
(32 experts; top-4 per token). As baselines, we adapt parameter-editing methods originally designed
for dense Transformers but directly applicable to MoE models: Fine-Tuning (FT) (Zhu et al., 2020), FT-L (FT with a norm constraint). AdaLoRA (Zhang et al., 2023), and UnKE (Deng et al., 2025).
Following prior work, use COUNTERFACT (single-hop counterfactual edits introduced with MEMIT) (Meng et al., 2023) and ZsRE (zero-shot relation extraction) (Levy et al., 2017). Visualized dataset examples are provided in Appendix D for unfamiliar readers. We report the standard editing metrics (Meng et al., 2022; 2023; Mitchell et al., 2022a): (i) Efficacy (edit success on edited prompts), (ii) Generalization (success on paraphrases and lightly perturbed contexts), and (iii) Specificity (locality on unrelated controls). Unless stated otherwise, we perform sequential batched edits. And to summarize the overall trade-off, we additionally report Utility as the mean of the three metries. See Appendix A for full calculation details.
5.2 MAIN RESULTS ON KNOWLEDGE EDITING
We perform 1,000 sequential edits on each dataset (COUNTERFACT and ZsRE) with a batch size of 50 edits for all methods. Table 1 summarizes results Qwen3-30B-Ae A3B and GPT-OSS-2OB. As shown Table 1, MoEEdit consistently delivers outstanding results. On COUNTERFACT, it achieves over 90 efficacy and specificity, respectively. On GPT-OSS-2000 20B, although FT attains slightly higher generalization, MoEEdit still provides the best overall balance, with clear gains in efficacy (+ and specificity (+7.2). On ZsRE, MOEEDIT also demonstrates large improvements in efficacy and generalization, over +30 points against the strongest baselines, while maintaining competitive specificity (within 1 point of AdaLoRA). These results highlight that MOEEDIT offers the most favorable trade-off' between accuracy and locality across models and datasets.
both backbones, substantially outperforming UnKE and FT-L in generalization
5.3 MAIN RESULTS ON ROUTING DISTRIBUTION SHIFT
We analyze routing distribution shifts under sequential editing. Similar to Section 5.2, we perform 1,000 edits with a batch size of 50. To control for depth, all methods are constrained to update at most the top editing layer (layer 7). Specifically, MoEEdit applies updates across layers (3.4.5,,,7) using BCD, while FT, FT-L. UnKE, and AdaLoRA are restricted to layer 7. We evaluate on Qwen3-30B. A3B / COUNTERFACT and report the routing-similarin (RS) metric between pre- and post-edit Top-K expert sets (Equ. 6), averaged over windows of 10 layers, for both the editing and preservation sets. The editing set consists of the 1,000 edited samples, while the preservation set is formed by
7

---

Published as a conference paper at ICLR 2026
Table Routing distribution shift on Qwen3-30B-A3B. Values are Jaceard similarity (RS1) between pre- and post-edit routing distributions. Higher is better. Best results are bold, second-best are underlined.
Editing Set RS1
Preservation Set RS1
Method
Model
Lay. 11-20
Lay. 21-30
Lay. 31-40
Lay. 11-20
Lay. 21-30
Lay. 31-40
FT
23.57
26.58
29.98
24.72
27.45
30.97
FT-L
Qween 0o0.33300 A38
47.01
51.20
53.68
48.80
50.17
53.45
AdaLoRA
16.63
24.11
27.00
16.38
23.84
26.60
UnKE
52.46
44.12
44.80
49.90
41.91
43.84
MoBEdit
86.62
88.16
89.93
87.02
88.55
90.22
Editing Set
Preservation Set
1.0
RS (Top-4 havard))
0.8
0.6.
0.6
(Top-8
0.2
0.2
Layer Index
Layer Index
McHEdR
FT-L
AdaLoRA
UnKE
Figure Routing similarity (RS) before and after editing on the editing and preservation sets. MoEEdit achieves consistently high RS, demonstrating strong routing stability.
sampling an equal number of untouched examples from the remaining dataset. Table summarizes the results.
Projection suppresses routing drift and preserves stability. As shown in Table 2. methods directly extended from dense models exhibit substantial routing drift, whereas MoEEdit maintains consistently high routing stability (average RS > 88 across all layer ranges for both sets). Considering that Qwen3- 30B-A3B activates experts per token, the average number of non-overlapping experts before and after editing is close to one, which is negligible. This observation aligns with the heavy-tailed nature of routing: small perturbations primarily affect low-weight expert selections that contribute little to the output. The average KL divergence between pre- and post-edit routing distributions for MoEEdit only 0.02, indicating minimal shift. Furthermore, Figure plots routing similarity across layers for both editing and preservation sets. AdaLoRA and FT exhibit the lowest RS values across layers due to their unconstrained updates that heavily disrupt routing patterns. contrast, MoEEdit preserves routing stability across all layers and consistently outperforms all baselines.
5.4 ABLATION STUDY
Effect of Projection. We ab- late the projection matrix in MoEEdit to evaluate its contri- bution to routing stability. As shown in Table 3, removing projection reduces RS by an average of 14.81 points on the editing set and 15.21 on preservation set. The KL diver- gence also increases from 0.02 to 0.0834, confirming that pro- jection is critical for suppress- ing routing drift. These results validate the projection design introduced in Section 4.2.
Table 3: Ablation on the projection matrix. Removing projection signifi- cantly weakens routing stability.
RS
Method
Set
Lay. 11-20
Lay. 21-30
Lay. 31-40
MoEEdit
86.62
88.16
89.93
Edit.
MoEEdit (w/ Proj)
73.64
72.90
73.75
MoEEdit
87.02
88.55
90.22
Pres.
MoEEdit (w/o Proj)
73.59
73.08
73.50
8

---

Published as a conference paper at ICLR 2026
Convergence of RCD (Reconstraction error)
Runtime Number of Experts
2- 01
Global solver
A
BCD (2 passes)
80
10'
Runtime
60
10
10"
20-
10"
10
100
120
Number experts
Reconstruction
(b) Scalability
Figure 3: Comparison of solvers. (a) BCD achieves fast convergence with suitable A. (b) BCD scales efficiently with the number of experts, while the closed-form solver quickly becomes infeasible.
BCD Solver vs. Closed-form Solver. We compare the proposed block coordinate descent (BCD) solver with a naive closed-form solution. The latter requires inverting large matrices that scale with the number of experts and constraints, making it computationally infeasible at realistic scales. To enable comparison, we construct a controlled synthetic batch and evaluate (i) convergence and (ii) scalability. Figure 3 illustrates two perspectives our BCD solver: (a) reconstruction error convergence under varying X, and (b) runtime scalability with respect to the number of experts. Smaller À values (e.g., 10 1010 1003) achieve lower error, whereas larger values converge to higher eroor floors. Panel (b) shows that the closed-form solver exhibits near-quadratic runtime growth and becomes infeasible beyond N 60, while BCD maintains nearly constant runtime up 128 experts. Thus, BCD scales linearly with hidden size rather than the total number of experts, ensuring practical efficiency.
Number of BCD Passes. We vary the number of BCD passes (2 4, 6, 8, 10, 12, 14, 16, 18, 20) while keeping all other settings fixed, and evaluate efficacy, generality, specificity, and their harmonic mean. As shown in Figure 4, as the number of BCD passes in- creases, both efficacy and generality rise rapidly the early phase and then plateau, reflecting dimin- ishing returns beyond moderate passes. Specificity shows more stable, gradual upward trend. Since each subproblem is convex, early passes remove dom- inant residuals, while later passes only refine small residuals, yielding slower gains. This hehavior sug- gests that small number of passes (e.g., 6-10) already achieves a favorable trade-off between performance and efficiency. Beyond this range, additional passes bring only marginal improvements while increasing runtime, highlighting the practicality of moderate BCD iterations in lcage-scale editing scenarios.
100
80
Score (%)
60
Efficiiency Generality
Specificity
Avg
10
is
20
Number of BCD Passes
Figure 4: Ablation the number of passes.
DISCUSSION AND CONCLUSION
In this work, we presented MoEEdit, routing-stable knowledge editing framework tailored for Exxture-of-Expertt (MoE) LLMs. Our approach addresses the unique challenges of computational cost, inter-expert coupling, and routing drift by combining per-expert null-space projection with an efficient block coordinate descent solver. Extensive experiments on COUNTERFACT and ZsRE benchmarks demonstrate that MoEEdit achieves high efficacy, strong generalization, and routing stability, all with superior efficiency compared prior methods. Beyond empirical performance, our findings highlight several broader insights. First, expert-aware design is crucial: naive adaptations

---

Published as a conference paper at ICLR 2026
of dense-model editors to MoEs fail to maintain stability and efficiency. Second, routing stability emerges as a central factor in editing sparse architectures, where even small perturbations can cascade through routing distributions. By explicitly controlling for this effect, MoEEdit offers principled solution that preserves locality without sacrificing scalability. In summary, MoFEdit establishes a robust foundation for precise and scalable knowledge editing in sparse architectures, advancing the state of the art and paving the way for more adaptive and trustworthy MoE-based language models.
ACKNOWLEDGEMENT
R. Wei and P. Li are partially supported by the NSF under awards PHY-21 113-2239655, IIS-2428777, and CCF-2402816; the JPMorgan Chase Faculty Award; and the OpenAl Researcher Access Program Credit, the Nvidia Academic Award, and the Google Academic Award.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain. Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lemnon, Scott Lessans, Mario Sezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki. Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo. Jongsoo Park, Leher Pathak, Carolina Paz. Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Prochl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer. D. Sculley, Harshit Sikehi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang. Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong. Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss-1200 120b & gpt-oss-20b model card, 2025. URL /ttps://areee wrg/obr//////////o aba//bbs///// 10925.
Damai Dai, Li Dong, Yaru Hao, Dian Sui, Songhao Piao, Longyue Dou, Weinan Wang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8493-8502, 8502, 2022. doi: 10.18653/v1/ a022.acl-long.55
Jingcheng Deng, Zihao Wei, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. Everything is editable: Extend knowledge editing to unstructured data in large language models. In ICLR, 2025. URL htt.t:tps //oe:nr////////////// eview. net/forum htt:/forumiiddxxe wir.iddd=xrrryrre
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph. Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Ka- malika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (cds.). Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR, 17-23 Jul 2022. URL https://proceent
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Represenaations 2025. URL https: wwwparrwweew.ntttffooomiiissssss
10

---

Published as a conference paper at ICLR 2026
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1111665/1120222.emlp-mmaala "aaclantholoog.orgg///2 wttp:///.22....mmmmmaai.ii.. 446/.
Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-
URL https:
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021. URL https: /ttp::://///////////eeeee wwwppenrreiw....m....... net/ pet/fooum?ee dd==rweeXHTTmmb.....
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 2 Conference on Computational Natural Language Learning (CoNLL 2017). pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/K17-I 1034. URL https:/ / https://aclanthologg. wrt.////og//1/11/////
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17): 18564-18555 18572, Mar. 2024. doi: 10.1669/aaii v38117.29815 URL https: / //ojs. aaai.orgaa org/index. phtp/aat//rriill///ieww///
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing, 33:3776-3786 2025. doi: 10.1109/TASLPRO. 2025. 3606231.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associa- tions in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387101
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. The Eleventh International Conference on Learning Representations, 2023. URL httpss//opent www.opppnneviii................m httt//oormmii=dMbccAAHt
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022a. URL https:/ httpss:///ppeeeveeeeeee www.tneeet/eeeeee (wt.forrmmddo0ooeeeeoooo
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory- based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1581 15817-15831. PMLR, 17-23 Jul 2022b. URL htttps://proceete/ee
mlr. www..preesvvvvve w2tmm/112mmmmmmmm wttp:/22atthhh1222...... html.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.11653/v1/D19- 1250. URL D19-1250/.
org/
Peter Richtárik and Martin Takáč. Iteration complexity randomized block-coordinate descent methods for minimizing composite function. Mathematical Programming, 144:1-38, 2014.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://opperreview.rr net/ forum?i d= d ckMDqlg.
11

---

Published as a conference paper at ICLR 2026
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 311-325, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 118853//112022.aaaccaaaa https: / http://///aaaahhlooggooo orgg2024, naacl-long. 18/.
18. URL
Paul Tseng. Convergence of block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications, 109(3)4475-494 2001.
Renzhi Wang and Piji Li. LEMoE: Advanced mixture of experts adaptor for lifelong model editing of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2551 2575, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emmll main. maapp.main.1999 149. URL https: httpss:///ccaaaaaaaaaaaae org /024..222222 emn01...mmmppppmmmmmmmmmmmmmpp p-mmnin. 149/.
emnlp
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey, ACM Comput. Surv., 57(3), November 2024. ISSN 0360-0300. doi: 10.1 145/3698590. URL hittps://dof org/10 /10. ......1556 1145/3698590.
Rongzhe Wei, Mufei Li, Mohsen Ghassemi, Eleonora Kreačié, Yifan Li, Xiang Yue, Bo Li, Vamsi Potluru, Pan Li, and Eli Chien. Underestimated privacy risks for minority populations in large language model unlearning. arXiv preprint urXiv:241 2,8559, 2024.
Rongzhe Wei, Peizhi Niu, Hans Hao-Hsun Hsu, Ruihan Wu, Haoteng Yin, Mohsen Ghassemi, Yifan Li, Vamsi K Potluru, Eli Chien, Kamalika Chaudhuri, et al. Do llms really forget? evaluating unlearning with knowledge correlation and confidence awareness. arXiv preprint arXiv:2506.05735 2025a.
Rongzhe Wei, Peizhi Niu, Xinjie Shen, Tony Tu, Yifan Li, Ruihan Wu, Eli Chien, Pin- Yu Chen, Olgica Milenkovic, and Pan Li. The trojan knowledge: Bypassing commercial I1m guardrails via harmless prompt weaving and adaptive tree search. arXiv preprint arXiv:25/2.01335 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL wttph:tps//aat //arxiv.. org/ abs/ 2505. 09388..
Ningyu Zhang, Bozhong Tian, Siyuan Cheng. Xiaozhuan Liang. Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, and Huajun Chen. Instructedit: enttuuttion-base knowledge editing for large language models. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24, 2024a. ISBN 978- 188-1-95667 19811--55792-04- doi: 2224963/jcai. URL
https://doi.org org/10 24963/ 14963/ijcai..e 222443/.......3.....
Ningyu Zhang, Zekun Xi, Yujie Luo, Peng Wang, Bozhong Tian, Yunzhi Yao, Jintian Zhang, Shumin Deng, Mengshu Sun, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. Oneedit: A neural-symbolic collaboratively knowledge editing system, 2024b. URL https://arxiv.or wwtprriw.orggabb/////ea htrg/abs/2409.7749/ 074977.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. A comprehensive study of knowledge editing for large language models, 2024c. URL wttptst/p///aiii wttpr/i...oggabb//////// 01286.
12

---

Published as a conference paper at ICLR 2026
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, In International Conference Learning Representations (ICLR), 2023. URL https:://ppnnreview. net/forum? idr
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https::/opeerewee www.:ne//orummiidssssoooooee
Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 15686-15702, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10.1 12858/6112223..mmmlll 18865//1200...mmmpmmmin.... URL https: // //aclanthology.o wwg.e//.rr/222/3 hatp://...rrr222.....mmmmmmaaaaa enn.lpmminnmmn 971/
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer. Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00:: 2020. URL https: //axxiv
http:///wawwwv..aaaaa////aaa org./bs/////...o.. 00363.
13

---

Eg [ ys = arg max Pr(y Published as a conference paper at ICLR 2026
A METRICS
In this section, we introduce the evaluation metrics we use for COUNTERFACT and ZsRE
A.1 ZSRE EVALUATION METRICS
Building on prior studies, define for each ZsRE metric a large language model M, factual prompt (8,, ri), a revised target output y/, and the model's initial output ÿ::
Effectiveness. Effectiveness is quantified as the average top-1 accuracy on edited inputs:
[1(M v arg max Pr(y (st,
Eg
[
(15)
Generalization. This measures the ability of the model to perform correctly on paraphrased prompts ri). is computed as the average accuracy over such rephrasings:
| [N [(S...))))
(16)
y M
Specificity, Specificity ensures that modifications do not affect unrelated cases (2 r1). is defined as:
0: arg max Pr(v (8s.)))) r:)s.
(17)
M
A.2 COUNTERFACTUAL EVALUATION METRICS
Similarly, we introduce Counterfactual metrics for M under the same setup (s,, r,) with target y: and original Ú;:
Effectiveness (success ratio). The share of cases where yı is assigned higher probability than y: under (st,
Es
M Pr(m (si,r:)) > Pr(0: (s.
(18)
Generalization (paraphrase success). The proportion of paraphrased prompts Ñ (st, r:) where y has higher likelihood than d::
E, Pr(u (ss.):)) > Pr(O. (s.r))))
(19)
Specificity (neighborhood success). For neighborhood prompts 2(s r:) that involve related but distinct entities, specificity is the fraction of cases where yi is favored over d::
E,
Pr(ys (((s))))) M
Pr(g. (((...)))). M
(20)
B PROOF & KNOWLEDGE EDITING
B. GENERAL FRAMEWORK OF
Knowledge editing seeks to precisely revise model's behavior to recall new fact (s,r,0") in place of obsolete or incorrect one (s, o), conditioned on prompt p(8, While various techniques exist, the dominant pocate-then-ee paradigm (Meng et al., 2022; 2023) typically decomposes the process into three distinct phases: locating the mediating parameters, computing the optimal local update targets, and solving for the new weights. We formalize this process below using notation consistent with Section 3.
Causal Localization. The initial phase identifies the specific layer and module (e.g., dense FFN or specific experts in an MoE layer) that mediate the retrieval of the factual association. This is commonly achieved via Causal Tracing (Meng et al., 2022). By corrupting hidden states
Step 1:
14

---

Published as a conference paper at ICLR 2026
with noise to degrade the model's prediction and subsequently restoring states at specific layers, one can quantify the Indirect Effect (IE) of each layer on the correct output probability. The layer I* exhibiting the maximal causal influence is selected as the target for editing:
1" = arg max IE()).
(21)
Step 2: Acquiring the Target Output Vector (v*), Once the target layer 1 is identified, must determine the optimal output representation required to successfully trigger the target token o*. Viewing the layer linear associative memory, maps key & (input features) to value (output features). The objective is to identify a new output vector v² such that, if the layer were to produce v², the final model prediction would be o".
This step is formulated as an optimization problem over the hidden state vector rather than the model parameters. Let G(v) denote the function mapping the layer output the final model logits. We freeze the model parameters and optimize perturbation 8 to the original output aiming to maximize the -ig-liielhood of the target object o":
1º + 8", where 8 = arg min log PM (o") do(u + 6))
(22)
Here, the do(.) operator signifies a causal intervention where the layer output is manually set to + 6. A regularization term (e.g., KL divergence) is typically included in the objective to minimize prediction drift for the subject s and relation r, ensuring the edit remains semantically consistent. This process effectively translates the semantic edit target (the token o²) into vector-space target v".
Step 3: Updating Parameters. The final step is to update the projection weights W (corresponding to W in dense models (W,) } in MoE experts) map the specific input key k: the new target v², while preserving unrelated associations. This is formulated as a constrained least-squares problem.
Let & denote the set of edit examples (new facts) and P denote the set of preservation examples (invariant knowledge). We require WK; of for edits E, and Wky vy for preservation samples j E P. The optimal update W minimizes the aggregated error:
W arg min Σ 11Wk uf + Σ IIk, 0311
(23)
JEP
Dense methods such as ROME and MEMIT solve this globally via closed-form solution involving the covariance matrix of the keys. As discussed in Section 4, our proposed MoEEdit adapts this general objective to address the unique constraints Mixture-of- Mfxture-of-Experts architectures.
B.2
SUPPLEMENTARY PROOF
B.2.1 PROOF THE GLOBAL CLOSED-FORM (ONE-SHOT) SOLUTION
Let {ki. nE[]] be the projected keys and Ji,z 2 0 the router weights. For each edit example i E, define the base residual
nE[N]
r: 1i
Rin W.,
(24)
and the design vector
4b: =
kg
Nd. Â N| E Rd x(Nd)), The projected objective
i,N
(25)
Stack the expert updates as Á أ
min
Å 3h - rel
A F
(26)
33g
n=1
has the unique minimizer
0+ M Mglub bglabs
un)e
(27)
15

---

Published as a conference paper at ICLR 2026
with
Mglab
Σ
2 Idm + A Ndu
Dglob Σ (ebs 2 Idmm ri-
(28)
3E
Let m = |
and define the data matrices
bb
R. Xd R [ri rml Rm xm
(29)
The objective becomes
min À (NR(
A R A1
(30)
Taking derivatives gives
V = 2(A R 22
(31)
Setting this to zero yields
Á (W T + (lNa)) RV
(32)
Using vec(XA) = (AT @ I) veec ()), Eqn. 32 becomes
(//)) @ Im + Mdm.Ndd )0 veel (R ))
(33)
where 0 veo Á). Noting that vec(RV Dimh (1b, Id )r, we obtain the system in the theorem statement.
Mglob (MA )) Id.. AI is positive definite for 0, hence invertible. The minimizer is unique.
Thus 0"
MT glob bglob, and reshaping yields Å unvec (0*).
B.2.2 DETAILED DERIVATION OF THE SINGLE-EXPERT SUBPROBLEM
Starting from the projected MoE objective (Eqn. 8),
min
Σ
Σ Ii.n (Waki, + Ank.,)) - 0i KEE ==1
2
+
(34)
(Á. 1
==1
we apply block coordinate descent (BCD) over experts, updating one expert at time and keeping the others fixed. For fixed expert n, collect all terms that do not involve Å, into the external residual
[-))
U. Σ gt (Wek,,e Seks,e), Extm
(35)
and substitute Eqn. 35 back into Eqn. 34. The expert subproblem for Å, the hidge-regularizen least-squares
min An
Σ
r (-))
An | ~112A.11?
(36)
Introduce compact notation
X An. I; = Rs,n,
yi = r (-n) = Jt.n
(37)
so that
Il 9:--giXu Xxil2 + 111 f(X) Σ
(38)
REE
Using the standard matrix derivative identity² yields
Vf(x) = - Σ I.laa +2 2 > o Xxça + 2AX.
(39)
For vectors a, b and matrix X, Ølla Xb112/0X -2(a Xb)b
16

---

Published as a conference paper at ICLR 2026
Setting the gradient to gives the normal equations
x
E Jæ + A1) - =E 9:3110..
(40)
Restoring expert-specifie symbols, define
M,
Ii, hi k AI,
B
Σ
Iin !
k
(41)
iEE
iEE
Then Eqn. 40 becomes Á, M, BR, with the unique minimizer
A
B,
(42)
Since the actual expert update is parameterized via the projection A., A,,,, (Sec. 4.2), the written update is
P
(43)
Why M, is invertible (positive definite). By definition,
M,
E ks k +AIda. IEE
(44)
Gram matrix G 20
For any nonzero 2 E TR
T M, == Σ
pim (= (TTTiin))) 22+ \||2||
(45)
SE8
Since À > 0, then T Mn A||=|12 for all + O, so Mn > 0 and invertible. Moreover, Amin (M,) 2 X, which ensures good conditioning (Tikhonov regularization). Because the projected keys are k; P,,k;, with an idempotent projector P., they lie in range(P,)) When À 0, M,, remains strictly positive definite on the full ambient space (not only on range(P,.) guaranteeing a unique closed-form update 42.
C EXPERIMENTS SETUP
Model Configuration We evaluate method on two Mixture-of- (MoE) models: Qwen3- 30B-A3B* and GPT-OSS-200 Qwen3-30B-AS contains 128 experts per layer with the top-8 experts activated per token, resulting in approximately 3.3B active parameters during inference. The latter, GPT-OSS-201 features 32 experts per layer with the top-4 experts activated, corresponding to approximately 3.6B active parameters.
Hardware and Quantization All experiments are conducted on single node equipped with an NVIDIA H20 GPU. To balance precision and memory constraints, utilize the BF16 format for model weights, while optimization is performed in FP32 to ensure numerical stability.
Fine-Tuning (FT) We evaluate both standard Fine- Tuning (FT) and Constrained Fineinuning (FT-L). The primary distinction is that FT-L imposes norm constraint on the weight update. For both 10-3 for FT-L. We adopt learning rate of 1 x 10-3 for both models. Updates are applied to layer 30 for Awen3-30--A31 and layer 0 for GPT-OSS-20B. For both methods, we target the mlp. experts down_proo module. We train for 25 epochs, setting both weight decay and the KL divergence factor to O.
Dwen3-30B-A3B and GPT-OSS-2OB,
set
UnKE As UnKE employs a two-stage structuring process, we configure the models as follows: For Owen3-30B-ATT the first stage uses learning rate of 5 x 10-1 with 25 optimization steps and weight decay coefficient
x 10 10-3, In the second stage, we apply a learning rate of x 10 and perform 50 optimization steps. For GPT-OSS-20B. the first stage similarly adopts learning rate of 5 x 1 but with 50 optimization steps, utilizing the same weight decay (1 x 10-3). The second
htttps://hugiiggf htttss///hignnnaaar..eeeee/////p wwtp.//peai//gtttoss-22000
17

---

Published as a conference paper at ICLR 2026
stage proceeds with learning rate of x 10 and 50 optimization steps. All experiments restrict parameter updates to layer 7. Consistent with the focus on structured knowledge editing, optimization performed on the last subject token for both models.
AdaLoRA For AdaLoRA, updates are applied across all layers. We set the hyperparameters a = 32 set the learning rate x 10-3 while for GPT--SS--20.
and rank 8. For Qwen3-30B-A.. we use a reduced learning rate x 10 The optimization run for 25 steps for both models.
edit layers {3, 4, 6, 7]. whereas for GPT-OSS-20B we target layer 5. For Qwen3-30B-A3B, we perform 25 optimization steps with a learning rate of 0.1 and execute 4 Block Coordinate Descent (BCD) passes. For GPT-OSS-20B,
MoEEdit (Ours) For Owen3-30B-AA3
perform 50 optimization steps with learning rate of 0.2 and 10 BCD passes. For both models, we set the regularization parameter
and the KL factor 0.0625. We utilize 100,000 samples compute the covariance matrix for the null-space projection with projection threshold = 0.02.
D EXAMPLES OF ZsRE AND COUNTERFACT
18

---

Published as a conference paper at ICLR 2026
"subject": "Watts Humphrey",
"src": "What university did Watts Humphrey attend?", "pred": "Trinity College",
"rephrase": "What university did Watts Humphrey take part in?", "alt": "University of Michigan",
"answers": [
"Illinois Institute of Technology" J,
"loc": "nq question: who played desmond doss father in hacksaw ridge", "loc_ans": "Hugo Weaving",
"cond": "Trinity College >> University of Michigan II What university did Watts Humphrey attend?"
o.
(
"subject": "Ramalinaceae"
"src": "Which family does Ramalinaceae belong to?", "pred": "Ramalinales",
"rephrase": "What family are Ramalinaceae?", "alt": "Lamiinae",
"answers": "Lecanorales"
J. "loc": "nq question: types of skiing the winter olympics 2018", "loc_ans": "Downhill",
"cond": "Ramalinales Lamiinae || Which family does Ramalinaceae belong to?"
o.
(
"subject": "Denny Herzig",
"src": "What role does Denny Herzig play in football?", "pred": "midfielder".
"rephrase"; "What's ss Denny Herzig's role in football?", "alt": "winger",
"answers": "defender"
J,
"loc": "nq question: where does aarp fall on the political spectrum", "loc_ans": "non-partisan",
"cond": "midfielder >> winger || What role does Denny Herzig play in football?"
Figure 5: Examples of ZsRE dataset
19

---

Published as a conference paper at ICLR 2026
:aas__i": 975, "pararel_idxx: 17275, "requested_rewritte. { "prompt": "O, from", "relation_id"; "P127", "target_new": "str": "Google", "id": "Q95"
}.
"target_true"" [
"str": "Microsoft", "id": "Q2283"
"subject": "Bing Videos"
"paraphrase_prompss"
"X"Old Jennifer: I'm Sadjectiveold! " Bing Videos is owned by", "J. Bing Videos is from"
"OneDrive is from",
"German Research Center for Artificial Intelligence's owner",
"Groove Music's owner", "Arkane Studios, from", "Yammer is from", "Yammer, by",
"Turn Studios, by", "German Research Center for Artificial Intelligence is owned by", "Mojang Studios is from",
"id Software's s owner"
]
}
Figure 6: example of COUNTERFACT
20


Reviews:
[{"review_id": "T9zVf7BEcp", "reviewer": "Reviewer_MzWn", "summary": "The paper targets knowledge editing (KE) for sparse Mixture‑of‑Experts (MoE) LLMs, arguing that naively adapting dense‑model editors leads to high compute, inter‑expert coupling, and routing distribution shift that destabilizes behavior in downstream layers. The authors  Introduce MoEEdit, which (i) reparameterizes per‑expert updates via a per‑expert null‑space projection constructed from preservation features to keep router inputs invariant (suppressing routing shift), and (ii) solves the resulting block‑structured problem with a randomized block coordinate descent (BCD) solver that updates only relevant experts.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear identification of MoE‑specific failure modes (compute, inter‑expert coupling, routing drift) and a principled block‑structured edit formulation tailored to MoE.\n- Practical, scalable solver: exact per‑expert ridge updates in a randomized BCD loop over active experts; clear normal equations and complexity discussion; convincing synthetic scaling vs. a global solver.\n- Strong results on two MoE LLMs and two standard editing benchmarks.", "weaknesses": "- The central stabilization mechanism is a null‑space projection constructed from preservation features, which the paper itself positions as inspired by AlphaEdit’s null‑space constrained editing for dense models; the novelty is its per‑expert application in MoE and its connection to router invariance. While this is a reasonable extension, the claim to be “the first” routing‑stable framework may be too strong without comprehensive comparison to prior MoE‑specific KE or adaptor‑based lifelong editing.\n- Missing comparison to MoE‑specific editing work (LEMoE) and absence of citation. The manuscript does not cite or compare to “LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models.” This omission weakens the novelty claim. Prior art already identified “inconsistent routing” as a key barrier in lifelong model editing and introduced a dedicated mechanism (KV anchor routing) to enhance routing consistency between training and inference. This directly contradicts MoEEdit’s novelty claim that it is “the first to formally identify routing-induced instability” as central. Both methods pursue the same core objective—maintaining routing stability during/after edits in MoE settings—via targeted constraints on the routing pathway. In addition, both target lifelong/sequential and batch editing regimes at scale with similar benchmarks and reporting conventions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oihFAGG4I", "reviewer": "Reviewer_5jmK", "summary": "This paper introduces MoEEdit the first systematic framework for knowledge editing in Mixture-of-Experts or MoE LLMs. The authors identify a tripartite challenge for MoE editing computational cost expert coupling and most importantly routing distribution shift. They argue that naively editing MoE models perturbs layer outputs which in turn causes downstream routers to select different experts destabilizing the model. MoEEdit solves this by reparameterizing updates using per-expert null-space projections. This constrains the edits to not affect a preservation set which keeps router inputs invariant and prevents routing shift. To make this tractable they solve the block-structured optimization with an efficient randomized block coordinate descent BCD solver. Experiments on large MoE models show MoEEdit achieves state-of-the-art performance while being highly efficient and maintaining routing stability.", "questions": "1. How do you build the preservation set $K_n^0$ for each expert? How many samples do you need for the projection matrix $P_n$ to be effective?\n    \n2. Your routing stability analysis looks at downstream layers. Does the edit also change the expert routing _within_ the edited layer for the tokens that follow the edit position?\n    \n3. The BCD solver needs multiple passes. Can you report the actual wall-clock time in seconds for a single edit batch? I'm trying to understand the real-world speed.\n    \n4. What happens if you just use a single global AlphaEdit-style projection on the active experts? I'm trying to figure out if the per-expert projection is the most critical part of your method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is the first paper to properly formalize and tackle the knowledge editing problem for sparse MoE models.\n    \n2. The identification of routing distribution shift as the key failure mode for MoE editing is a novel and very important insight.\n    \n3. The proposed solution is elegant. The per-expert null-space projection directly targets the routing shift problem.\n    \n4. The BCD solver is a crucial component that makes the method practical. The paper clearly shows why a global one-shot solution is computationally infeasible and provides a scalable alternative.", "weaknesses": "1. The analysis focuses entirely on the routing shift in _subsequent_ layers. It's unclear how the edit affects routing for _subsequent tokens_ within the _same_ edited layer.\n    \n2. The null-space projection is critical but it relies on a preservation set $K_n^0$. The paper never explains how this set is collected how big it is or how sensitive the method is to its quality.\n    \n3. The BCD solver is efficient in terms of scaling with expert count $N$ but it's still an iterative process. Figure 4 shows it needs 6-10 passes. This makes it much slower per edit than one-shot dense editors like ROME. A wall-clock time comparison is needed.\n    \n4. The novelty versus AlphaEdit could be clearer. The paper should test what happens if you just apply a standard global AlphaEdit projection to the K-active experts. This would help isolate the benefit of the per-expert projection design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhRSqj5t0r", "reviewer": "Reviewer_zj25", "summary": "The paper proposes MoEEdit, a method for routing-stable knowledge editing in MoE LLMs. This method reparameterizes expert updates through null-space projections for each expert, suppressing routing shifts. The method also introduces randomized block coordinate descent to efficiently implement expert updates, addressing the computational complexity issues caused by direct solving.", "questions": "- How is the v used in editing obtained? What impacts does v have on the routing shift problem?\n- What are the experimental settings used in the paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper's proposed MoEEdit effectively mitigates the routing shift problem in MoE LLMs editing while requiring only minimal computational cost. Experiments demonstrate that this method achieves better performance than other baseline methods across three metrics: Efficacy, Generalization, and Specificity.", "weaknesses": "- The paper does not explain how the v used in editing is obtained. As the output of the edited layer, v should also be one of the factors contributing to the routing shift problem, but the authors do not discuss this element.\n- The paper lacks detailed descriptions of the experimental setup.\n- The paper has some writing deficiencies, such as when the metric RS first appears in Section 4.1, it does not introduce that the full name of this metric is routing-similarity, which only appears later in Section 5.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v71ELUfdFn", "reviewer": "Reviewer_fg5e", "summary": "This paper proposes MoEEdit, a novel framework for knowledge editing in sparse Mixture-of-Experts (MoE) large language models. To address the challenges of high computational cost, expert coupling, and routing distribution shift when applying existing methods designed for dense models, MoEEdit introduces two key techniques: 1) per-expert null-space projection, which preserves outputs on retained samples and prevents routing shifts; and 2) a randomized block coordinate descent (BCD) solver that decomposes the large-scale optimization into smaller, expert-wise subproblems, improving computational efficiency. This work establishes a strong baseline for knowledge editing in MoE-based models.", "questions": "Question1: see Weakness 1\nQuestion2: see Weakness 2\nQuestion3: Have the authors considered or analyzed how the sampling strategy and size of the preservation set affect the model's editing performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength 1: The identification and analysis of \"routing distribution shift\" is a key contribution of this paper. The authors clearly explain how parameter edits in one MoE layer can trigger cascading changes in downstream routing patterns, thereby disrupting the model's expert specialization pathways, providing a theoretical foundation for future research.\n\nStrength 2: The experiments are relatively comprehensive and demonstrate superior performance, effectively supporting the theoretical analysis.", "weaknesses": "Weakness 1: The actual computational cost of the BCD solver is not clearly presented, especially in comparison with traditional approaches—for example, the total time required to complete 1,000 edits. This lack of comparison could undermine the claim of \"efficiency\" made in the title.\n\nWeakness 2: Experiments are conducted with up to 1,000 sequential edits. While this is a reasonable benchmark, the true stress test for knowledge editing lies in much larger scales (e.g., 3k or more) of continuous edits. It remains unclear whether MoEEdit’s performance and routing stability would degrade after such extensive editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)