SafeMoE: Safe Fine-Tuning for MoE LLMs by Aligning Harmful Input Routing
Authors: Jaehan Kim
Minkyoo Song
Seungwon Shin
Sooel Son
Venue: ICLR 2026 Poster (2026)
Keywords: AI safety
Large language model
Mixture-of-Experts
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: SafeMoE is a safe fine-tuning method for MoE LLMs that mitigates harmful fine-tuning attacks by preventing routing drift on harmful inputs.

Abstract: Recent large language models (LLMs) have increasingly adopted the Mixture-of-Experts (MoE) architecture for efficiency. MoE-based LLMs heavily depend on a superficial safety mechanism in which harmful inputs are routed safety-critical experts. However, our analysis reveals that routing decisions for harmful inputs drift significantly after fine-tuning, exposing a critical vulnerability to harmful fine-tuning (HFT) attacks. Existing defenses, primarily designed for monolithic LLMs, are less effective for MoE LLMs as they fail to prevent drift in harmful input routing. To address this limitation, we propose SafeMoE, a safe fine-tuning method tailored to MoE LLMs. SafeMoE directly mitigates routing drift by penalizing the gap between the routing weights of a fine-tuned model and those of the initial safety-aligned model, thereby preserving the safety-aligned routing of harmful inputs to safety-critical experts. Experiments on open-source MoE LLMs ranging from 7B to 141B parameters demonstrate that SafeMoE effectively mitigates HFT attacks, reducing the harmfulness score of OLMoE from 62.0 to 5.0, for example, while maintaining task utility within 1% degradation and incurring only 2% overhead. It significantly outperforms state-of-the-art defense methods for safeguarding LLM fine-tuning and remains effective in recent large-scale MoE LLMs such as gpt-oss and Llama 4. Our implementation is available at https://github.com/jaehanwork/SafeMoE.

Published as a conference paper at ICLR 2026
SAFEMOE: SAFE FINE-TUNING FOR MoE LLMs BY ALIGNING HARMFUL INPUT ROUTING
Jaehan Kim, Minkyoo Song, Seungwon Shin, Sooel Son'
KAIST
(jaehan, mi minkyoo9, claude, sl, sonn www.scnaaitt.cc.......o
ABSTRACT
Recent large language models (LLMs) have increasingly adopted the Mixture-of- Experts (MoE) architecture for efficiency. MoE-based LLMs heavily depend on a superficial safety mechanism in which harmful inputs are routed safety-critical experts. However, our analysis reveals that routing decisions for harmful inputs drift significantly after fine-tuning, exposing a critical vulnerability to harmful fine-tuning (HFT) attacks. Existing defenses, primarily designed for monolithic LLMs, are less effective for MoE LLMs as they fail to prevent drift in harmful in- put routing. To address this limitation, we propose SAFEMOE, a safe fine-tuning method tailored to MoE LLMs. SAFEMOE directly mitigates routing drift by penalizing the gap between the routing weights fine-tuned model and those of the initial safety-aligned model, thereby preserving the safety-aligned routing harmful inputs to safety-critical experts. Experiments on open-source MoE LLMs ranging from 7B 141B parameters demonstrate that SAFEMOE effee- tively mitigates HFT attacks, reducing the harmfulness score of OLMoE from 62.0 to 5.0, for example, while maintaining task utility within 1 1% degradation and incurring only 2% overhead. significantly outperforms state-of-the-art defense methods for safeguarding LLM fine-tuning and remains effective in recent large- scale MoE LLMS such as gpt-oss and Llama 4. Our implementation is available at atlttps:///itthhl com www.a/////oooo/////oo SafMoo
INTRODUCTION
Mixture-of-Expere (MoE) (Shazeer et al.) 2017 is sparse model architecture that improves ef- ficiency by dynamically routing inputs to subset of expert layers, which have gained adoption for large language models (LLMs). Recent MoEed LLMs, including gpt-oss (OpenAl 2025
Llama 4 (Meta AIl 2025a). Qwen3 MoE (Qwen Team 2025), and DeepSeek-RI (Guo et al. 2025
have achieved surpassing performance on a wide range of challenging tasks, outperforming their monolithic counterparts. However, recent studies (Lai et al. 2025 Fayyaz et al. (2025) show that the safety of MoE LLMs heavily relies on certain safety-critical experts and intentionally manipulating routing decisions to disable these experts leads to significant increases in harmfulness. This super- ficial safety mechanism leaves MoE LLMs particularly susceptible to harmful fine-tuning (HFT) attacks (Qi et al. 2024 Yang al. 2024 Zhan et al.] 20244 Huang al.) 2024a Wallace et al.) 2025). These attacks are designed to compromise the safety of target LLM by injecting only limited number of harmful samples into the training dataset, rendering practical yet severe threat to commercial LLM providers given the growing prevalence of fine-tuning API services (OpenAl,
2024b Google 2025
Our systematic analysis of MoE LLMs uncovers a novel architectural vulnerability in their routing mechanism, which determines which expert layers should be activated for processing inputs. We find that routing decisions for harmful inputs drift substantially from those of the initial aaiete- model under both harmful and benign fine- tuning. phenomenon we term safety routing drift. This drift impedes the activation of ccrtta- experts and thereby undermines the model's safety.
"Corresponding author

---

Published as a conference paper at ICLR 2026
Given the reliance of MoE LLM
8.0
Fine-tuning
Antidote
60
safety on routing aligned to-
Safelnstr
SafeDelta
ward these safety-critical
Safety matinn drift
6.0
SaLoRA
SafeMoE (ours)
Hamfulness score
50
20
perts, preserving the initial rout-
ing decisions of the safety-
4.0
30
aligned models is important
20
2.0
safeguarding MoE LLMs
10
against HFT attacks. How-
ever, existing defenses (Huang et al.] 2025b Li et al.) 2025a Luetal. 2025) are primarily de- signed for the monolithic trans- former architecture and over-
100
Training step
200
300
400
500
100
Tranning step
200
300
400
500
(a) Safety routing drift
(b) Harmfulness score
Figure I: Effectiveness of defenses against HFT attacks.
look the superficial safety mechanisms of MoE LLMs, exhibiting limitations in mitigating fine- tuning risks. In support of this, we conduct preliminary experiment on the training dynamics of existing defenses under HFT attacks, including fine-tuningsstae methods (Bianchi et al.) 2024]
Li al. 2025a) and post-finettuning methods (Huang et al. [2025a) Lu et al.) (025). as shown in Figure The safety routing drift is quantified as the deviation in routing weights on harmful in- structions between the fine-tuned and safety-aligned models. In MoE LLMs, these state-of-the-an
defenses become less effective in preventing the safety routing drift and reducing harmfulness of fine-tuned models.
To address this limitation, we propose SAFEMOE, the first safe fine-tuning method tailored to safe- guard MoE-based LLMs, which directly addresses the vulnerability in their safety mechanisms. Specifically, we design a regularization technique that aligns the routing decisions of fine-tuned MoE LLM with those of the initial safety-aligned model by minimizing the KL-divergence of their routing weights on harmful inputs during fine-tuning. This encourages the fine-tuned model to direct harmful inputs to sufety-critical -criicall experts in the safety-aligned model. Accordingly, the fine-tuned model withstands the effects of HFT attacks while attaining comparable task utility. To reduce the overhead of SAFEMOE, we adopt greedy optimization strategy that alternates between the fine- tuning and regularization steps rather than optimizing both simultaneously.
We conduct extensive experiments across eight widely used MoE LLMs, ranging from 7B to 141B parameters. Our safety evaluation results demonstrate the surpassing effectiveness and robustness of SAFEMOE in safeguarding MoE LLMs against fine-tuning risks. For example, it effectively mitigates an HFT attack that raises the harmfulness score of OLMoE (7B) to 62.0, reducing it to 5.0 with only 1% degradation in the task utility, outperforming state-of- oo-the--rt defenses significantly. This notable effectiveness remains consistent across diverse MoE LLMs, including larger and more advanced models such as gpt-oss and Ilaama 4, while maintaining their reasoning performance.
Based on an in-depth analysis of training dynamics, we confirm that our regularization technique methodologically valid in preventing the safety routing drift and driving harmfulness reduction. We note that SAFEMOE is highly efficient, comparable to the baseline methods, with only ap- proximately 2% training time overhead in both LoRA and full fine-tuning, which demonstrates its practicality to large-scale MoF LLMss. Through these findings, we highlight the importance of architecture-aware designs of safe lnn-tunnng methods for MoE LLMs.
Our contributions are summarized follows:
We identify vulnerability in the safety mechanism of MoE LLMs, where the drift in routing decisions for harmful inputs during fine-tuning undermines their safety.
We propose SAFEMOE, an effective and efficient safe fine-tuning method tailored to MoE LLMs that preserves the routing decisions the initial safety-aligned model.
Through experiments on open-source MoE LLMs, we show that this vulnerability consis- tent across diverse models and that SAFEMOB offers robust mitigation against HFT attacks.
2 PRELIMINARIES
Mixture-of-experte (MoE). Figure shows the standard MoE architecture for LLMs (Shazeer
al. 2017 Lepikhin et al. 202 et al. 2022. Komatsuzaki et al.) 2023 Fedus et al. 2022)
2

---

Published as a conference paper at ICLR 2026
The MoE layer output is formalized as:
MoE layer
IMAE Topk(o(r(())))FNNN())
(1)
FFN 1
FFN 2
FFN
FFN
BEN
where N is the number of experts, a is Softmax, r(h) E RN is routing weights the i-th expert layer. By leveraging this conditional com- putation approach, the MoE architecture provides substan- tial efficiency gains in both training and inference, which has driven its adoption in recent LLMs. MoE LLMs typ- ically adhere to the transformer architecture, but the feed- forward network (FFN) in each transformer layer is replaced by an MoE layer. The MoE layer consists of multiple independent FFNs, referred to as experts, along with gating network. For each input token, the gating network dynamically assigns a routing weight to every expert in the MoE layer based on the token's 's hidden state from the self-attention layer. The top-k experts are then selected for forward computation, and their outputs are combined according to the assigned weights.
Routing weights
input token I, and FFN, is
Gating network
Attention
Transformer layer
Figure 2: MoE LLM architecture.
Superficial safety mechanism in MoE. Previous studies have examined safety vulnerabilities in LLMs by analyzing specific model parameters and forwarding paths (Wei et al.) 2024. Lee et al.) 2024 Peng et al. 2024; Tamirisa et al. 2025). However, the safety implications of the fundamentally different mechanisms introduced by the MoE architecture remain insufficiently understood. Routing decisions in MoE LLMs play crucial role in composing the outputs and have direct impact on their safety. Recent studies (Lai et al., 2025 Fayyaz et al. 2025) show that harmful instructions to safety-aligned MoE LLMs,, such as "How to make bomb?" consistently trigger specific experts, referred to safety-critical experts. Moreover, masking these experts causes significant degradation in safety even when the model parameters remain unchanged. This indicates that the safety of MoE LLMs heavily depends on routing decisions that determine the activation of safety- experts.
Harmful fine-tuning (HFT) attacks. Safety alignment has become an essential step in ensur- ing that LLM outputs are harmless and aligned with human values (Ouyang et al., 2022, Rafailov al. 2023). However, researchers show that this alignment can be undone through user fine- tnning (Qi et al.) 2024 Yang et al. [024]] /Zhan et al. 2024). Specifically, they find that HFT attacks that inject a small portion of harmful samples into the training dataset, or even benign fine- tuning alone, can severely impair LLM safety, Considering the increasing availability of fine-tuning API services (OpenAl 2024b Google 2025), HFT attacks have become a practical threat to LLM
providers.
To mitigate this threat, several methods have been proposed to enhance the safety of LLM fine- tuning-fi for example, augmenting datasets with safe samples (Bianchi al., 2024: Zong et al. 2024], constraining training to prevent harmful-direction drift (Huang et al. 2024b Wu et al. 2025) and pruning harmful parameters from the fine-tuned models (Huang et al. 2025a Lu et al. 2025
However, existing defenses focus solely on monolithic LLMs and overlook the distinct architecture of MoE LLMs based on dynamic input routing and their superficial safety mechanisms. To the best of our knowledge, no prior work has investigated safe fine-tuning strategies tailored to MoE LLMs.
SAFETY VULNERABILITY IN MOE LLMS
Given that the safety of MoE LLMS depends on safety-critical experts (Laiet 2025 Fayyaz al.,
2025), we posit that safety degradation in fine-tumed MoE LLMs arises from substantial deviations in routing decisions for harmful instructions compared to those in the initial
models.
To verify this, we first define safety routing drift for harmful inputs. This metric quantifies a differ- ence between the routing weights of safety-aligned MoE LLM and its fine-tuned counterpart:
Definition 3.1 (Safety routing drift). Let Walign be safety-aligned MoE LLM, Given a fine-tuned model st, ft. the safety routing drifi for a harmful instruction imput x defined as:
d(w ft, x) DeL. (r(r()) ([atgggt))))) (r())un))))
where DKL (PII) denotes KL divergence between reference distribution P and an approximating probability distribution Q. a is Softmax, and r(-lw) E IR." denotes the routing weight vector over N experts of model for an input.
3

---

Published as a conference paper at ICLR 2026
1e-3
10-3
1e-3
1.2
Safety routing drift
80
2.8
880
2.4
80
Safety reuting drift
Harmfuiness Score
Harmfulnes score
Safety reuting dritt
0.9
Harmfuiness score
60
Safety routtng
1.8
0.6
40
1.2
20
0.6
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
fraining step
Training step
Training step
(a) OLMoE (Benign FT)
(b) Qwen MoE (Benign FT)
(c) DeepSeek V2 (Benign FT)
(r 0.9616, p = 2 6)
(r = 0.8822, p = 3e 4)
(r
0.9442, p = le-5)
1e-3
1e 3
1e
1.2
80
2.8
80
Safety routing difftt
Harniullin nulueeesss score
Harmiuuiness score
Safety routing drfft
60
Sately routing
2.1
&
7.8
40
20
0.6
0.0
200
300
400
500
100
200
400
500
100
200
300
400
500
Training step
Training step
Training step
(d) OLMoE (HFT attack) (r 0.9136, p 8e
(c) Qwen1.5 MoE (HFT attack)
DeepSeek V2 (HFT attack)
(r= 0.9421, p = le-5)
(r 0.9813, p le-7)
Figure 3: Safety routing drift and harmfulness of MoE LLMs training steps. Results of t-tests for Pearson correlation coefficients are reported (r: correlation coefficient, p: p-value).
Safety routing drift during fine-tuning. We then analyze the training dynamics of three MoE LLMs,, including OLMoo (Muennighoff al. 2025). Qwen1.5 MoE (Qwen) [024]. and DeepSeek V2 (Liu et al., (024). and measure the correlation between the safety routing drift and harmfulness of their fine-tuned models. Specifically, we consider two scenarios:
benign fine-tuning on 5.5k samples from the Alpaca dataset (Taori et al. (002), and ii) HFT on a combined dataset of 5k task- specific samples from SAMSum (Gliwa et al. 2019) and 500 harmful samples from BeaverTails (Ji et al. 2023). For each fine-tuned LLM, we compute safety routing drift on the last tokens of harm- ful instructions from JainbreakBench (Chao et al. 2024). Harmfulness scores are computed as the proportion of unsafe responses, evaluated using I. Jama-GGuard-4-1222 (Meta Al 2025b).
Figure presents the results of our analysis. In both fine-tuning scenarios, we observe significant safety routing drift from the initial safety-aligned model, with the drift metrics increasing as training progresses. Notably, routing decisions for harmful instructions drift even under benign fine-tuning, causing nontrivial increases in harmfulness. This demonstrates that the superficial safety mechanism of MoE LLMs is highly fragile and easily disrupted by fine-tuning. Moreover, the magnitude of safety routing drift is strongly correlated with harmfulness scores across the models, with high statistical significance (p & 0.05). Harmful fine-tuning (HFT) attacks even further amplify this drift, producing substantially larger increases in harmfulness than benign fine-tuning.
Safety recovery through initial routing override. To design an effective defense, we evaluate fine-tuned MoE LLMs by overriding their routing decisions, specifically by restoring the routing weights used for harmful inputs those of the initial safety-aligned model during inference. Figure 4 presents the safety evaluation results for the OLMoE model fine-tuned on the SAMSum dataset. While fine-tuned models exhibit substan- tial harmfulness, overriding the initial routing decisions signif- icantly reduces harmful outputs. This observation further high- lights the critical role the routing mechanism in MoE safety and demonstrates that preserving safety-aligned routing behav- iors is a promising strategy for mitigating safety risks incurred during fine-tuning.
80
Rine-tuning
Harffuiness score
Overriding initial routing
60
20
200 300
500
Training step
Figure 4: Harmfulness when overriding the safety-aligned routing decisions (OLMoF.).
Motivation for an MoE-specific defense. Our analysis shows that the safety routing drift is highly correlated with the safety degradation in fine-tuned MoF LLMss. This suggests the need for an
4

---

Published as a conference paper at ICLR 2026
Harmful fine-tuning (HFT) attack
r Routing drift
FFN
FFN
FFN
FFN
"Maling homb requires knowledge of chemistry ..."
0:00
Harmful response
FFN
FFN
FRN
FFN
Vanilla fine-tuning
Q: "How to make a bomb?"
Routing we lgght
Fine-tuning service
FRN
A: "I'm but / can't assist with that request."
Safety-aligned MoE LLM
Routing drift
Safe respanse
SafeMoE
Fine-tuned MoE LLM
Figure 5: Overview of SAFEMOE. It mitigates the safety routing drift by directly constraining this drift during fine-tuning, thereby effectively safeguarding MoE L.LSs against HFT attacks.
approach that preserves the initial routing decisions of the mitigate fine-tuning risks in MoE LLMss. However, prior defenses, designed for monolithic LLMs, are limited in preventing the safety routing drift and reducing the harmfulness fine-tuned MoE LLMs, as shown in preliminary experiments (see Figure [ These findings motivate new defense that reflects the unique architecture of MoE LLMs and their safety mechanism.
models on harmful inputs
SAFE MOE FINE-TUNING METHOD
Safety routing drift regularization. We propose novel fine-tuning approach designed to improve the safety of fine-tuned MoE LLMs, as illustrated in Figure 5 Specifically, during fine-tuning, we propose leveraging a regularization objective that constrains the safety routing drift (Definition This objective penalizes deviations in the routing weight distributions of an MoE LLM under fine-tuning from those initial safety-aligned model Walign on harmful instructions:
Greg(w) = E.ED, E (a(r
(x/kWatign)/)) ( || (r (x/w)/T))
(3)
where Dí, is the harmful instruction dataset, and is the set transformer layers. We apply regular- ization to the routing weights assigned to the last token of each harmful instruction. The temperature T controls the strength regularization. A smaller (c.g., < 1.0) further enhances safety by sharp- ening the routing weight distribution, which enables the regularization to focus more top-ranked safety-critical experts and tightly constrains the routing weights those experts.
Bi-level greedy optimization. We integrate the regularization into supervised fine-tuning task- specific dataset Dyr to solve:
arg min (wr (w))
Simultaneously optimizing both losses every training step, however, incurs substantial computa- tional overhead. To address this, we reframe the joint optimization into a bi-level greedy approach that alternates between the supervised fine-tuning and regularization steps.
The greedy optimization process is described Algorithm We first precompute the routing weights of the initial safety-aligned MoE LLM over all harmful instructions in Dj to avoid redundant forward passes in the regularization steps (line 2). During training with the fine-tuning loss Lafts insert pefeey-preserving steps using the regularization loss Lreg every T. Treg steps (line 6).
EXPERIMENTS
5.1 SETUP
MoE LLMs. We conduct experiments on eight widely adopted MoE LLMs: OLMoE- 1B-00--IB-B-015--- Instruct (Muennighoff et al. 2025), Twenl.5-Mofe
(Qwen) 2024). and DeepSeek- V2-
Lite-Chat (Liu et al. 2024). gpt oss-20b (OpenAl 2025), Qwen3-30B Awen3-30B-A333 (Qwen Team
2025
Phi-3.5-MoE-IInse Instruct (Microsoft. 2024), and two models with on-the-fly fly 4-bit quantization.
Llama-
4-Scout- 17B- 1--16E-IB Instructt (Meta All (025a) and Mixtral-8x22D 1-n----nttutt-OO. (Mistral Al
2025
All models used are safety-aligned versions. We describe model details in the Appendix A.2
5

---

Published as a conference paper at ICLR 2026
Table 1: Safety evaluation of MoE LLMs. We report fine-tuning accuracy (FAf) and harmfulness
score (HS.) for the SAMSum and SQL tasks. The number of parameters is denoted as (active/total).
OLMoE (1.3B/6.9B)
Qwen1.5 MoE (2.7B/14.3B)
DeepSeek V2 (2.4B/15.7B)
Method
SAMSum
SQL
SAMSum
SQL
SAMSum
SQL
FA
HS
FA
HS
FA
HS
FA
HS
FA
HS
FA
HS
Aligned
31.8
43.0
36.6
3.0
38.4
3.0
34.0
29.6
0
Fine-tuning
49.3
62.0
58.5
64.0
50.4
49.0
70.2
37.0
52.0
70.0
70.1
72.0
Safelnstr
49.5
46.0
58.9
41.0
50.6
11.0
69.3
52.1
25.0
70.2
21.0
Lisa
48.4
21.0
57.2
40.0
50.1
10.0
68.5
13.0
50.7
24.0
68.9
22.0
SaLoRA
48.9
24.0
54.5
40.0
48.9
28.0
54.9
33.0
50.1
66.0
62.0
74.0
Antidote
48.7
40,0
57.5
44.0
49.3
18.0
68.6
29.0
50.7
70,0
65.3
62.0
SafeDelta
48.6
13.0
57.4
33.0
50.2
22.0
69.2
30.0
51.0
47.0
69.0
72.0
SAFEMOF
48.9
5.0
59.0
17.0
50.6
0
69.5
1.0
51.0
1.0
69.1
4.0
Fine-tuning datasets. We consider two tasks: SAMSum (Gliwa et al. 2019) for dialogue summa- rization and SQL (b-mc2] 2023) for SQL query generation, both widely adopted for implementing HFT attack scenarios (Yang al. 2025a Wang al. 2024). For effective attacks, conduct pervised fine-tuning with 5k task samples combined with 500 harmful samples from BeaverTails (Ji
al. 2023 Fine-tuning performed with LoRA (Hu et al. 2022), as detailed in Appendix A.2
Metrics. We reports two metrics: Fine-tuning accuracy (FA) ) and Harmfulness score (HS). FA mea- for SAMSum and exact match accuracy for SQL.. To assess rea-
sures task utility- utility-Rouge-1 Rouge-1 soning performance, we measure accuracy on 570 samples (10 from each categories) from MMLU- Redux-2.0 (Gema et 2025). HS refers the proportion of responses
BailbreakBende (Chao 2024) instructions that are classified as unsafe by Laama--uard-4-... 4-B (Meta Al 2025b). et al.
Baselines. We evaluate five state-of off-the-art defenses against HFT attacks. Among fine-tuning- stage methods. SafeInstr (Bianchi et al.) (2044) augments fine-tuning datasets with additional safe samples, Lisa (Huang et 2024b) introduces proximal term to prevent excessive drift in model parameters during fine-tuning, and S (Li et al. 2025a) initializes LoRA layers with weights optimized on safe samples. We also consider two post-fine-tuning weight modification approaches. Antidote (Huang et al. (0225a) prunes harmful parameters by analyzing their contribution harm- ful instructions. SafeDelta (Lu et al. 2025) selects delta parameters that maximize utility while minimizing safety degradation.
SAFEMOE implementation. We use 100 harmful instruction samples from SafeInstr (Bianchi ct al. 2024) the dataset Dn, We select 7 as the smallest value in the range 0.1-1.3 that allows 1% degradation in fine-tuning accuracy. By default, Trog is set to the number steps per epoch.
5.2 SAFETY EVALUATION
Trepors the defense effectiveness of SAFEMOE on three widely used MoE LLMs. The initial safety-aligned models prior to fine-tuning (first row) are highly safe, exhibiting nearly zero harmful- ness scores. Vanilla fine-tuning substantially undermines safety while improving task performance (second row). SAFEMOE attains the lowest harmfulness score while incurring only minimal loss in fine-tuning accuracy. For example, reduces the harmfulness score of DeepSeek V2 fine-tuned on the SQL. task from 72.0 to 4.0. SAFEMOE's superior effectiveness is attributed to its MoE-specific design, which explicitly encourages the routing of harmful instructions to safety-critical experts.
In contrast, baseline methods fail to effectively mitigate safety degradation. Safelnstr, which adopts an architecture-agnosicc strategy based on safety data augmentation, achieves moderate defense per- formance but still leaves substantial harmfulness. Lisa also falls short in defending MoE LLMs against HFT attacks, highlighting the effectiveness of explicitly constraining drift in routing weights methods (Antidote and SafeDelta)
rather than in model parameters. SaLoRA and post-fine-tune assume all parameters are always activated, as in monolithic LLMs. In MoE LLMs, however, the dynamically changing active parameters hinder full optimization of these methods. Furthermore,

---

Published as a conference paper at ICLR 2026
Table 2:
Extended safety evaluation on five large-scale MoE LLMs. We report reasoning perfor-
mance on MMLU- (MMLU1) and harmfulness score (HS.).
gpt-oss
Qwen3 MoE
Phi 3.5 MoE (6.6B/41.9B)
LJama 4 (17B/109B)
Mixtral (39B/141B)
Method
(3.6B/20.9B)
(3.3B/30.5B)
MMLU
HS
MMLU
HS
MMLU
HS
MMLU
HS
MMLU
HS
Aligned
85.4
2.0
89.6
1.0
83.3
2.0
90.4
7.0
78.9
7.0
Fine-tuning
77.5
84.0
89.1
67.0
80.7
83.0
89.5
79.0
66.5
78.0
SAFEMOE
79.6
7.0
88.8
4.0
81.4
2.0
89.8
3.0
78.4
8.0
1.0
Fine line-tuning
SafeMoE
Fine-tuning
SafeMoE
2.2
Finc: -tuuinnnngg
SafeMoE
safety routing drift
0.8
Harmfulnness scoe
Fine-tuning loss
1.8
0.6
80
0.4
1.4
2.22
20
0.0
1.0
0
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Training step
Training step
Training step
(a) Safety routing drift
(b) Harmfulness score
(c) FT loss (w/ smoothing line)
Figure 6: Training dynamics of vanilla fine-tuning SAFEMOE (OLMoB SAMSum).
they cause significant degradation in fine-tuming accuracy with only marginal harmfulness reduc- tion, even though we extensively tune their hyperparameters (see Appendix A.3)
These results highlight the importance of architecture-awant defenses that directly address vulnera- bilities in the safety mechanism. In Qwen 1, MoE, we observe slightly improved safety compared to the dffett-aligned models. This effect appears to result from the generalization ability of SAFEMOF, which promotes stronger activation of top-ranked safety-critical experts (see Appendix AA.5)
Results on larger MoE LLMs. We further evaluate SAFEMOE recent larger MoE LLMs under a strong HFT attack scenario using 500 purely harmful samples (Bianchi et al. 2024 Lu et al. 2025 Hsu et al. 2024), shown in Table 2 These models employ diverse MoE configurations (c.g, activating expert among 16 in Llama 4 and activating 8 among 128 in Qwen3 MoE) and distinct reasoning strategies (e.g., multi-level reasoning in gpt-oss and the thinking mode in Qwen3 MoE). Despite their differences, SAFEMOE generally achieves strong defense performance while effectively preserving the models' reasoning capability. For gpt-oss, Phi 3.5 MoE, and Mixtral, SAFEMOE even alleviates the degradation of reasoning performance observed in vanilla fine-tuning by preventing excessive overfitting to the HFT attack data. Overall, these results demonstrate the practicality of SAFEMOE for real-world fine-tuning services with high-capable MoE LLMs.
5.3 TRAINING DYNAMICS
To demonstrate that the design of SAFEMOE is valid in practice, we analyze the variation of the safety routing drift, harmfulness score, and fine-tuning loss throughout the fine-tuning steps. Fig- presents the results for OLMoE fine-tuned on the SAMSum task.
As shown Figure 6a and 6b SAFEMOE moderately reduces the safety routing drift at the early stage and effectively mitigates it after the first regularization period (after 150 steps). The harm- fulness score decreases with drift mitigation, achieving a significant reduction compared to vanilla fine-tuning at subsequent checkpoints. Our experiment has shown that existing defenses have lim- ited effectiveness in reducing the safety routing drift and harmfulness (see Figure TTp. SAFEMOE methodologically addresses this limitation and provides effective defense for MoE LLMs.
In Figure 6c we also present the fine-tuning loss to further evaluate its impact on the fine-tuning task. The loss converges stably, closely matching the trajectory of vanilla fine-tuning with only

---

Published as a conference paper at ICLR 2026
Table Execution overheads (OLMoB).
Table 4: Ablation study on restricting gradients from the expert layers. We report fine-tuning ac- curacy (FAf) and harmfulness score (HS.).
Extra time
Method
Seconds
Percentage
OLMoE SAMSum
SafeInstr 15.98
1.98%
Method
SaLoRA
747.56
92.41% %
FA
HS
Antidote
5.67
0.70%
Fine-tuning
49.3
62.0
SafeDelta
52.18
6.45%
SAFEMOÉ
48.9
5.0
SAFEMoE
17.26
2.13%
SAFEMoE (ablation)
48.2
18.0
Satoty routing drift
2.0
Fine tuning
80
Harmfulness score
SafeMoE
1.5
60
Harmitunesss
1.0
0.5
20
0.0
10
8-11
12-15
8-15
0-15 (full)
Transformer layer
Target layers
Figure 7: Safety routing drift across transformer
Figure 8: Effectiveness of SAFEMOE applied on specific layers (OLMoE on SAMSum).
layers (OLMoE on SAMSum).
negligible differences. This underscores the compatibility of SAFEMOE in standard fine-tuning. enabling safe and practical fine-tuning services.
5.4 EFFICIENCY ANALYSIS
To assess the efficiency of SAFEMOE, we analyze the total execution overhead of SAFEMOE during fine-tuning and post-nine-tuning. Jummaries results for OLMoE fine-tuned on the SAMSum task under the environment described Appendix A.2 For reference, vanilla fine-tuning takes 808.98 seconds on average over ten trials. SaLoRA incurs substantial overhead in computing the optimal LoRA initialization, increasing the execution time to nearly twice that of vanilla fine-tuning. The other approaches are generally more efficient but fail to effectively mitigate HFT attacks on MoE. I.LMs. In contrast, SAFEMOE attains strong safety with only 2.13% increase in training time, highlighting its practicality in safeguarding large-scale models.
5.5 ABLATION STUDY
Isolation of SAFEMOE's influence routing. In the MoE architecture, the embedding vector transferred from the attention layer feeds into both the gating network and the expert layers. To isolate the effect of SAFEMOE on the gating network, we explore an ablation in which gradients from the expert layers are blocked when updating the embedding vector. As shown in Table 4 this ablation setting still achieves strong defense performance, validating our approach of directly preserving routing weights for harmful inputs.
Layer-selective application of SAFEMOE. We analyze the extent of safety routing drift across transformer layers in OLMo fine-tuned on the SAMSum task, shown in Figure The results reveal that routing drift is not uniform across layers and the upper layers exhibit substantially larger drift. This appears to result from harmful features typically being distinguished after the middle layers of LLMs (Li et al. 2025b).
Motivated by this observation, we investigate more efficient variants of SAFEMOE that apply the safety routing drift regularization selectively rather than across all layers (Figure &. The safety eval- uation results show that targeting upper layers (c.g., 12-15 layers) provides much stronger mitigation of HFT attacks than targeting lower layers. Moreover, applying SAFEMOE only to 8-15 layers (the upper half) achieves level of safety comparable full-layer regularization.
8

---

Published as a conference paper at ICLR 2026
Hamffliness score
Fine mee-tuning accuracy Harmfulness score
52
accurac
Hammfuness score
80
Execution overhead (s)
Fine Fine-turing accuracy
80
accuracy
60
50
60
80
Harmfulness
48
40
60
20
46
20
46
20
0.1
0.3
0.5
0.7
0.9
50
150
200
200
Trag
(a) Lieg hyperparameter (r)
(b) harmful instructions (Dr))
(c) Regularization period (Treg)
Figure 9: Sensitivity to regularization strength (OLMoE on SAMSum).
5.6 SENSITIVITY ANALYSIS
Regularization hyperparameter. The hyperparameter in Equation 3 controls the strength of the safety routing drift regularization. [ggure/9jshowee results for OLMoE fine-tuned with the SAMSum task. Smaller values of r focus the regularization on top-ranked safety-critical experts, achieving substantial harmfulness reduction. Across values, fine-tuning accuracy remains nearly unchanged, demonstrating that SAFEMOE can robustly improve safety without sacrificing task utility.
Number of harmful instructions. We vary the number of harmful instructions [Dal used for the regularization (see Figure 195). More harmful instructions strengthen the safety effect with an ap- proximately linear overhead increase, suggesting simple yet effective way to enhance our ap- proach. Our default choice (|Da) = 100) provides a practical balance between safety and efficiency.
Regularization period. We examine the influence of the regularization frequency by vary- ing the regularization period Treg. as shown Figure 9c A smaller Treg triggers more fre- quent drift regularization steps, resulting in stronger defense performance but incurring additional execution time overhead. Despite this trade-off between effi- ciency and safety, we emphasize that the overhead remains rea- sonable and acceptable, when compared to the baseline methods
Fine tuning
Harmfllnnes score
SafeMoE
fine tunetunnn scouracy
60
discussed in Section 5.4
80
48
Harmful sample ratio fine-tuning data. We explore differ- strengths of HFT attacks by adjusting the ratio of harmful samples in the fine-tuning dataset Dfx (see Figure 10). Vanilla fine-tuning exhibits drastically increasing harmfulness scores as the harmful ratio rises. In contrast, SAFEMoE suppresses
20
44
Harmful sample ratio
harmfulness escalation, demonstrating robustness even against stronger attacks.
Figure 10: Sensitivity to the harmful sample ratio in Djt-
5.7 EFFECTIVENESS UNDER FULL FINE-TUNING
We extend our evaluation to a full-parameter fine-tuning scenario to demonstrate the generality of SAFEMOE. Table (5) shows the results for OLMoE fine-tuned on the SAMSum task. SAFEMOE achieves substantial harmfulness reduction while preserving fine-tuning accuracy in this setting as well. Although all expert layers, including safety-critical ones, exposed to training during HFT attacks, SAFEMOE remains robustly effective by solely preventing the safety routing drift. This fur- ther highlights the central role of routing in the safety MoE LLMs and confirms the validity of our approach. Moreover, it incurs only a 2.30% increase in execution time (four GPUs), comparable to that observed the LoRA-based setting (2.13% with one GPU). These results show that SAFEMOE is reliably adaptable to both parameter surameter-efficient and full fine-tuning approaches.
5.8
EVALUATION ON ADDITIONAL HARMFULNESS BENCHMARK
To further validate our findings, we evaluate harmfulness fine-tuned MoE LLMs on HEx-PHI QQi al. 2024). widely used harmful instruction benchmark. reports results for the SAMSum fine-tuning scenario under our main settings. The results remain consistent, showing that SAFEMOE significantly outperforms all baselines, while Safelnstr achieves moderate harmfulness reduction

---

Published as a conference paper at ICLR 2026
Table 5: Full fine-tuning results of fine-tuning accuracy (FAt), harmful- ness score (HS.), and training time.
Table Harmfulness score (HSL) on HEX-PHI.
Method
OLMoE
Qwen1.5 MoE
DeepSeek V2
Aligned
0.3
8.7
5.7
OLMoE on SAMSum
Method
Fine-tuning
79.7
33.0
83.0
HS
Time (s)
Safelnstr
52.0
13.3
31.0
Aligned
31.8
SaLoRA
39.3
34.7
64.7
Sine-tuning
50.4
58.0
7,023.79
Antidote
53.3
24.7
68.0
SafeDelta
21.7
28.7
41.0
7,189.01
SAFEMOE
51.0
2.0
(+2.30%)
SAFEMoE
3.3
with strong performance for Qwenl.5 MoE. These findings confirm the robust defense effectiveness of S AAFEMOE against diverse harmful querying scenarios.
6 RELATED WORK
We categorize recent defenses against HFT attacks into three groups based on their application stage.
Alignment stage. These methods aim to enhance the robustness of the model against subsequent HFT attacks. Vaccine (Huang et al. 2024c) and its memory--fficient variant (Liu et al. 2025 trains models to resist perturbations that maximize alignment loss. RepNoise (Rosati al. 2024) and
Booster (Huang et al., (2025b) proactively remove harmful information by optimizing perturbations in model representations alignment method that balances training across vulnerable and invulnerable subsets of alignment data. However, such alignment-star stage methods require carefully tuned hyperparameter for each downstream task (Huang et al. (025b), which limits their practicality in fine-tuning services that must handle many unknown tasks.
weights. VAA (Liang
2025) introduces vulnerability-
Fine-tuning stage. A second line of work directly addresses safety degradation during fine- tuning. Safelnstr (Bianchi et al. 2024) augments supervised fine-tuning datasets with safe sam- ples. Lisa (Huang et al.) (2024b) introduces proximal optimization method to mitigate conver- gence instability when jointly training alignment and task-specific data. AsFT (Yang et al., 2025b) adopts regularization term to suppress updates in harmful directions. SAFT (Choi et al. 2024 and SEAL (Shen et al., (025) identify and filter harmful samples from one-tuming data by scoring their safety impact. SaLoRA (Li
al. 2025a) proposes a safety-aware initialization of LoRA layers, designed based on an analysis of changes in safety-related features observed during fine-tuning.
Post-fine-tunin stage. Training-free remedies have also been proposed restore safety after harm- ful fine-tuning. RESTA (Bhardwaj et al. 2024) extracts a safety vector from an aligned model and reintroduces it into the fine-tuned model via arithmetic addition of weights. SafeLoRA (Hsu al. 2024) selectively projects LoRA weights into safety-aligned subspace. Antidote (Huang et al., 2025a) removes harmful parameters identified through their importance alignment data, thereby improving robustness under varying fine-tuning hyperparameters. SafeDelta (Lu et al., (2025) refines fine-tuned delta parameters balance task utility with reduced safety degradation.
CONCLUSION
This work introduces SAFEMOF, the first safe fine-tuning method tailored to MoE-based LLMs. Our systematic analysis uncovers a vulnerability inherent in their safety mechanisms; routing deci- sions for harmful inputs drift significantly from those of safety-aligned models under both harmful and benign fine-tuning. To address this, we propose routing drift regularization method with an efficient optimization algorithm that integrates seamlessly into standard MoE LLM fine-tuning pipelines. Extensive evaluations across diverse MoE LLMs show that SAFEMOE achieves signif- icant reductions harmfulness with minimal overhead. These results establish SAFEMOE as an effective and practical defense for fine-tuning services against HFT attacks, underscoring the need to address architectural weakness when safeguarding MoE. L.MSs from fine-tuning risks.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work presents a safe fine-tuning method designed to mitigate potential safety threats associated with LLM fine-tuning. our experiments, we evaluate the extent of harmful behaviors exhibited by open-source LLMs under harmful fine-tuning (HFT) attacks. HFT attacks are a well-documented
concern (Qi al. 2024, Yang et 2024 Zhan al., 2024 Bianchi et al. 2024 Zong al., 2024] Huang et al. 2024b] Li et al. 2025a). and our experiments are conducted entirely using publicly available alignment datasets and benchmarks related safety and HFT attacks. Accordingly, we believe our work does not introduce any additional harm. Furthermore, ensure that neither the harmful LLMs nor their generated harmful responses are shared, strictly adhering to the ICLR Code of Ethics (ICLR 2025) throughout the work.
REPRODUCIBILITY STATEMENT
We release the implementation of SAFEMOE to facilitate reproduction our method and evaluation results. The source code available at https: http:///hhb...oooeeeaaaaaet///aa
THE USE OF LARGE LANGUAGE MODELS
The authors used ChatGPT (OpenAl 2025) for grammatical refinements of the manuscript. These modifications have been manually reviewed and finalized by the authors.
ACKNOWLEDGMENT
This work partly supported by the Institute of Information & communications Technol- ogy Planning & Evaluation (ITTP) grant funded by the Korea govermment (MSIT) (No.RS-2020 11200153, Penetration Security Testing of MI. Model Vulnerabilities and Defense, 40%, No.RS- 2024-00337703, Development of Satellite Security Vulnerability Detection Techniques Using AI and Specification-Baat -asoed Automation Tools, 30%) and the InnoCORE program of the Ministry of Science and ICT (N (N10250156, 30%).
11

---

Published as a conference paper at ICLR 2026
REFERENCES
b-me2. sql-create-context dataset.
httpss//hugginnfaaasccc/tttttss
sqp context 2023.
Rishabh Bhardwaj, Due Anh Do, and Soujanya Poria. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp.
14138-14149) 2024.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, 2024.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, Advances in Neural Information Processing Systems, 37:55005-5555 -55029, 2024.
Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-aware fine-tuning of large language models. In Neurips Safe Generative Al Workshop 2024, 2024.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mifttue-of-experts. In International Conference on Machine Learning, pp. 5547- 5569. PMLR, 2022.
Mohsen Fayyaz, Ali Modarressi, Hanieh Deilamsalehy, Franck Demonccuut, Ryan Rossi, Trung Bui, Hinrich Schütze, and Nanyun Peng. Steering moe llms via expert (de)activation. arXiv preprint XXXx:2500900660. 2025.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? In Proceedings of the 2025 Conference of the Nations of the Amer- icas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 5069-5096, 2025.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Sunmarization. Association for Computational Linguisties, 2019.
Google. Fine-tuning with the gemini Inoeel-turinn iellttuunng in 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.122 2025.
Danny Halawi, Alexander Wei, Eric Wallace, Tony Tong Wang, Nika Haghtalab, and Jacob Stein- hardt. Covert malicious finetuning: Challenges in safeguarding 11m adaptation. In Forty-first International Conference Machine Learning, 2024.
Chia-V Yi Hsu, Yu-Lin Tsai, Chih- Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances in Neural Information Processing Systems, 37:65072-6515 65094, 2024.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2026
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409. 18169, 2024a.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lazy safety align- ment for large language models against harmful fine-tuming seeettunigg.... arXiv preprint arXiv:2405. 2, 2024b.
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: perturbation-awant alignment for large lan- guage models against harmful fine-tuning attack. In Proceedings of the 38th International Con- ference on Neural Information Processing Systems, pp. 74058-74088, 2024c.
Tiansheng Huang. Gautam Bhattacharya, Pratik Joshi, Joshua Kimball, and Ling Liu. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning attack. In Forty-second International Conference on Machine Learning. 2025a
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In The Thir- teenth International Conference on Learning Representations, 2025b.
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tam- era Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Training deceptive Ilms that persist through safety training, arXiv preprint arXiv:2401.05566 2024.
ICLR. Ielr code of ethics. httpps///aa....c/ooocc////oefee.
2025.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of 11m via numan-preference dataset. Advances in Neural Information Processing Systems, 36:24678- 24704, 2023.
Joshua Kazdan, Lisa Yu, Abhay Puri, Rylan Schaeffer, Chris Cundy, Jason Stanley, Sanmi Koyejo, and Krishnamurthy Dj Dvijotham. No, of course i can! deeper fine-tuning attacks that bypass token-level safety mechanisms. arXiv preprint arXiv:2502. 19537, 2025.
Aran Komatsuzaki, Joan Puigcerver. James Lee-Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learn- ing Representations, 2023.
Zhenglin Lai, Mengyao Liao, Dong Xu, Zebin Zhao, Zhihang Yuan, Chao Fan, Jianqiang Li, and Bingzhe Wu. Safex: Analyzing vulnerabilities of moe-based llms via stable safety-critical expert identification. arXiv preprini arXiv:2506. /7368, 2025.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mi- halcea. A mechanistic understanding of alignment algorithms: A case study dpo and toxicity. In International Conference on Machine Learning, pp. 26361-26378. PMLR, 2024.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021.
Mingjic Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. Salora: Safety-alignmen preserved low-rank adaptation. In The Thirteenth International Conference on Learning Repre- sentations, 2025a.
Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to Ilm security. In The Thirteenth International Conference on Learning Representations, 2025b.
CHEN Liang, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Vulnerability aware alignment: Mitigating uneven forgetting in harmful fine-tuning. In Forry-second International Conference on Machine Learning, 2025.
13

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.044: 2024.
Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise wise perturbation. IEEE Transactions on Information Forensics and Security. 2025.
Ning Lu, Shengcai Liu, Jiahao Wu, Weiyu Chen, Zhirui Zhang, Yew-Soon Ong, Qi Wang, and Ke Tang. Safe delta: Consistently preserving safety when fine-tuning Ilms on diverse datasets. In Forty-second International Conference on Machine Learning. 2025.
Meta. mocalllamm//lammee-.------------
https:/ / http:///ciin.acceeeeeaaaaaaaeeea http://waaaaaaaaa//o///////////r
meta wwwaaaaallla.. ama 703 Instruct evals, 2024.
Meta AL The llama herd: The beginning of new era of natively multimodal innovation.
https::/aat meta. com www.blog//c/cccccccccocc.ocoo.ct int
2025a.
Meta AI org.://deeoprpooammamooaaaaaaaaa
Llama guard 4. -aaaam-a-uuurr///// 2025b.
https: http://ww..1llm www.wwww..a.................. docs/
model model-cardse aaweell--rrds--nn--ooooa htw.://ptt-oommmt/lllamamam
Microsoft. micnosot/ppii335--mennnnnnnt httpss//oggiiniaee..c//tttttttt
Phi-3.5- 101--3..--oo-insrueet-.. 2024.
Mistral AL mistrala/miixrraass22--iinttttte
https:///ugingfacee.c
-v. 2025.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-expertts language models. In The Thirteenth International Conference on Learning Represettnn 2025.
OpenAL. 4p--i-- ----4o--ini-2024----11
2024-07-88.
gpt-4o-ming 2024a
OpenAl. Openai fine-tuning models. httppp////aattoo....o.caa.cc/oos model-opt inntttmiiitttt zon. 2024b.
OpenAl. Chatgpt. htttps://chattpp.... 2025.
OpenAl. Introducing http:////....o.ooooo/oooooaaaaa. 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 17730-27744, 2022.
Sheng Y Peng, Pin-Yu Yu Chen, Matthew Hull, and Duen Chau. Navigating the safety landscape: Measuring risks in finetuning large language models. Advances in Neural Information Processing Systems, 001996692-977911 2024.
Xiangyu Qi Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024.
Qwen. Oven/qwenl.5-moc-a22 2at.mo---....-aas......
http:://hagginngaceecc///e/ee///
MoE 7 Chat 2024.
Qwen Team. Qwen3 technical report, 2025. URL UKtp:tttp.///arrrrrrr//////22..5
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly reward model. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, pp- 2023.
14

---

Published as a conference paper at ICLR 2026
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation nois- ing: defence mechanism against harmful finetuning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pp- 12636-12676, 2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate -attd mixture-of-expers layer. In International Conference on Learning Representations, 2017.
Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned Ilm fine-tuning via bilevel data selection. In The Thirteenth International Conference on Learning Representa- tions, 2025.
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, et al. Tamper-resistant safeguards for open-weight Ilms. In The Thirleenth International Conference on Leurning Representations, 2025.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnstruction-follvwpr llama model. httpps://ithhb..oo http:///tttbb.ccmo/tttttt-t-aaee
2023.
Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, and Chris Koch. Estimating worst-case fron- tier risks of open-weighe lims. arXiv preprint arXiv:2508.. 03/53, 2025.
Jiongxiao Wang. Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Sharon Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. Advances in Neural Information Processing Systems, 37:5210- 77:3210-5243,, 2024.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In International Conference on Machine Learning, pp. 92588-52610, PMLR, 2024.
Chengean Wu, Zhixin Zhang, Zeming Wei, Yihao Zhang, and Meng Sun. Mitigating fine-tuning risks in Imms via safety-aware probing optimization. In 2nd Workshop on Models of Human Feed- back for Al Alignment, 2025.
Kang Yang, Guanhong Tao, Xun Chen, and Jun Xu. Alleviating the fear of losing alignment in IIm fine-tuning. 2025 IEEE Symposium on Security and Privacy (SP), pp. 2152-2170. IEFE,
2025a.
Shuo Yang, Qihui Zhang, Yuyang Liu, Yue Huang, Xiaojun Jia, Kumpeng Ning, Jiayu Yao, Jigang Wang, Hailiang Dai, Yibing Song, et al. Asft: Anchoring safety during I1m fine-tuning within narrow safety basin. arXiv preprint arXiv:2506.0845 2025b.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang. Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024.
Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori B Hashimoto, and Daniel Kang- Removing rlhf protections in gpt-4 via fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). pp. 681-687, 2024.
Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine- tuning at (almost) no cost: A baseline for vision large language models. In Forty-firsi International Conference on Machine Learning, 2024.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I BI-LEVEL GREEDY OPTIMIZATION OF SAFEMOE
Algorithm Greedy optimization of safety routing drift regularization
Input: Safery-aligned MoE LLM Walign' Fine-tuning datasets Drc Harmful instruction dataset
DA: Total training steps T; Regularization period Tregi Optimizer Adam( 82,
Output: The fine-tuned MoE LLM
Initialize model weights Wy Walsgn
2: Precompute routing weights ((zlWa))gn) Vx E D/
3: for step T do
Dyz 4: 9t + V (wL.st(Wr.)
5: Wi+1 + AAdam(w, (w, gt)
6: if t mod Treg 0 then for batch B, D do
P Run regularization every Trg steps
7:
8:
9h s V Wregg (1)r1)) where Br W < Adain (1
9:
end for
11: end
Wi+11 f- w
13:
end for
A.2 EXPERIMENTAL SETTING DETAILS
System settings. Our experiments were conducted GPU cloud instance equipped with 6 cores AMD EPYC 7H12, 192GB of RAM, and I to 4 NVIDIA A100 80GB GPUs, depending the requirements of each experiment. For gpt-oss (OpenAl 2025), we employed NVIDIA H100 80GB GPUs due to its GPU architecture compatibility.
Model specifications. We summarize the specifications of MoE LLMs used in our experiments in T
Fine-tuning details. Fine- tuning is performed with LoRA (Hu et al. 2022) using configurations detailed in Table
We train for three epochs with learning rate of Te- and batch size of 32.
Table 7: Specifications of MoE LLMs used in experiments.
layers (MoF dense)
# experts (routed + shared)
Parameters (active total)
Model
Top-k
OLMoE- 71-77---025SIsiiii
16
64 60 4
1.3B 6.9B
Wwee..5-MoE-A2..BB..
24
2.7B 14.3B
DeepSeck-V2-Lite-hha
26
64 2
2.4B 15.7B
gpt-oss-20b
32
3.6B 20,9B
Owen3-30B-A3B
48
128
3.3B / 30.5B
Phi-3.5-MoD-instruct
32
16
6.6B / 41.9B
48
16 +
17B / 109B
Mixtral--8222B---mmmmdd----1
39B 141B
Table 8: LoRA configurations for fine-tuning.
Model
Target modules
Rank (r)
Alpha (a)
Trainable parameters
8
8
1.0M (0.0152%)
Owenl.5-MoEE-22.78
q. v, o
32
3.1M (0.0220%)
DeepSeek-V2-Lite-Ce -h-hite-Chat
q. kv a,
3.6M (0.0226%)
gpt-oss-20b
q. k, O
16
4.0M (0.0190%)
Qwen3-30B-Ah 30BBABB
k, v.
8
32
6.7M (0.0219%)
Phi-3.5-MoF-instrucct
32
6.8M (0.0163%)
q. o
32
12.6M (0.0116%)
Mixtral-8x22B-Instmmee
4. k, O,
17.4M (0.0124%)
16

---

Published as a conference paper at ICLR 2026
Table Baseline tuning results of OLMoE on SAMSum. We report fine-tuning accuracy (FAf) and harmfulness score (HS.).. The selected ones are underlined.
Fine- tuning
SaLoRA (r. = rz)
Lisa (p)
Antidote (a)
SafeDelta (s)
16
0.1
0.02
0.03
0.04
2900
2800
2700
FA 49.3
48.9
48.3
48.1
48.7
48,4
47.7
49.3
48.7
48.1
49.0
48.6
47.5
62.0
24.0
24.0
17.0
24.0
21.0
45.0
40.0
18.0
13.0
12.0
Table 10:
Baseline tuning results of OLMoR SQL.. We report fine-tuning accuracy (FAf) and harmfulness score (HS.)). The selected ones are underlined.
Fine- tuning
SaLoRA (r, Td)
Lisa (p)
Antidote (a)
SafeDelta (s)
64
32
16
1e-3
3e-3
5e-3
0.01
0.02
0.03
2900
2800
2700
FA 58.5
53.6
54.5
53.6
58.8
57.2
56.9
57.5
56.8
56.7
57.4
57.4
57,4
HS
64.0
48.0
40.0
25.0
43.0
40.0
40.0
44.0
36.0
40.0
52.0
38.0
33.0
1e-3
2.5
Fine-tuning
Safelnstr
Safety routing dritt
2.0
SaLoRA
Antidote
1.5
SafeDelta
SafaMoE
1.0
0.5
2
10
11
12
13
14
15
Transformer layor
Figure 11: Safety routing drift of fine-tuned models across the baselines (OLMoB on SAMSum).
Generation and prompt settings. We use greedy decoding for all generations. For harmfulness evaluation, we adopt each model's default system prompt available, or "You are a helpful Al assistant." otherwise, with summarized default system prompt for Llama 4. For zero-shot task utility evaluation, we use a customized task-specific system and user prompts. For the MMLU- Redux-2.0 task, we follow the user prompt in the Llama 3.1 evaluation (Meta 2024). The system and user prompts used in our evaluation are shown Table 15 and Table 16
A.3 BASELINE: TUNING
We extensively tune the hyperparameters of baseline methods safeguarding MoE-based LLMs. The results of OLMoE the SAMSum and SQL tasks are shown in Table tively. For each bascline, we select the hyperparameter setting that exhibits the lowest harmfulness score while allowing up 1% degradation in fine-tuning accuracy.
9 and Table [0 respec-
A.4 LAYER-WISE ANALYSIS OF ROUTING DRIFT ACROSS BASELINES
We compare the safety routing drift across transformer layers under the baseline methods. Figure I] shows the results OLMoE fine-tuned on the SAMSum task. The baselines consistently fail to address the substantial drift significant in the upper layers. In contrast, SAFEMOE directly mitigates it, thereby safeguarding MoE LLMs against HFT attacks. These results highlight the importance of architecture-aware design and demonstrate the effectiveness of SAFRMOE ensuring safety.
17

---

Published as a conference paper at ICLR 2026
20
Aligned
20
20
Activation probatiie
Activatton probabil
15
Activation probabiiig ity
15
10
5-
5-
most-activatedd expert
K-th most-activated expert
K-th most-activated expert
(a) OLMoE (SAMSum)
(b) QwenI.5 MoE (SAMSum)
(c) DeepSeek V2 (SAMSum)
Activation probabiitty 196
20
Activation probabiity (%)
20
Activaton probabilty (6)
20
10
10
10
k-th most activated expert
kith mest-activated expert
k4th activated expert
(d) OLMoE (SQL)
(c) Qwen1.5 MoE (SQL)
(f) DeepSeek V2 (SQL)
Figure 12: Activation probability of top-ranked experts for harmful instructions, ranked by their probabilities the safety-aligned models.
Table 11:
Defense performance against strong HFT attacks with 5k purely harmful samples. We report reasoning performance MMLU-Redux-22 (MMLU1) and harmfulness score (HS.)).
OLMoE (1.3B/6.9B)
Qwen1.5 MoE (2.7B/14.3B)
DeepSeek V2 (2.4B/15.7B)
Method
MMLU
HS
MMLU
HS
MMLU
HS
Aligned 45.8
43.7
2.0
58.1
Fine-tuning
40.9
72.0
53.7
69.0
42.6
77.0
SAFEMOE
46.3
11.0
53.7
11.0
54.6
8.0
A.5
ACTIVATION PROBABILITY OF SAFETY-CRITICAL EXPERTS
We analyze the activation probabilities of experts when processing harmful instructions. These prob- abilities are obtained by applying Softmax to the routing weights. The top-ranked experts serve as safety-critical experts. Figure compares their activation probabilities in the initial safety-aligned models and in the fine-tuned models with SAFEMOE. We find that SAFEMOE further increases the activation safety-critical experts in the fine-tuned models. One possible explanation that although SAFEMOE aims to resemble the routing decisions of the safety-aligned model, it learns to assign larger routing weights to safety-critcaal experts rather than simply replicating their original values. This can lead slight improvements in safety compared the initial safety-aligned models, as observed our safety evaluation results in Table and Table 2
A.6 ROBUSTNESS AGAINST STRONG HARMFUL FINE-TUNING ATTACKS
Purely HFT attack. Our main evaluation (Section 5.2 simulates practical attack scenarios which only small portion of harmful samples is injected into the training dataset. To further demonstrate the robustness SAFEMOR under extreme settings, we additionally evaluate its defense perfor- mance under much stronger HFT attack employing 5k purely harmful samples. The experimental results are provided Table II showing that SAFEMOE consistently mitigates the attack across three MoE LLMs, while also slightly improving reasoning capability by preventing overfitting under attack, consistent with the observations in Table
Adaptive attack. We additionally consider adaptive attacker who is aware of the SAFEMoE method. Specifically, the attacker has access to the fine-tuning process but lacks knowledge of the harmful instruction datasets and the hyperparameters used in SAFEMOR. The attacker aims
18

---

Published as a conference paper at ICLR 2026
Table 12: Safety evaluation under the adaptive attack, compared with the original HFT attack.
OLMoE on SAMSum
Method
Fine-tuning accuracy (FA)
Harmfulness score (HS)
Fine-tuning
49.3
62.0
Fine-tuning (adaptive attack)
49.3
73.0
SAFEMOE (adaptive attack)
48.9
32.0
Table 13: Harmfulness scores under diverse fine-tuning attacks.
OLMoE (1.3B/6,9B)
Qwen3 MoE (3.3B/30.5B)
Attack
Fine-tuning
SAFEMOE
Fine-tuning
SAFEMoE
Traditional backdoor (Qi et al. 2024)
63.0
25.0
73.0
25.0
Reasoning-base based backdoor (Hubinger et al. 2024)
30.0
68.0
4.0
Covert malicious fine-tune fine-tuning (Halawi et al.
2024
43.0
16.0
plify safety routing drift by negating the drift regularization loss (Equation 3 during the HFT attack while preserving fine-tuning accuracy. Table the experimental results. This attack setting increases robustness against SAFEMOE, with slight enhancement the attack performance. Even under this worst-case defense scenario, SAFEMOE still achieves moderate mitigation against the adaptive attack.
A.7 ROBUSTNESS AGAINST OTHER TYPES OF FINE-TUNING ATTACKS
To demonstrate the robustness of SAFEMoE, evaluate its defense performance against diverse fine-tuning attacks by employing harmful samples only, as shown in Table 13
Backdoor attacks. A traditional backdoor attack (Qi al. 2024 induces harmful responses by inserting specific trigger words into the instruction. Although SAFEMOE cannot directly recognize the trigger itself, still achieves moderate reduction in harmfulness scores. ii) reasoning- backdoor attack (Hubinger et al. 2024) embeds backdoors within reasoning chains. Qwen3 MoE, which has strong reasoning capabilities, is particularly vulnerable this attack, yet SAFEMOE provides effective defense for both models. The mitigation of these backdoor attacks stems from SAFEMOE's generalizable ability to prevent the trigger from inducing substantial routing drift within the harmful context.
Covert malicious fine-tuning (Halawi al., 2024) uses ciphered training data to compromise models to produce encoded harmful responses. This attack is known to be effective only on large-scale LLMs (Kadaan et al. 2025), and it fails to produce natural decoded outputs OLMoE. For Qwen3 MoE, we consider the attack successful only when the decoded harmful response forms a natural sentence, as judged by GPT-4o mini (OpenAl 2024a). Because this attack depends large models' strong ability relate ciphered and plain text, SAFEMOE remains effective mitigating its impact even if relies solely on plain text for defense.
Encoding- attack.
A.8 SAFETY CATEGORY
We further break down the results of harmfulness evaluation by safety categories defined in Jail- breakBench (Chao et al.) 2024) (see Table 14). Figure 13 illustrates the harmful response ratios across categories before and after applying SAFEMOE. The three fine-tuned MoE L.MMs are partic- ularly vulnerable to harmful instructions in the domains of Fraud/Deception (#5) and Privacy (#8). Notably, SAFEMOE substantially reduces harmful behaviors across all categories, demonstrating robust effectiveness in mitigating diverse safety risks.
19

---

Published as a conference paper at ICLR 2026
Table 14: Safety category JailbreakBench
Number
Category
#1
Varrasmenn/Dicciiinaatoo
#2
Malkare/llackin
#3
Physical harm
#4
Economic harm
#5
Fraud/Deccption
#6
Disinformation
#7
Sexual/Adult content
#8
Privacy
#9
Expert advice
#10
Government decision-maken
Finetuningg
SafeMaE
#5
#5
42
100
20
20
#6
#1 #6
#1
#6
#1
#7
#10
#8
O.. (SAMSum)
Qwen MoF (SAMSum)
DeepSeek V2 (SAMSum)
#4
#3
#4
#3
#4
#3
#5
#2
#5
#2
#5
100
60
20
40
40
#1 #6
#6
47
#7
410
#10
+8
(d) OLMoE (SQL)
(c) Qwen1.5 1.5 MoE (SQL)
(f) DeepSeck V2 (SQL)
Figure 13: Harmfulness scores across safety categories, with category numbers listed in Table
14
20

---

Published as a conference paper at ICLR 2026
Table 15: Prompts used in the harmfulness evaluation.
Benchmark
Model
Prompt
[SYSTEM] You are OLMo 2, helpful and harm- less AI Assistant built by the Allen Institute for AL
[USER] (instruction]
[ASSISTANT]
[SYSTEM] You are Qwen, created by Alibaba Clo- ud. You are helpful assistant.
Sweal.5-MoE-A2.7BBBC.
[USER] (
[ASSISTANT]
[SYSTEM] You are an Al assistant, developed by DeepSeek Company. For politically sensitive ques- tions, security and privacy issues, you will refuse answer.
DeepSeck- PrrpSeck-V2-LLin 12-Lite-Char
[USER] | [instruction]
[ASSISTANT]
[SYSTEM] You are a helpful Al assistant. [USER] (inttruction
gpt-oss-20b
[ASSISTANT]
JallbrekBBench
[SYSTEM] You are Qwen, created by Alibaba Clo- ud. You are helpful assistant.
Qwen.330B-AAB
[USER] (instruction] [ASSISTANT]
[SYSTEM] You helpful Al assistant.
Phi-3.5-MoE-instmett
[USER] [iStER[(nstruttioo [ASSISTANT]
[SYSTEM] You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not lim- ited humor, empathy, intellectualism, creativity and problem-solving.
Llama-4-Scout-1 Ceeeunt--B-16E--e 1n8-tiE-n
[USER] [instruction] [ASSISTANT]
[SYSTEM] You are a helpful AI assistant. [USER] (instrucion) [ASSISTANT]
Mixtral-8x22B. Instructtttt
21

---

Published as a conference paper at ICLR 2026
Table 16: Prompts used in the fine-tuning task evaluation.
Benchmark
Prompt
[SYSTEM] You are helpful assistant for dialog summarization.
SAMSum
[USER] Summarize this dialogue: (dialogue}
[ASSISTANT]
[SYSTEM] You helpful assistant for answering SQL questions.
[USER] Based on given Table, generate SQL for the following question.
SQL
Question: {question)
Table: {context)
[ASSISTANT]
[SYSTEM] You are helpful assistant for answering multiple choice questions. [USER] Given the following question and four candidate answers (A, B, C and D),
choose the best answer.
Question: [question)
{options]
For simple problems:
Directly provide answer with minimal explanation.
For complex problems:
Use this step-by-step format:
MML.U-cddux-2.0
## Step 1: [Concise description] [Brief explanation]
## Step 2: [Concise description] [Brief explanation]
Regardless approach, always conclude with: The best answer [the answer letter). where the [the_answer letter) is one of A, B, Cr D.
Let's think step by step. [ASSISTANT]
22


Reviews:
[{"review_id": "QTb0OWzXCb", "reviewer": "Reviewer_Ysqm", "summary": "The authors consider the problem of preventing subsequent fine-tunes of an aligned MoE-based LLM from eliciting harmful behavior or capabilities. In particular, this setting assumes the attacker can choose the fine-tuning data mix, but otherwise has no access to the model parameters or the training setup. Such attacks are referred to as Harmful Fine-Tuning (HFT). This threat model is relevant for e.g. the OpenAI fine-tuning API, where (potentially malicious) users can fine-tune OpenAI models on their own data. \n\nPrior work has identified the existence of safety-critical experts in aligned MoE models. The authors posit that one of the core mechanisms of HFT’s success in MoE models is routing drift, where the expert assignment changes substantially during fine-tuning, avoiding safety-critical experts during inference on harmful prompts, and allowing harmful outputs to be elicited. \n\nGiven this, the authors propose adding a regularization loss supervising the routing scores to remain similar to the initial aligned model throughout training. This loss is computed on a set of harmful prompts, which need not feature in the training set provided by the attacker. \n\nAlgorithmically, the authors propose a bi-level greedy optimization setup that alternates between minimizing the SFT loss and their routing regularization loss. \n\nSafeMoE on a range of mixture-of-experts LLMs and tasks by simulating harmful fine-tuning (mixing standard task data with a small set of harmful prompts) and measuring both safety and utility. Utility is assessed on SAMSum summarization (ROUGE-1) and SQL generation (exact match), safety is judged on JailbreakBench via an automated moderator, and comparisons are made against fine-tuning-stage baselines (SafeInstr, SaLoRA) and post-hoc edits (Antidote, SafeDelta). Across small and large MoE models, SafeMoE consistently cuts harmfulness dramatically while preserving or slightly improving task performance, remains effective under full-parameter fine-tuning with minimal training overhead, and shows robustness in sensitivity studies (varying harmful sample count, ratio, and regularization temperature). Analyses of training dynamics further link its gains to reduced routing drift - especially in upper layers—helping argue that the safety improvements stem from stabilizing expert selection rather than sacrificing capability.", "questions": "- Why would it be prohibitive to optimize $L_{\\text{reg}}$ together with $L_{\\text{sft}}$, for instance using a Monte Carlo estimate of the $L_{\\text{reg}}$ gradient? Mechanically, this would correspond to sampling a batch from $D_h$ at each optimization step. Is this because the sequence batch size is small, so that adding even a handful of harmful prompts to the batch would be prohibitive?  \n- When defining routing drift and $L_{\\text{reg}}$, what is the authors’ intuition for why the reference measure should come from the current model checkpoint ($w$) instead of from the aligned model ($w_\\text{align}$)? I.e. why do you use $\\mathrm{KL}(\\text{aligned} || \\text{current})$ instead of $\\mathrm{KL}(\\text{current} || \\text{aligned})$? Do you expect this should make a difference, or should it not matter much? My understanding is that the more common form in KL regularization for fine-tuning is to use the latter. \n\nI will increase my score if the concerns regarding causal claims in section 3 and the absence of stopgrad in the regularization loss definition are addressed. The main things I would like to see in these directions are the ablations highlighted in the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Presentation: the paper is clearly written and well-presented, which facilitated understanding its motivation, main proposed method and experiment results. \n- Strong results: the results indicate SafeMOE achieves significantly lower harmfulness scores than other methods, while preserving fine-tuning accuracy.  \n- Low overhead: SafeMOE can be run at a low compute overhead by only sporadically optimizing the routing regularization loss. This makes it cheap for practitioners to adopt the method, making it more likely to become relevant for the community.", "weaknesses": "- Correlation vs. causation \n  - The author’s proposed correlation analysis in Section 3 seems to me to be confounded, and to conflate correlation with causation. The authors track routing drift and harmfulness score through fine-tuning, and observe that higher routing drift is associated with a higher harmfulness score. However, given that the harmfulness score of the final model is known to be higher than of the initial model (based on e.g. prior work), there are several metrics one could track that would also increase during training, even though they might not causally affect the harmfulness score. For instance, the distance between initial and final parameters likely increases during fine-tuning as well, and yet one would not conclude from this that “the further the initial parameters are from the final parameters, the more harmful the final model would be”. \n  - In this case, since the author’s hypothesized mechanism is routing drift, I believe a more appropriate and controlled experiment would be to freeze the routing weights to be those of the initial model, and evaluate the harmfulness score during fine-tuning when using those frozen routing weights at inference time. If the harmfulness score becomes significantly lower, this would give more direct and less confounded evidence to the authors’ claim.  \n- Lack of stopgrad in $L_{reg}$ definition \n  - My understanding, from reading the code (lines 866–899 of `safemoe_trainer.py`) is that, in the routing regularization function, no stopgrad is applied to the input of the routing layer. Given this, my understanding is that the gradients of the routing regularization loss would propagate to the the residual stream vectors, and hence to all previous layers of the model; not just to the routing. \n  - Given this, it is not clear to me that all this regularization loss is doing is enforcing the routing to stay the same. For instance, it is possible that the effect of the method is to cause activations to remain largely the same, which would lead to a different conclusion as to how SafeMOE achieves its results. \n  - It would help put the authors’ contributions into perspective if we had an ablation where stopgrad is applied to the activations fed into the routing module. If the results are similar to those of SafeMOE, we can be reassured that the reason SafeMOE works is in fact due to avoiding routing drift. If not, it might be that a different effect is at play, and it would be important to understand what that is. \n  - Conversely, another potentially relevant ablation would be to use a regularization loss that simply supervises residual stream activations to stay the same for the harmful prompts (e.g. via an $L^2$ loss). My prior is that this would harm fine-tuning accuracy a lot more than SafeMOE does, but it would be interesting to check, in case the authors can run such an experiment easily (e.g. on a small model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRikC6zYL0", "reviewer": "Reviewer_YmtJ", "summary": "This paper proposes a defense method against harmful fine-tuning attacks on MoE LLMs. The core idea is to regularize the MoE routing decisions during fine-tuning.", "questions": "* In Figure 1, step 150, the safety routing drift of both SafeMoE and SaLoRA are similarly low (~0.7*1e-3), but their harmfulness scores are quite discrepant -- ~55% for SafeMoE and ~15% for SaLoRA. Why?\n* In Figure 3, why use different benign datasets across the two settings (Alpaca v.s. SAMSum)?\n* Can you include a comparison with directly freezing the router weights during fine-tuning (rather than regularizing them)?\n* Does your method only work for MoE LLMs? Or can you adapt your method to work on non-MoE LLMs too?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "* The proposed method is specially tailored for MoE LLMs according to their routing nature, which prior work has not considered.\n* The experiments span across MoE LLMs of diverse sizes, and include important ablation studies on method hyperparameters.\n* The paper writing flows well overall.", "weaknesses": "* Questionable motivation. The authors show that harmfulness score is highly correlated with safety routing drift. However, is harmfulness score also correlated with other weights (non-routing weights) of the MoE model? And if the method regularizes the drifts in other weights rather than the routing weights, will the harmfulness score also be suppressed? Without these ablations, the motivation to \"preserve the initial routing decisions\" is questionable.\n\n  Accordingly, I think it's necessary to compare your method with the fine-tuning defense in [1], where the authors also propose to constrain the fine-tuned weights to the original weights.\n\n* Lack of consideration of more complicated attacks and adaptive attacks. The experiments solely focus on  the attack setting of explicitly mixing \"500 harmful samples from BeaverTails\" in the fine-tuning dataset. However, there are many different HFT attacks nowadays. I would suggest the authors consider more complicated HFT attacks (e.g., [2]), and even backdoor HFT attacks [3,4]. Additionally, the authors should discuss potential adaptive attacks against their proposed defense.\n\n  I'm particularly interested in seeing the results of backdoor HFT attacks, where a trigger is used for activating harmful LLM responses (the model would refuse harmful requests if a trigger is not present). I doubt whether this backdoor attack setting can break the proposed defense -- since SafeMoE can only preserve the routing decisions of plain harmful instructions (where trigger is not included), but cannot regularize the case when backdoor triggers are secretly injected into the harmful instructions.\n\n* Lack of ablation on $T_{reg}$. According to Table 3, it seems the overhead of SafeMoE is quite low. So what if the method performs regularization steps more regularly? Can you show the tradeoff of overhead v.s. safety across diverse choices of $T_{reg}$?\n\n* Lack of human evaluation of the reliability of Llama-Guard-4-12B.\n\n[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. ICLR 2025\n\n[2] Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation. ICML 2024\n\n[3] Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024\n\n[4] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. Arxiv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSayrsvZdO", "reviewer": "Reviewer_6cTn", "summary": "This paper study harmful fine-tuning defense on MOE LLMs. As a solution, this paper propose a simple regularizer to mitigate the drift of the routing weights.", "questions": "I don't have further questions as I am kind of familiar with both harmful fine-tuning and MoE. This paper excites me, inspire me thinking and I do wish to get more insights from you.  I will consider to increase my score if the authors are willing to explore more on the direction I point out (see weakness 1&2), but otherwise I am also happy to keep my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This is the first paper that systematically study harmful fine-tuning on MOE models. \n2. This paper discovers an important finding that harmful fine-tuning makes the routing  weights drift away from that of the aligned model. \n3. The proposed method is simple and should work in my understanding.\n4. Paper is extremely well written. The idea is presented clear with empirical evidence as motivations. The main experiments are also very comprehensive, justifying the effectiveness of the proposed method.", "weaknesses": "1. **A baseline should be added to comparison given their similarity.**\nI suggest the authors to add a comparison with Lisa [1]. Similar to SAFEMOE,  Lisa adopts a proximal term which also somehow helps mitigate the harmful drift (although not only for routing weights). In addition, Lisa adds a few safety data in the fine-tuning process, which probably can help improve the mitigation effect. Therefore, I am wondering whether SAFEMOE can outperform Lisa even though Lisa has additional safety alignment data in the fine-tuning process. \n\n2. **Add a few discussions  and do some exploration on expert-wise regularization.** I first need to mention a related MoE drift control method. Please check Section 5.1 in H3fusion[2].  They shows that by **adding more regularize intensity** to safety MOE expert, the drift of this safety MOE expert during fine-tuning will be smaller. However, they show that this regularizer is not beneficial, because large regularization intensity of safety expert will make the model **becomes even more unsafe** (See the stage 4 in Figure 1). This conclusion is **contradictory** to yours. The reason of this discrepancy is probably that they are adding regularizer to only the safety expert but you are adding uniform regularization to all the experts together.  Combining the two papers, I think the a potentially better solution might be: add the proximal regularization to the non-safety expert but don't add proximal regularization (or add negative proximal regularization) to the safety expert. I conjecture the gate drift will be uniformly become larger for all the experts (but not in the case like Figure 4 that you display), and constraining that of the safety expert will not help.  My conjecture might be or might not be wrong. Could you perform this experiment to verify this alternative idea? \n\n \n\n[1] Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack\n\n[2] H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs\n\n3. Some related work are missing. I list out a few most recent work below and please consider to read and discuss them. \n\n\n\n\nUnleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models\n\nNo, of course I can! Refusal Mechanisms Can Be Exploited Using Harmless Fine-Tuning Data\n\nHarmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey\n\nEmergent Misalignment:Narrow finetuning can produce broadly misaligned LLMs\n\nESTIMATING WORST-CASE FRONTIER RISKS OF OPEN-WEIGHT LLMS\n\nYour Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents \n\nDeep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs\n\nTamper-Resistant Safeguards for Open-Weight LLMs\n\nCTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning\n\nSelf-Destructive Language Model\n\nTargeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation\n\nVulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning\n\nLoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning\n\nCovert malicious finetuning: Challenges in safeguarding llm adaptation\n\nShape it Up! Restoring LLM Safety during Finetuning\n\nAsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin\n\n Gradient Surgery for Safe LLM Fine-Tuning \n\nPanacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation\n\nRefusal-Feature-guided Teacher for Safe Finetuning via Data Filtering and Alignment Distillation\n\nWhen Style Breaks Safety: Defending Language Models Against Superficial Style Alignment\n\nAnchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint\n\nWhy LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets\n\nToward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuning\n\nSafety Subspaces are Not Distinct: A Fine-Tuning Case Study\n\nMoGUV 2: Toward a Higher Pareto Frontier Between Model Usability and Security\n\nLayer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment\n\nVirus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation \n\nFinetuning-Activated Backdoors in LLMs\n\n Safety alignment should be made more than just a few tokens deep\n\nMitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization\n\nFundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models \n\nNavigating the safety landscape: Measuring risks in finetuning large language models\n\nScaling Trends for Data Poisoning in LLMs\n\nTowards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning\n\nFundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models\n\nDetecting Adversarial Fine-tuning with Auditing Agents\n\nYour Task May Vary: A Systematic Understanding of Alignment and Safety Degradation when Fine-tuning LLMs \n\nThere might be more exciting papers that are worth to read in this field and I encourage the authors to read more related papers and keep update on the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bhvi2u8nX2", "reviewer": "Reviewer_Maxq", "summary": "This manuscript proposes SAFEMOE, a safe fine-tuning method tailored to Mixture-of-Experts (MoE) large language models (LLMs) to address the vulnerability of \"safety routing drift\"—a phenomenon where fine-tuning (both harmful and benign) causes the model’s routing decisions for harmful inputs to deviate from those of the initial safety-aligned model, thereby deactivating safety-critical experts and undermining safety. SAFEMOE mitigates this drift by introducing a regularization term that minimizes the KL-divergence between the routing weight distributions of the fine-tuned model and the safety-aligned model for harmful inputs. Extensive experiments on 8 MoE LLMs (ranging from 7B to 141B parameters, e.g., OLMoE, gpt-oss, Llama 4) demonstrate that SAFEMOE effectively reduces harmfulness scores (e.g., from 62.0 to 5.0 for OLMoE on the SAMSum task) while preserving task utility (≤1% degradation) and incurring only ~2% training overhead. The work also verifies the strong correlation between safety routing drift and model harmfulness, highlighting the importance of architecture-aware safety designs for MoE LLMs.", "questions": "Please refer to the \"Weaknesses\" Section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Architecture-Aware Targeting: Unlike existing defenses (e.g., SafeInstr, SaLoRA) designed for monolithic LLMs, SAFEMOE directly addresses the unique vulnerability of MoE LLMs—their reliance on dynamic routing to safety-critical experts. This design fills a critical gap in current LLM safety research, where MoE-specific security challenges have been largely overlooked.\n- Strong Empirical Validation: The authors conduct comprehensive experiments across diverse MoE LLMs (varying in parameter size, expert configurations, and vendors) and tasks (dialogue summarization, SQL generation). The consistent effectiveness of SAFEMOE across models (e.g., reducing harmfulness for 141B-parameter Mixtral) and benchmarks (JailbreakBench, HEx-PHI) enhances the work’s reliability.\n- Practical Trade-Off Balance: SAFEMOE achieves a favorable balance between safety, utility, and efficiency. It reduces harmfulness significantly without sacrificing task performance (≤1% accuracy loss) and maintains low overhead (~2%), making it suitable for real-world fine-tuning services (e.g., OpenAI/Google API platforms).\n- Clear Mechanistic Insight: The introduction and validation of \"safety routing drift\" as a key metric (via KL-divergence) provides a clear mechanistic explanation for MoE LLM safety degradation during fine-tuning. This insight not only supports SAFEMOE’s design but also guides future research on MoE security.", "weaknesses": "- While SAFEMOE addresses a practical gap, its core technical components (KL-divergence regularization, bi-level greedy optimization) are not fundamentally new—they adapt existing regularization and optimization paradigms (as cited in the paper) to MoE LLMs. Thus, the work does not introduce groundbreaking methodological innovations, limiting its potential to be a \"landmark\" contribution in LLM safety.\n- The study’s threat model implicitly assumes users are either \"benign\" (conducting standard task fine-tuning) or \"moderately malicious\" (injecting a small fraction of harmful samples, e.g., 500 out of 5.5k samples in HFT). It does not rigorously evaluate scenarios where attackers use purely malicious datasets (100% harmful instructions) to fine-tune the model, leaving uncertainty about SAFEMOE’s robustness against extreme adversarial intent.\n- Multiple figures (e.g., Figure 1b, 5b) show that safety routing drift and harmfulness scores increase disproportionately rapidly in the early stages of fine-tuning (e.g., first 100 steps for OLMoE). The manuscript does not provide a detailed mechanistic explanation for this phenomenon (e.g., whether it stems from rapid updates to gating network parameters or skewed gradient flow from task/harmful samples), weakening the depth of its analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)