Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Authors: Xu Zhang
Kaidi Xu
Ziqing Hu
Ren Wang
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Experts
Adversarial Defense
Adversarial Robustness
Primary area: deep_learning->robustness

Abstract: Mixture of Experts (MoE) have shown remarkable success in leveraging specialized expert networks for complex machine learning tasks. However, their susceptibility to adversarial attacks presents a critical challenge for deployment in robust applications. This paper addresses the critical question of how to incorporate robustness into MoEs while maintaining high natural accuracy. We begin by analyzing the vulnerability of MoE components, finding that expert networks are notably more susceptible to adversarial attacks than the router. Based on this insight, we propose a targeted robust training technique that integrates a novel loss function to enhance the adversarial robustness of MoE, requiring only the robustification of one additional expert without compromising training or inference efficiency. Building on this, we introduce a dual-model strategy that linearly combines a standard MoE model with our robustified MoE model using a smoothing parameter. This approach allows for flexible control over the robustness-accuracy trade-off. We further provide theoretical foundations by deriving certified robustness bounds for both the single MoE and the dual-model. To push the boundaries of robustness and accuracy, we propose a novel joint training strategy JTDMoE for the dual-model. This joint training enhances both robustness and accuracy beyond what is achievable with separate models. Experimental results on CIFAR-10 and TinyImageNet datasets using ResNet18 and Vision Transformer (ViT) architectures demonstrate the effectiveness of our proposed methods.

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Xu Zhang' Kaidi Xu²
Ziqing Hu
Ren Wang'
Abstract
1. Introduction
Mixture of Experts (MoE) have shown remarkable success leveraging specialized expert networks for complex machine learning tasks. However, their susceptibility to adversarial attacks presents critical challenge for deployment in robust ap- plications. This paper addresses the critical ques- tion of how to incorporate robustness into Moïs while maintaining high natural accuracy. We be- gin by analyzing the vulnerability of MoE com- ponents, finding that expert networks are notably more susceptible to adversarial attacks than the router. Based on this insight, we propose a tar- geted robust training technique that integrates a novel loss function to enhance the adversarial ro- bustness of MoE, requiring only the robustifica- tion of one additional expert without compromis- ing training or inference efficiency. Building on this, we introduce a dual-model strategy that lin- early combines standard MoE model with our robustified MoE model using smoothing param- eter. This approach allows for flexible control over the robustness- accurracyyyyyyyyyyccco trade-off. We fur- ther provide theoretical foundations by deriving certified robustness bounds for both the single MoE and the dual-model. To push the bound- aries of robustness and accuracy, we propose novel joint training strategy JTDMMeE for the dual-model. This joint training enhances both robustness and accuracy beyond what is achiev- able with separate models. Experimental results on CIFAR-10 and TinylmageNet datasets using ResNet18 and Vision Transformer (ViT) archi- tectures demonstrate the effectiveness of our pro- posed methods. The code is publicly available at hattps://gt thub. , com TIm//TIMM--oooo. Robust MooE- Duee--ual--a 1 MModel.
The Mixture of Experts (MoE) architecture has emerged as powerful framework in machine learning, enabling models to capture complex patterns by combining the strengths of multiple specialized expert networks. Originally introduced to enhance model capacity without proportional increase in computational cost, the MoE framework operates in a straightforward but effective way: different components of the model, known as experts, specialize in distinct tasks or features of the data (Jacobs et al., 1991; Jordan & Jacobs, 1994). MoE leverage router dynamically assign input data to the most appropriate expert (Shazeer et al., 2017). Such mechanism makes them particularly valuable in large- scale applications such as natural language processing (Du et al., 2022) and computer vision (Riquelme et al., 2021).
Despite their success in achieving high accuracy, MoE, simi- lar to other deep learning models, vulnerable adversar- ial attacks. Adversarial examples, which are input samples perturbed by imperceptible noise, can induce deep learning models to produce incorrect predictions with high confi- dence (Goodfellow et al., 2014; Carlini & Wagner, 2017; Papernot et al., 2016), posing significant risks in safety- critical applications (Finlayson al., 2019; Li et al., 2023). The same modular structure that empowers MoE renders them particularly susceptible to adversarial attacks, each expert presents potential vulnerability. Our experiments in Section 4.1 confirm that targeting MoE experts can be an ef- fective strategy for compromising the model, underscoring the need for enhanced robustness measures in MoE.
Adversarial Training (AT) and its variants have been ex- tensively researched to defend against adversarial attacks, attracting considerable research interest (Madry et al., 2017; Zhang et al.. 2019; Wang et al., 2020a; Wong et al., 2020; Wang et al., 2021: 2023). However, these methods predom- inantly focus on standard architectures and don't 't directly address the unique challenges of MoE. The heterogeneous structure of MoE, comprising a router and multiple expert networks, complicates the application of traditional robust- ness techniques. A few existing works have explored ad- versarial robustness in MoE architectures (Puigcerver et al., 2022; Zhang et al., 2023). Nonetheless, they often fail to analyze the robustness of each component and typically only
'Illinois Institute Technology Drrexel University "Perplxxity AL Correspondence to: Ren Wang
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
consider traditional AT impose architectural restrictions. Moreover, the robustness enhancements in these works sig- nificantly degrade standard accuracy, limiting their practical utility.
This approach has been successfully applied in Natural Lan- guage Processing (Du al., 2022; Lewis al., 2021) and Computer Vision (Riquelme et al., 2021; Xue et al., 2022).
Despite their widespread application, there has been limited research on the robustness of MoF, especially in adversarial settings, Small, carefully crafted perturbations, known as adversarial examples, can cause deep neural networks to make incorrect predictions & Wagner, 2017: Papernot et al., 2016). Defenses against such attacks often rely on adversarial training methods based on min-max optimization (Madry et al., 2017; Zhang et al., 2019; Wang et al., 2020a; Wong et al., 2020). In the context of MoEs, only a few initial studies have begun exploring ad- versarial robustness with obvious limitations. The first work focused on Vision Transformers (ViTs) with MoE structures, examining the relationship between model capacity and ro- bustness, only considering traditional adversarial training to robustify the model (Puigcerver et al., 2022). Another study investigated adversarial robustness of MoBs with a method only working on convolutional neural networks (Zhang et al., 2023). Moreover, these methods sacrifice standard accuracy for robustness, limiting their practical applications. Another line of related works is ensemble methods, which combine predictions from multiple models. Ensembles can improve robustness by aggregating predictions from multiple mod- els, reducing the impact of individual vulnerabilities (Liu et al., 2018; Alam et al., 2022; Co et al., 2022; Bai et al., 2024a;b). While MoE models share conceptual similarities with ensembles by leveraging multiple sub-models they dif- fer fundamentally due to their dynamic routing mechanism, where a router assigns inputs to specific experts rather than combining outputs from all experts. This distinction neces- sitates tailored approaches for enhancing MoE robustness. Our work introduces a comprehensive framework to robus- tify MoEs while optimizing the balance between robustness and accuracy.
In this paper, we aim to bridge the gap between adversarial robustness and accuracy in MoEs. We begin by examining MoE components to identify their susceptibility to adversar- ial attacks, finding that expert networks are more vulnerable than the router. Based on this, we propose Robust Training with Experts' Robustification (RT-ER) to enhance MoE ro- bustness by applying robust training to the expert networks without sacrificing training or inference efficiency. Addi- tionally, we explore combining standard and robust MoE models as dual-models to balance aceuracy and robustness. We also derive theoretical robustness bounds for both the sin- gle MoE and limits of adversarial robustness and guiding the develop- ment of a joint training strategy, JTD Our methods are illustrated in Figure I.
et al., 2014; Carlini
offering insights into achievable
Our contributions are summarized as follows:
Assessment of MoE Vulnerabilities and Incorpora- tion of Robustness: We identify key MoE components vulnerable to adversarial attacks and propose the RT- ER method, which enhances the robustness of expert networks within the MoE architecture while maintain- ing efficiency in both training and inference.
Dual-Model Strategy: We introduce a framework combining standard and robust MoE models to balance the trade-off between accuracy and robustness.
Theoretical Foundations: We derive robustness bounds for MoE and the dual-model, offering deeper insights into the capabilities and constraints of our ap- proach.
Joint Training Strategy: Leveraging our theoretical findings, we develop a joint training strategy, JTDMoEE for the dual MoE model that enhances both robustness and accuracy.
3. Preliminaries
In this section, we introduce MoE architecture considered in this paper and outline how adversarial attacks can target different components of the MoE model.
2. Related Work
MoE has long been explored in the machine learning com- munity as a method for tackling complex tasks by combin- ing specialized expert networks (Jacobs et al., 1991). Each expert focuses on certain aspects of the data, and their out- puts are combined through a weighted sum determined by gating mechanism or router (Yuksel et al., 2012). A no- table advancement is the sparsely gated MoE (Shazeer et al., 2017; Wang et al., 2020b; Riquelme et al., 2021; Fedus et al., 2022; Xue et al., 2022), which activates only a subset of experts based on a routing mechanism, allowing conditional computation and enabling models to scale parameters in- dependently of computational cost (Patterson al., 2021).
MoE Architecture. The MoE (Jacobs et al., 1991; Jordan Jacobs, 1994) is neural network architecture that leverages multiple specialized sub-networks, known experts, to im- prove modeling capacity and performance on complex tasks. A router (also referred to as gating network) determines the contribution of each expert to the final prediction based on the input data. Formally, let E' denote the number of experts in the MoE model. Each expert is function fa (x), where 1, 2, E, and x E Rd represents the input data. The router computes routing weights a,(x), which are typically non-negative and sum to one, i.e., ΣE (x(x) = 1. The
2

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Alack
Inputxx
Single Rohurticction
Dual-Model
Updater
Standard MoB
Robust MoEi
Robustmess
Model
los.Fal* 6).31)
Pa(a))
+
Reater
Reuter
Expert Networks
Exparts Robustmess
Update
Craifola (.11lo
Fals)
Update
( o) Fole)) Folo)
Output
Cr max 11
Los 5),8) Bxdlaffle fx())
(estFu(r).
Hully
Fo(*)
Subyos
mij
Figure 1. Illustration of methods enhancing the robustness of single MoE and our joint training strategy for the dual-model. Right: Our single MoË robustification method enhances the robustness of a single MoE FR by introducing an additional term to reinforce the robustness of second-top expert fz beyond standard adversarial training. Left: The dual-model linear combination of standard MoE Fs and robust MoE Fn. The jointly-trained dual-model (JTDMoE) improves robustness while maintaining high standard accuracy using a bi-level alternating training approach.
final prediction of the MoE model given by weighted sum of the experts' outputs F(x) = E a, fx In setup, the router (x) is implemented using fully con- nected layer, and each expert is neural network tailored to capture specific aspects of the data.
tions of MoE models, as they are largely independent the router's behavior. Empirically, our experiments show that in 98% of cases, the perturbed inputs still routed the same expert(s), indicating that the expert- targeted attacks remain effective despite this assumption.
Adversarial Attacks on MoE Models. An adversarial at- tack on the entire MoE model seeks to find perturbation 5 such that the model's output on the perturbed input x + 8 differs significantly from the output on the original input or its ground truth label y: F(x 8) DEn ax(x+ f) (x + 5), where the 8 is usually generated by maximiz- ing the loss function arg ma L(F(
4. Robust Training for Mixture of Experts
In this section, we aim to enhance the adversarial robustness of MoE. The key question we pose here is as follows.
(Q1) Which part of MoE most vulnerable attacks and how should we robustify the critical part?
5). y) under the tp norm and the attack budget e. We also consider loss terms that do not rely on y. Notice that the attack is on the whole MoE. We implement adversarial attacks focusing on specific components of the MoE model. An attack targeting the router aims alter the routing weights without affecting the experts' outputs. The perturbation 8 is crafted by chang- ing the router's decisions while the experts receiving the original input: F(x+5) = E (xxx))).(xx... An attack targeting the experts aims to alter the experts' outputs with- out changing the routing weights.
4.1. Assessing the Robustness of MoE Components
To enhance the robustness of MoE, it is essential to under- stand which component is most susceptible to adversarial attacks. We begin by assessing the robustness of both the router and expert networks MoE when subjected to adver- sarial attacks. We analyze the vulnerability of these compo- nents by evaluating standard MoE model trained with the cross-entropy loss arg mine, fC (Fs(x). is the output of the standard MoE model Og that has only seen clean data during the training.
designed such that:
(x). v), where Fs(.))
We remark that once the perturbation ò is generated, will be implemented on both the router and experts despite of its target. When attacking the expert networks, the perturbations are generated under the assumption that the routing scores remain unchanged. The generated perturbations can still alter the final predic-
F -E a (x
To specifically evaluate the accuracy and robustness of MoE, and the robustness of the router and the experts, we intro- duce four metrics: O Standard Accuracy (SA) is the ac- curacy on clean test data; @ Robust Accuracy (RA) is the
3

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
accuracy on adversarially perturbed test data generated by attacking the whole MoE; @ RA-E is the accuracy on ad- versarially perturbed test data generated by attacking the experts; O RA-R is the accuracy on adversarially perturbed test data generated by attacking the router. The RA-E/RA- R reflects the experts'/ron /router's ability to maintain correct predictions when subjected to adversarial perturbations.
where (+()) is the output of the robust MoE model R un- der robust training. However, when applied to MoE models, traditional adversarial training exhibits several issues: (II) Low Robust Accuracy: The final RA remains unaccept- ably low. In our experiments on CIFAR-I0 with ResNet18 experts, the model achieves 79.08% SA and only 53.74%
RA on the test set using PGD after 130 training epochs. (12) Training Instability: The training process is unstable, with sudden drops in both SA and RA. As shown in Figure 2, both metries decline sharply between epochs 80 and 90, This is due to the router selecting different expert as the primary contributor when faced with adversarially perturbed inputs. Since standard adversarial training primarily focuses on optimizing the robustness of the overall model output, it may overlook individual experts' robustness, resulting in some experts remaining vulnerable attacks. Consequently, when the router routes to one of these weaker experts, the model's robustness is compromised. These issues suggest that traditional adversarial training is not well-suited for MoE architectures due to the complex interplay between the router and expert networks.
Table 1. Robustness assessment of MoE components on CIFAR-10 and TinyImageNa RA-E values are small under both Projected Gradient Descent Attack (PGD) (Madry et al., 2017) and AutoAt- tack (AA) (Croce& & Hein, 2020), indicating the vulncrability of the experts. Throughout the paper, we highlight the most vulnerable metric in bold emphasize its susceptibility.
CIFAR-10
Attack Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
PGD
92.14
52.54
3.11
54.67
AutoAttack
92.14
14.25
2.95
55.2
Tiny Imagentt
Attack Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
PGD
82.05
34.53
29.21
59
AutoAttack
82.05
14.24
26.11
58.95
Based on our earlier assessment (in Section 4.1) that the experts are the most vulnerable components, we propose a new robust training approach that specifically targets enhane- ing the robustness of the expert networks. Our approach modifies the loss function to include a Kullback-Leibler (KL) divergence term that encourages the experts outputs on adversarial examples to be similar to their outputs on clean inputs. We propose a Robust Training with Expert Robustification (RT-ER) approach:
We conduct a pilot study on CIFAR- 10 and TinyImageNet datasets using MoF. models with four experts. For CIFAR- 10, we use 8 as experts and set e 8/255. For TinylmageNet, we use ViT-small as experts with c = 2/255. We employ 50-step Projected Gradient Descent Attack (PGD) (Madry et al., 2017) and AutoAttack (AA) (Croce& & Hein, 2020). The results are summarized in Table 1. On both datasets, the RA-R remains above 50% under both PGD and AA, suggesting that the router maintains reasonable per- formance under attack. In contrast, the RA-E drops below 4%/30% on CIFAR-IO/TinyInaeNNN experts are highly susceptible to adversarial perturbations. This vulnerability causes the overall RA of the MoE model to be low. The experimental results indicate that the expert networks are significantly more vulnerable to adversarial attacks than the router. This is due to the fact that the ex- perts are more complex than the router architecture. These findings highlight the importance of focusing on enhancing the robustness of the expert networks to improve the overall resilience of MoE models against adversarial attacks.
min Crin = max (CE(FR(x* + 8),
0
110
(2)
+ +B- + f2.
indicating that the
where f2(x) denotes the output of the expert with the second-largest router weight for the adversarial example. The term (xz(fe(x + 8), fx()) represents the KL diver- gence between the output of this expert on the adversarial example and its output on the clean input. RT-ER is com- patible with various routing strategies. For instance, when the router employs top- strategy, RT-ER selects an addi- tional expert-one one not initially chosen by the router bastrr-based on the router weights, leveraging the router's 's output. We adopt KL. divergence instead of cross-entropy for two pri- mary reasons. First, cross-entropy relies on a well-defined ground-truth target distribution, which is unavailable in our setting. Second, employing cross-entropy in the second term would compel the predictions to closely match the ground truth, potentially resulting in overfitting. In contrast, KL divergence enables a softer alignment between experts, thereby reducing the risk of overfitting and offering greater flexibility in modeling uncertainty. The hyperparameter controls the trade-off between MoE-wide robustness and
4.2. Incorporating Robustness into MoE Architecture
To incorporate robustness into the MoE architecture, straightforward approach is to apply traditional adversarial training (Madry et al., 2017), which involves training the model on adversarial examples generated during training. The traditional adversarial training is defined as:
min max
(1)
@x م|
4

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
expert-specific robustness. For clarity, we refer the ex- pert with the second-largest router weight the second-top expert in the following discussion.
over, RT-ER requires training only one additional expert, introducing minimal overhead to training efficiency. Further robustness evaluations under smooth attacks on MoE are provided in Appendix A.3. In summary, RT-ER not only strengthens the robustness of MoE architectures but also preserves both training and inference efficiency.
The concept of RT-ER is illustrated in the right panel of Fig- ure 1. By incorporating the KL. divergence term, we explic- itly encourage second-top expert to produce similar outputs for both clean and adversarial inputs. RT-ER offers sev- eral advantages: (A1) By penalizing deviations in experts' outputs due adversarial perturbations, we strengthen the experts against attacks and thus contributes to better over- all RA for the MoE model. (A2) The inclusion of the KL divergence term helps stabilize training by regularizing the experts' behavior. When second-top expert is also robust to adversarial inputs, the model maintains greater stability, even when the router activates different experts. This en- hanced consistency reduces performance fluctuations under adversarial attacks, allowing the model to sustain stronger performance on adversarial samples. By ensuring all experts are robust, the model avoids scenarios where only subset of experts is resilient, leaving others vulnerable. This uniform robustness across experts leads to a more reliable defense against adversarial perturbations, improving the model's overall resilience. The effectiveness of our proposed robust training method is demonstrated through the experimental results presented in Section 6.2. We observe significant improvements in RA compared to traditional adversarial training and a more stable training process, validating the efficacy of focusing on expert robustness in MoE models.
5. Dual-MMoel Strategy for Robustness and Accuracy
While our proposed RT-ER method enhances robustness against adversarial attacks (measured by RA), it may inad- vertently degrade performance on clean data (measured by SA). This prompts the following research question:
(Q2) How can we incorporate robustness into Mixture of Experts while minimizing the robustness-accuracy trade-off?
5.1. Dual-Model with Post- Dost-Training MoE
Traditional MoE models are efficient but vulnerable to at- tacks, while robust MoEs withstand attacks but often reduce standard accuracy. We explore whether combining both models can balance robustness and accuracy. Let Fs(x) be standard MoE and FR((x) robust MoE. The dual-model, (xx).. is defined as:
Fx (1 (1 -((x)) a FR(();
(3)
where controls the robustness-accurag trade-off. In gen- eral, the standard MoE exhibits higher SA, while the robust MoE exhibits higher RA. The parameter a controls the contribution of the robust MoE to the final prediction. As a increases, the robust MoF contributes more to the final prediction, enhancing adversarial robustness but potentially degrading performance on clean data. We set a 2 0.5 to ensure that the dual-model remains robust.
Table 2. Robustness evaluation of adversarial training (AT) and method (RT-ER) under PGD attacks across different routing strategies. RT-ER consistently achieves higher robust accuracy (RA) than AT under both top-1 and 2op-2 routing strategies.
Top-1 Strategy
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
AT
79.08
53.74
73.85
78.91
RT-ER
77.81
69.09
75.71
72.28
Although the dual-model structure helps balance perfor- mance on clean and adversarial data, it introduces addi- tional complexity in understanding its robustness. To clarify the influence of each parameter on robustness, we derive a certified robustness bound for the dual model FD.
Top-2 Strategy
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
AT 79.20
34.65
74.66
76.09
RT-ER
78.78
70.05
76.2
72.35
We first provide the definition of certified robustness in MoE under the perturbation bounded by l, norm!
Compared to AT, our proposed method, RT-ER, is more effi- cient and adaptable to various routing strategies, as demon- strated in Table 2. We evaluate robustness under both the top-1 and 2 routing strategies, where the router activates one or two experts, respectively. First, AT, as an end-to- end training approach, proves insufficient for robustifying MoE architectures. Even with AT applied, the MoE model achieves only around 54% RA, whereas RT-ER improves RA by more than 16%. Second, RT-ER effectively enhances robustness across different routing strategies, consistently outperforming AT under both top-1 and 2 routing. More-
Definition 5.1. Consider robust MoE FR Rd - R and arbitrary input E Rd. Let arg max; FR(x), with bound E 2 0. We say FR is certifiably robust at x with bound c if for all k + and E IRdd such that ||0||p S c, the following holds:
Fly) (x + 8)
2
((() R (x + 5)
(4)
Whiie our theorems hold for p [1,, oo), we implement p 00 in all our experiments.
5

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
This definition formalizes the certifiable robustness of FR 2; ensuring that the model's top prediction remains con- sistent under perturbations within an Ey-norm ball of radius e, For practical classifiers, the robust margin FR (x 6) FR
neural networks, the robustness of the robust MoE is primar- ily determined by the Lipschitz constants of its individual experts. If any expert lacks robustness, the output of the vulnerable expert will vary significantly when handling ad- versarial inputs, indicating a large Lipschitz constant L.R,- Consequently, the achievable certified robustness bound e for the robust MoE becomes smaller. This implies that, to enhance the robustness of the robust MoE, is essential to robustify each expert. This conclusion provides theoretical support for our proposed RT-ER method.
(x + 6) can be estimated by evaluating the confidence gap between predicted and runner-up classes on a strong adversarial input.
Definition 5.2. A function f Rd + E is Ep Ep-Lipschitz continuous if there exists L E (0, oo) such that for all
x'xx E RR lf((x'- f(x)] < L||x' - xIlp- The Lipschitz
constant of f is defined as
Motivated by Theorem 3.5 in (Bai et al., 2024a), we derive a certified bound to provide a formal guarantee on the dual- model's resistance to adversarial perturbations, stated as follows:
(ipp(f) = (ino(f))=ff(())>> : If(x)) f(x) S L|Ix' - x (5)
Following the definition of e, Lipschitt continuity, we can introduce the following assumption:
Theorem 5.5. For a dual-moded FD(x) comprising a stan- dard MoE (x(x) and a robust MoE Fx(x), with smoothing
Assumption 5.3. Each expert and the router in the robust
parameter o E [ 11, we have that F FD + ō) - y for all 5 E R.d. such that
MoE (x) is Ep-Liischitz continuous, where the experts' Lipschitz continuity is given by
a(F (x) (k) (x)) + -
1 (x + 5) - fly) (x)) S LR,
|01|p
(6)
||81|p S c== min
kuf a Ei 2FR. + +a (L (2) R. + +I
(R)
and the router's 's Lipschitz continuity at i-th output by
ann, (x + 5)- an, (x)] S (rrlsrrllll 16
The proofs for Theorems 5.4 and 5.5 are provided in Appendix A.1. Theorem 5.5 demonstrates that the dual- model's robustness is fundamentallyy rooted in the robust MoE, and increasing the margin Ff" (x) - F(A n (x) could enhance the overall robustness bound. Additionally, each expert in the robust MoE influences both the robust MoE and the dual-model, underscoring the critical role of forti- fying each expert's robustness. This also supports the use of our proposed loss function in Equation (2), designed to reinforce the robustness of individual experts in the MoE.
Our theoretical results are all based on Assumption 5.3, which constrains the applicability of our method. Therefore, it is important clarify the scope under which this assump- tion holds. We identify three representative scenarios where Assumption 5.3 naturally applies. First, in sparse MoE after robust training, robust optimization techniques typically en- courage large margins in expert scores. This leads to a clear separation between the top-n experts and the rest, making difficult for small adversarial perturbations to change the top-n set. As result, the router becomes locally stable and selects the same expert(s) for both clean and perturbed inputs. Second, in dense MoE (Zhang et al., 2024), all ex- perts are activated and the routing function is continuous, which trivially satisfies the assumption. Third, soft MoE (Puigcerver et al., 2023) also employs a continuous routing function with soft assignments, ensuring that small input changes lead smooth changes in routing outputs. In all these cases, the routing behavior is stable under small per- turbations, thus meeting the requirement of Assumption 5.3.
Based on Theorem 5.5, there are three ways to further en- hance the robustness of the dual-model: increasing the value of a, enlarging the margin, and decreasing the Lipschitz con- stant of both the experts and the router. The maximum value (x) is 1. If a falls below 0.5, the numera-
of F(2) (x) - - (* tor in the certified robustness bound becomes negative, ren- dering the robustness radius undefined. Therefore, to ensure that the dual-model maintains a valid certified robustness guarantee, a must lie within the range [0.5, 1]. Furthermore, a higher value of a, which implies a larger contribution of the robust MoE to the final prediction, would result in a decrease in SA, which contradicts the motivation for using the dual-model architecture. As for the Lipschitz constants, they have already been optimized during RT-ER. In sum- mary, the most efficient way to enhance the dual-model's performance is by enlarging its margin. Additionally, a larger margin also leads to better accuracy on clean data.
Theorem 5.4. Under Assumption 5.3, let MR. < 1 be an upper bound on fR. (x) for any input x E Rdd Then the robustness bound for Fπ(x) is:
F(2) (x) FR (x)
c= min KY MR + aR, LR.)
(8)
where is the true label and & represents other classes. From Equation (8), we observe that when the experts are complex
After analyzing the implications of the theorem, it becomes evident that the dual-model must excel as a unified system.
6

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
This observation raises the following research question:
side effect, further improving the dual-model's robustness. Based this analysis, we argue that the dual-model can enhance its performance in both SA and RA through the JTDMoE approach.
(Q3) Can we further enhance performance by em- ploying a joint training strategy?
5.2. Boost Dual-Model with Joint Training
6. Experiments
Inspired by Theorem 5.5 and the insights from Section 5.1, we explore the joint training strategy that considers both the dual-model and single components. By jointly train- ing, we aim to increase the robust MoE's margin, thereby boosting both robustness and accuracy. Traditional adver- sarial training methods focus primarily on the overall model robustness, often overlooking the individual robustness of each MoE component, making these methods less effective in improving MoE-specific robustness.
In this section, we present the effectiveness of our proposed RT-ER and the JTDMMoE approach.
6.1. Experiment Setup
Datasets and Model Architectures. Our experiments use the MoR architecture on CIFAR-10 (Krizhevsky et al., 2009) and TinylmageNet (Deng et al., 2009), with fully connected top-1 router and 4 experts (E 4). We use ResNet18 (He et al., 2016) for CIFAR-10 and pre-trained ViT-small (Liu et al., 2021) for TinyImageNet. Instead of training the dual-model from scratch, we apply the JTD- MoF algorithm to pre-trained models for efficiency, using ST MoE as the standard and RT-ER MoE as the robust MoE, with ranging from 0.5 to 1. More details can be found in Appendix A.2.
To address this gap, we introduce a novel adversarial training framework for the dual-model based on bi-level optimiza- tion. It provides hierarchical learning approach, where the upper-level objective depends the solution of the lower- level problem. Specifically, it is formulated as follows:
min (cE((D(x),)) subject to min Lrobs Ø
(10)
e RR
Robustness Evaluation. We use PGD (Madry et al., 2017) and AutoAttack (Croce & Hein, 2020) to assess model per- formance under adversarial conditions, with e = 8/255 for CIFAR-10 and = 2/255 for TinylmageNet. We train ResNer18- 1--bsedd MoE for 130 epochs on CIFAR-I0 and fine-tune pre-trained ViT-small-ber -aased MoE for 10 epochs on TinylmageNet. A Cyclic Learning Rate strategy (Smith, 2017), starting at 0.0001, and data augmentation (Rebuffi et al., 2021) are used to enhance performance. Evaluation is done using either a 50-step PGD or AutoAttack with the same step size.
where Frob denotes the proposed loss function in Equa- tion (2). Here, the dual-model parameters are divided into variables Ø for the robust MoE and variables 3 for the standard MoE. We term this new approach Joint Training for Mode-Model based on MoE (JTDMoE). JTDMoE pro- motes alignment between the standard MoE and robust MoE, ultimately enhancing the dual-model's performance. The complete training process is outlined in Algorithm 1.
Algorithm 1 The JTDMoE algorithm
1: Initialize: robust MoE parameters O R. standard MoE. parameters s, batch size [. and attack step count K. 2: for iteration t 0, 1.... do
Baseline Methods. For comparative analysis, we define three approaches for the single MoE: (1) ST: Standard train- ing on MoF. (2) AT: Adversarial training on MoE (Madry, 2017). (3) RT-ER: Robust training with expters robustifi- cation on MoE. In addition we cover (4) TRADES (Zhang et al., 2019) and (5) AdvMoE (Zhang et al., 2023) (a new router-expert alternating Adversarial training framework for MoB) in the Appendix A.3.
3: Lower-level e ra-update: update 2n by minimizing Frob using K-step PGD attack on batch B. 4: Upper-level θ, ④ R-update: update O5 and O by minimizing EC (CE(F (x)... y) on batch B.
5: end for
The reason we use this bi-level alternating training approach is to enhance the dual-model's accuracy on clean data while maintaining its robustness as much as possible. For the lower-level optimization problem, we use RT-ER to robus- tify the robust MoE, as the dual-model's robustness derives from the robust MoE (Theorem 5.5). This approach leads to smaller Lipschitz constants for the experts and router, thus guaranteeing the dual-model's robustness. For the upper-level optimization problem, our goal is to enhance the dodee's s SA by minimizing lce (CE(FD(x), v). Ad- ditionally, the margin of the robust MoB can increase as a
6.2. Evaluation of RT-ER
To showcase the improved robustness and training stabil- ity of RT-ER, we compare its SA and RA to those of AT during training. The experimental results of MoE-Resnet18 on CIFAR-10 dataset are illustrated in Figure 2. RT-ER demonstrates greater stability compared to traditional adver- sarial training (AT). In the case of AT-trained MoE, both SA and RA drop significantly- over 20% and 10%, respec- fivel--betee epochs 80 and 90. Although the ViT-small

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Table Evaluation of our proposed methods, RT-ER, Dual-Model (Pretrained), and Dual-Modes (ITDMoF), compared with Standard Training (ST) and Adversarial Training (AT) (Madry et al., 2018) on the CIFAR-10 10 and Tiny Inylmaeeeet datasets. We report standard accuracy (SA), robust accuracy against attacks on the entire model (RA), on experts (RA-E) E), and on the router (RA-R) using 50-step PGD attack (Madry et al., 2018) in the single MoE scenario. The RT-ER method achieves the highest RA, RA-E, and RA-R among single MoE models. For dual-model scenarios, RA-R and RA-E are replaced with robust accuracy against attacks on the robust MoE (RA-RMoE) and on the standard MoE (RA-SMoE), The results show that the Dual-Model improves upon RT-ER in the accuracy-robusteey tradc-off, and our JTDMoE method outperforms the Oude-Model (Pre-trained) under the same smoothing parameter a. TinyImageNet
CIFAR-10
Single MoE Performance
Method
SA(%)
RA(S)
RA-E(S))
RA-R(S)
Method
SA(S)
RA(S)
RA-F(%)
RA-R(S))
ST
92.14
52,54
3.11
54.67
ST
82.05
34.53
29.21
59.00
AT
79.08
53,74
73.85
78.91
AT
70.77
50.94
42.09
83.81
RT-ER
77.81
69.09
75.71
72.28
RT-ER
68.51
56.79
46.32
81.26
Mudl-Model (Pre-trained)
SA(%)
RA(%)
RA-RMoE(S)
RA-SMoE(SS
SA(%)
RA(%)
RA-RMoE(S)
RA-SMoE(9)
0.7
89.81
67.81
79:47
52.93
0.7
80.74
53.11
68.92
47.39
0.8
87.58
68.27
77.25
54,99
0.8
77.89
55,27
65.94
55.16
83.27
70.73
75.03
60.87
0.9
74.2
56.58
61.95
62.80
Dudl-Model (ITDMoE)
0
SA(S)
RA(S)
RA-RMoE(SS)
RA-SMoE((%)
a
SA(%)
RA(S)
RA-RMoE((55)
RA-SMol((9)
0.7
92.29
74.62
85.42
68.73
0.7
84,91
57.39
69.58
53.32
MoE model shows some fluctuations as well, the variation is less pronounced, due to the robustness of the pre-trained ViT- small experts used. The experimental results of ViT-small presented in Appendix A.3. In terms of RA, our method consistently outperforms AT across all epochs, achieving over 15.35% and 5.5% improvements in final performance on CIFAR-10 10 and TinylmageNet, respectively. This sug- gests that RT-ER is a more effective approach for enhancing the robustness of MoE architectures. Overall, our method enables faster, more efficient robustness enhancement for MoE models while reducing SA by < 1.3%.
and ViT-small on TinylmageNet under PGD attack. RT- ER improves RA-E by over 70% and 17% on ResNet18 and ViT-small, respectively, compared to ST, and achieves 5.8% and 4.2% improvements in RA and RA-E over AT using ViT-small. These results show RT-ER's 's effectiveness in robustifying single MoE models. For single MoE, ro- bustness is measured by the minimum of RA, RA-E, RA-R, reflecting the worst-case performance. RT-ER outperforms AT and ST in this regard. We also compare our method with TRADES (Zhang et al., 2019) and AdvMoE (Zhang et al., 2023), and examine the effect of expert number on RT-ER. Details are in Appendix A.3.
70
Standard CendaaddAcuui Accurasy (%)
30
Robust Axurary (%)
AT
RT-ER
6.3. Evaluation of the Dual-Model Based on Pre-Trained MoE
80
30
86
The pre-trained dual-model combines standard MoE and robust MoE with a fixed a to improve clean data perfor- mance. On CIFAR-10, the standard MoE achieves 92. 14% SA and 52.54% R.A, while the robust MoE reaches 77.81 % SA and 69.09% RA. On TinylmageNet, the standard MoE achieves 82.05% SA and 34.53% RA and the robust MoE reaches 68.51 1% SA and 56.79% RA. As a increases, SA de- creases but RA improves, as the robust MoE's contribution grows. When a 1, the dual-model becomes RT-ER, with only the robust MoE. Results for a = 0.7, 0.8, and 0.9 under PGD attack show that increasing a decreases SA (by 6.54% and 13.54% on CIFAR-10 and TinyImageNet, respectively) but improves RA (by 2.92% and 3.47%, %, respectively) and RA-SMoE (by 7.94% and 15.41%).
80
60
35
20
60
(1)
50 Epoch
300
120
140
20
40
6U
50
100
120 140
Epoch
(a) CIFAR-10 SA
(b) CIFAR-IO RA
Figure 2. Performance evaluation of AT MoE and RT-ER MoB models with ResNet18 8 the CIFAR-10 test dataset. We report standard accuracy (SA) and robust accuracy (RA) under a 50-step PGD attack, using models trained with 10-step PGD attack. Our results indicate that RT-ER achieves consistently higher RA and demonstrates greater stability than AT MoE. For comparable analysis using Villsmall, please refer to Appendix A.3.
Specifically, when adversarial perturbations target the ro- bust MoE, they have minimal effect on the standard MoE,
In the Single MoE Performance section of Table 3, we compare ST, AT, and RT-ER using ResNet18 on CIFAR-10
8

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
19.91% % improvement in RA for CIFAR-10 and 1.64% for TinylmageNet. Additionally, JTDMoE outperforms the baseline in both RA-RMoE and RA-SMoE, further validat- ing the effectiveness of aligning the standard and robust MoR models. These improvements are consistent with our theoretical expectations and support Theorem 5.5. Further analysis of margin improvement and additional experiments with different a values can be found in Appendix A.5.
90
Acturnty (%)
Accuracy
60- 60
73
RA-RMMF -MA-SMLF
SA
RA
RA-RMoE RM-SMMF 0.9
(a) CIFAR-I0
(b) TinylmageNet
Margin Comparison. Based on Theorem 5.5, increasing it (x) - -F (x) enhances the robustness of
the margin F F(() the dual-model. To validate this, we compare the margins between JTDMoE and the pre-trained dual-model under 0.7 on the CIFAR-10 dataset. The results are summarized in Table 5.
Figure 3. Performance evaluation of the Dual-Model using pre- trained MoE models. We assess the performance of the Dual- Model, which combines a standard MoF (ST) and a robust MoE (RT-ER) from Table 3. The weighting parameter is incremented from 0.5 to 1.0 in steps of 0.1; at a = 1, the Dual-Model relies exclusively on the robust MoE. All other configurations are consis- tent with those detailed in Figure 2.
Table 5. Margin comparison of JTDMoE and pre-trained dual- model on the CIFAR-10 test dataset. Using the pre-trained dual- model the baseline, we report margin improvements for each class, Results show that JTDMoE consistently improves margins across all classes, supporting Theorem 5.5.
so RA-RMoE decreases as a increases. Conversely, when the standard MoE is attacked, the robust MoE is minimally impacted, leading to an increase in RA-SMoE with a. How- ever, when a shifts from 0.5 to 0.7, RA-SMoE shows only slight improvement, suggesting the robust MoE lacks confi- dence in its predictions and can be influenced by the stan- dard MoE, This motivates our joint training approach to improve dual dual-model performance. Further analysis of the dual-model's performance with pre-trained MoE under Au- toAttack is in Appendix A.4.
CIFAR-10
Class
Airplane
Automobile
Bird
Cat
Deer
Improvement
5.11%
3.495
7.5%
5.55%
10.27%
Class
Dog
Frog 6.46%
Horse
Ship 1.25%
Truck
Improvement
14.42%
4.03%
4.58%
As shown in Table 5, JTDMME improves the margin for every class, with improvements ranging from 1.25% to 14.42%. The class Dog exhibits the most significant margin improvement of 14.42%, followed by Deer with a 10.27% improvement. These results empirically verify Theorem 5.5, demonstrating that margin enhancement contributes to JTD- MoE's superior robust accuracy (RA).
6.4. Evaluation of JTDMMEE
Table 4. Performance evaluation of JTDMoE versus the baseline (Pre-trainee Dual-Modol when attacked by AutoAttack. In the case of a = 0.7, we report standard accuracy (SA), robust ac- curacy (RA), robust accuracy of the robust MoE (RA-RMoE), and robust accuracy of the standard MoE (RA-SMoE) using Au- toAttack (AA) (Crace & Hein, 2020). Results demonstrate that JTDMoE consistently outperforms the baseline across all metrics.
7. Conclusion
CIFAR-10
This work presents a comprehensive framework for enhanc- ing adversarial robustness in MoE models. Our approach is motivated by the key observation that expert networks in MoEs are significantly more susceptible to adversarial perturbations than the router-a a structural vulnerability spe- cific to the MoE architecture. To address this, we propose a targeted robustification strategy with accompanying the- oretical robustness bounds, which adversarially trains an additional expert not selected by the router. Building upon the theoretical tobustness-accuran trade-off, we further in- troduce a dual-model framework (JTDMoE) that integrates standard MoE and robust MoE via bi-level joint train- ing scheme, achieving strong adversarial robustness without compromising standard accuracy. Extensive experiments across diverse models and datasets validate the effectiveness and scalability of our proposed methods.
Method SA(%)
RA(%)
RA-RMoB(S() RA-SMoB(S)
Pre-trained
89.81
33.64
34,20
14.34
JTDMoE
92.29
54.55
51.13
45.55
TinyImageNe
Method
SA(%)
RA(S)
RA-RMoE(SS)
RA-SMoE(%)
Pre-trained
80.74
51.27
56.25
26.88
JTDMoE
84,91
52.91
60,98
28.82
To compare with the results in Section 6.3, we use the same standard MoE and robust MoE models for a 0.7. As shown in Table 3 and Table 4, our JTDMoE algorithm sig- nificantly improves the performance of the pre-trained dual- model across all evaluation metrics. Specifically, it increases SA by 2.48% and 4. 4.17%, and RA by 6,81 % and 4.28% un- der PGD attacks for CIFAR- 10 and Tiny ImtgeNet,,, respec- tively. When evaluated with AutoAttack, JTDMoE achieves

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Impact Statement
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-244222 248-255. leee. 2009.
This paper presents work aimed at advancing the field of Machine Learning, specifically in improving the adversarial robustness of Mixture of Experts (MoE) models. Our contri- butions enhance the security and reliability of MoEs, which are widely used in large-scale and specialized Al applica- tions. While this research primarily focuses on technical advancements, its broader implications include improving the deployment of MoE-based models in safety-critical do- mains. However, we not identify any specific societal consequences that must be highlighted here.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y.. Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of- experts. In International Conference on Machine Learn- ing. pp. 5547-5569. PMLR. 2022.
Fedus, W. Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Acknowledgement
This work is supported by the NSF under Grants 2246157 and 2319243. We are thankful for the computational re- sources made available through NSF ACCESS and Argonne Leadership Computing Facility.
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, L., Beam, L.,, and Kohane, S. Adversarial attacks on medical machine learning, Science, 363(6433)
2019,
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv:1412.657 1412.6572, 2014.
References
Alam, M., Datta, S., Mukhopadhyay, D., Mondal, A., and Chakrabarti, P. P. Resisting adversarial attacks in deep neural networks using diverse decision boundaries. arXiv preprint ar wwXxi:2208.086999 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bai, Y., Anderson, B. G., Kim, A., and Sojoudi, S. Improv- ing the accuracy-robustmess trade-off classifiers via adaptive smoothing. SIAM Journal on Mathematics of Data Science, 6(3):788-81) 814, 2024a.
He, S., Cheng, Q., Huai, Y., Zhu, Z., and Ding. J. Mixture- offexperts for semantic segmentation of remoting sensing image. In International Conference Image Processing and Artificial Intelligence (ICIPAI 2024), volume 13213, pp. 478-483. SPIE, 2024.
Bai, Y., Zhou, M., Patel, V. M., and Sojoudi, S. Mixednuts: Training-free robusaeyssobustness balance via nonlin- carly mixed classifiers. arXiv preprint arXiv:2402.02263, 2024b.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. leee, 2017.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Chen, B., Chen, K., Yang, M., Zou, Z., and Shi, Het- erogeneous mixture of experts for remote sensing image super-resolution. IEEE Geoscience and Remote Sensing Letters, 2025.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
Lewis, M., Bhosale, S.,, Dettmers. T., Goyal, N., and Zettle- moyer, L. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 6265-6274, PMLR, 2021.
Co, K. T., Martinez-Rego, D., Hau, Z., and Lupu, E. C. Jacobian ensembles improve robustness trade-offs to ad- versarial attacks. In International Conference on Artificial Neural Networks, pp. 680-691. 68piner,,, 2022.
Li, W., Deka, D., Wang, R., and Paternina, M. R. A. Physics- constrained adversarial training for neural networks in stochastic power grids. IEEE Transactions Artificial Intelligence, 2023.
Croce, F. and Hein, M. Reliable evaluation of ad adversarial robustness with an ensemble of diverse parameter-fin attacks. In International conference on machine learning, pp. 2206-2218 PMLR, 2020.
Liu, X., Cheng, M., Zhang, H., and Hsich, C.-J. Towards robust neural networks via random self-ensemble. In
10

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
Proceedings of the european conference on computer vision (ECCV), pp- 369-385, 2018.
Videau, M., Leite, A., Schoenauer, M., and Teytaud, O. Mixture of experts in image classification: What's the sweet spot? arXiv preprint arXiv:2411.1832. 2024.
Liu, Y.. Sangineto, E.., Bi, W., Sebe, N., Lepri, B., and Nadai, M. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 34:23818-23888 2021.
Wang. R., Xu, K., Liu, S., Chen, P.-Y,, Weng. T.-W., Gan, C., and Wang, M. On fast adversarial robustness adap- tation in model-agnostic meta-learning- In International Conference on Learning Representations, 2020a.
Madry, A. Towards deep learning models resistant adver- sarial attacks. arXiv preprint arXiv: 3xxiv:7006060833. 2017.
Wang, R., Xu, K., Liu, S., Chen, P.-Y., Weng. T.-W., Gan, C., and Wang, M. On fast adversarial robustness adap- tation in model-agnostic meta-learning. In International Conference on Learning Representattonss 2021.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. stat, 1050(9), 2017.
Wang, R., Li, Y., and Liu, Exploring diversified ad- versarial robustness in neural networks via robust mode connectivity. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 2345-2355 2023.
Madry, A., Makelov, A., Schmidt, L.,, Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018.
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Wang. X., Yu, F., Dunlap. L., Ma. Y.-A., Wang, R., Mirho- seini, A., Darrell, T., and Gonzalez, E. Deep mixture of experts via shallow embedding. In Uncertainty in artificial intelligence, pp. 552-562. PMLR, 2020b.
Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. In International Confer- ence Learning Representations, 2020. URL https:
Patterson, D., Gonzalez, J., Le. Q., Liang, C., Munguia, L.- M., Rothchild, D., So, D., Texier, M., and Dean, Car- bon emissions and large neural network training. arXiv preprint ar arXiv:2100 10350, 2021.
/ //openreview... net
Xue, F., Shi, Z., Wei, F., Lou, Y., Liu, Y., and You, Y. Go wider instead of deeper. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pp. 8779- 8787, 2022.
Puigcerver, J., Jenatton, R., Riquelme, C., Awasthi, P., and Bhojanapalli, On the adversarial robustness of mixture of experts. Advances in Neural Information Processing Systems, 35:9660-995 9677,, 2022.
Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00 00951, 2023.
Yuksel, S. E., Wilson, N., and Gader, D. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177-1199 2012.
Rebuffi, S.-A., Gowal, S., Calian, D. A.. Stimberg, F., Wiles, O., and Mann, T. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXhv:2103.01946 2021.
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and
Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Susano Pinto, A., Keysers, D., and Houlsby, N. Scaling vision with sparse mixture of experts. Ad- vances in Neural Information Processing Systems, 34: 8583-8595, 2021.
Zhang, H., Zhou, Y., and Wang, G.-H. Dense vision trans-
former compression with few samples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15825-15834, 2024.
Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P.-Y,, P.-Y.,Chani Chang, S., Wang, Z., and Liu, S. Robust mixture- of-expert training for convolutional neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 90-101, 2023.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gater mixture-of-expeer layer. arXiv preprint arXiv: /001: 066588, 2017.
Smith, L. N. Cyclical learning rates for training neural net- works. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464 472. IEFE, 2017.

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
A. Supplementary Material
Bounding A: Combining the bounds for A and Ag,
have:
In this supplementary material, we provide the proofs of Theorems 5.4 and 5.5 Section A.I. Additional experimen- tal results are organized as follows: single MoE experiments reported in Section A.3, dual-model experiments based on pre-trained MoEs are presented in Section A.4, and JTD- MoE experiments are detailed in Section A.5. In addition, Section A.6 presents RT-ER experiments conducted using large model (ViT) on the large-scale ImageNet dataset.
A Σ (TR,MRR + aR, (R)
By defining the total Lipschitz constant Liotal
Lattal = Σ (rn,MR MR. + an, (x)
A.1. Proof of Key Theorems
the upper bound on the change in the final output becomes:
Proof of Theorem 5.4: When the input is perturbed from x to x + 5, the change in the final output of the MoE can be expressed as:
F x + 6)
RR. (xx - Eeooaal10)
To ensure the prediction remains unchanged under perturba- tion ò, the change in output must not exceed the classifica-
( R (X- + 8) -F (x)
tion margin m mint poy FR (x) [.(k) (x). Thus:
Σ
aRR (x + + R2 (x 6 -RR (x) p(u) (2)((()))))) (x)
Liotal | S
|| S m
We decompose A into terms:
Ltotal
1 Σ (x 60 - aan.(x)) f(0)) (x 6)
The overall robustness bound e therefore:
m
A2 Σ aRR (x) (Fhe (x+ +6) - (x))
(r MR, + an, Lx)
F (x) FR (x) = =min kgy (. MR. + aR (x)
A == A= +
By this decomposition, A captures the change due to the router's output, while Ae represents the change due to the experts' outputs. To derive certified bound for the overall MoE, we need to bound both A₁ and Az.
This concludes the proof.
Proof of Theorem 5.5: To derive the certified bound for the dual-model, we utilize Theorem 3.5 from (Bai et al., 2024a), which relates the robustness bound of dual-model to the properties of the robust model. Since a is assumed to be in [t) 1]. let ō E I2 satisfy |/01p S r (f)..p Further- more, for é [cl My). we have
Bounding A₁:
A < Σ aR. ()) - -aR, (x)1 -T. (x + 6)
Σ Tn, T MM.
ao ro 8)
h, + 6) = o
h,(2) + hy + 8) - -oohy(2) + a h4(2)) o hi(r + 6)
where MR. S. 1 is an upper bound on f(y) (x 8) an up- per bound fR. (x) for any inputs and rn, is the Lipschitz constant of the router aR;
2 200h((z) - oo ha(x) Lip, (a h2y) 1|01pp
-Lip, h:) ||0||
2 (o)g(x) a h4(x)
Bounding A2:
(Lip, (oo + Lip, (o o hi)) ríp (z)
2 Σ an (x) [H) + // (x)l
< Σ aR. (x)
(11)
where a is the parameter controlling the contribution the robust model to the dual-model, a h represents the out- put of the robust model for the true class y and Lip, (a o ha)
where I.n, is the Lipschitz constant of the expert
((0)
12

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
denotes the Lipschitz constant of the robust model for class
Standord Accumry (%)
Then, the robustness bound of the dual-model can be ex- pressed as:
Robust courccy
12
a (a o hy(()) iy a (Lipp (a o hyy + Lip, (o o h:))
h₂(x))) + -
|00| 2 min
Epoch
'Epocti
From the previous proof, we already derived the Lipschitz constant for the robust MoE. Using this result, we can now represent the robustness bound E for the dual-model as:
(a) TinylmageNet SA
(b) TinyImageNet RA
Figure 4. Performance evaluation of AT MoE and RT-ER MoE models with ViT-small on the TinylmageNet test dataset. We report standard accuracy (SA) and robust accuracy (RA) under 50-step PGD attack, using models trained with a 10-step PGD attack. Our results indicate that RT-ER achieves higher RA and demonstrates greater stability compared to AT MoE.
a(Fle) (x) - F (x)) ---1
= min k/y
Σi 2TR.. a (x) (2) +LR))
This concludes the proof.
A.2. Model Architecture Details
Smooth Attack. Throughout this paper, we adopt the top- strategy for expert selection, which is inherently non- differentiable and may cause standard gradient- based at- tacks fail. To enable effective robustness evaluation, we employ adaptive attacks by replacing the top-1 selection with a dense smooth approximation. Specifically, pute weighted average of expert outputs using the routing weights and generate perturbations 5 based on this smoothed output. This approach ensures that expert selection remains differentiable, allowing gradients to propagate. The results, presented in Table 6, demonstrate that our method, RT-ER, continues to outperform ST and AT RA.
Following several recent works (Videau et al., 2024; Chen et al., 2025; He et al., 2024), the default setting is a single MoE layer as a replacement for the classification layer. We further conduct experiments using the architecture proposed by Riquelme et al. (2021), which inserts MoE layers in place of the dense MLPs within transformer blocks. Af- ter standard training, the MoE model achieves 90.35% SA, 38.02 38.02% RA, 32.16% RA-F, and 64.97% RA-R. These re- sults further support our observation that expert networks are generally more vulnerable adversarial perturbations than the router. This vulnerability likely stems from the fact that expert networks are deeper and more complex than the router, making them inherently more susceptible to such attacks. Regardless of the specific MoE architecture used, this trend remains consistent: the expert networks exhibit greater vulnerability due to their complexity.
com-
Table 6. Robustness evaluation of standard training (ST), adver- sarial training (AT), and our method (RT-ER). We adopt dense smooth approximation to ensure differentiable expert selection. The experiment is conducted on CIFAR-10 using an MoE model with ResNet18 experts.
A.3. Additional Experiments on Robust MoE
CIFAR- R- 10 (Smooth Attack)
Performance of RT-ER on TinyImageNet Dataset. For the TinylmageNet dataset (Deng et al., 2009), we adversari- ally trained an MoE model using ViT-small as the experts. The experimental setup follows the same settings outlined in Section 6. The RA and SA performance curves shown in Figure 4.
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
ST
92.14
46.02
3.11
54.67
AT
79.08
51.03
73.85
78.91
RT-ER
77.81
68.96
75.71
72.28
Comparison with Trades and AdyMoE. We compare our method, RT-ER, with TRADES (Zhang et al., 2019) and AdyMoE (Zhang et al., 2023) using an MoE archi- tecture based on ResNet18 on the CIFAR-10 dataset. All experiments are conducted under the same settings for fair comparison. The results for standard accuracy (SA) and robust accuracy (RA) are presented in Figure 5.
RT-ER achieves noticeably higher RA and demonstrates more consistent performance compared to the AT MoE model. While the AT model exhibits some fluctuations in performance, these are less pronounced than those ob- served on the CIFAR-10 dataset. This improved stability can be attributed to the use of pre-trained ViT-small model, which inherently possesses adversarial robustness due to its training on a large and diverse dataset. In summary, RT-ER improves RA by over 5.8% while incurring only a mod- est 2.2% decrease in SA, highlighting its effectiveness in enhancing the robustness of MoE models.
All methods are trained from scratch to ensure consistency. While TRADES and AdvMoE exhibit performance fluc- tuations similar to those observed in standard adversarial training, RT-ER demonstrates significantly improved stabil-
13

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
configurations of the MoE architecture.
Sandard Accunity (1%
Robost Accumity (%%
Table 8. Robustness evaluation of RT-ER with varying numbers of experts E on the CIFAR-10 10 dataset. Increasing E enhances model capacity, leading to improvements in both standard accuracy (SA) and robust accuracy (RA).
Tinces
Vizdee XT-ER AddMMM
40
66
66 Epoch
100
120
14
CIEAR-I0
Epoch
E
SA(%)
RA(%)
RA-E(%%
RA-R((%)
(a) SA
(b) RA
3
75.85
69.03
74.79
72.42
4
77.81
69.09
75.71
72.28
5
79.77
70.00
76.51
71.86
Figure 5. Performance comparison of TRADES, AdvMoE, and RT-ER with & on the CIFAR-10 test dataset. SA and RA are evaluated under a 50-step PGD attack, using models trained with 1t-step PGD attack. RT-ER achieves higher RA and exhibits greater stability compared to TRADES and AdvMoE. Numerical analysis comparisons are presented in Table 7.
The Impact of Different 3. Recall that the loss function Lrob used in the RT-ER method is defined as:
E
ity. Although RT-ER achieves marginally higher SA SA-by by less than 1% compared to TRADES, it achieves notable 10.42% improvement in RA,, underscoring its effectiveness in enhancing robustness. In conclusion, RT-ER is the supe- rior choice for robustifying the MoE layer, delivering both improved robustness and more reliable performance.
max (F |8
E (x (x)).
We examine the impact of varying B on the model's perfor- mance, setting /3 to 1, 3, 6, and 9. The results are summa- rized in Table 9.
Table 9. Performance evaluation of RT-ER with different values of B3 in Crob. Metrics reported are standard accuracy (SA), robust accuracy on the entire model (RA), robust accuracy on experts (RA-E), and robust accuracy on the router (RA-R) under a 50-step PGD attack. Results show the influence on robustness and standard accuracy.
Table Performance comparison of TRADES, AdvMoE, and RT- ER with ResNet18 8 on the CIFAR-10 test dataset under PGD and AutoAttack. As AdvMoE is specifically designed for CNN-based MoE, this experiment is conducted using MoE with ResNet18 experts on the CIFAR-10 10 datasct.
CIFAR-10 (PGD)
CIFAR-10
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
3
SA(%)
RA(%)
RA-E(%)
RA-R(%)
Trades 83.81
52.54
72.72
76.63
81.68
64.13
68.67
69.64
AdvMoE
76.83
58.67
73.79
74.03
3
79.91
65.92
70.75
72.05
RT-ER
77.81
69.09
75.71
72.28
6
77,81
69.09
75.71
72,28
CIFAR-10 (AA)
9
10
10
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
Trades
83.81
38.90
54.72
56.84
AdvMoE 76.83
44.17
54.62
55.06
As shown in the table, increasing 3 leads to a consistent de- crease in SA and corresponding increases in RA, RA-F, and RA-R, reflecting improved robustness. However, for 3 = 9, all metrics collapse to 10%, indicating random guessing by the model. This phenomenon arises because the second term in Crob, Σe= L (x + 8), fr. (x)). dominates the loss at this value of B. This term forces experts ap- proximate their adversarial outputs fR. (x++) to their initial outputs fn, (x). Since the MoE is trained from scratch, its initial outputs are meaningless at the beginning of training. Consequently, the model fails to learn effectively, resulting in degenerate performance.
RT-ER
75.92
54.36
57.86
54.84
The Impact of Number of Experts F.. We investigate the impact of the number of experts on RT-ER in Table 8.
Specifically, we consider three cases: E E {3, 4)5,, us- ing the default 10 settings, with the only change being the number of experts. As the number of experts (E) increases, the capacity of the MoE architecture grows. Af- ter 130 epochs of training, we observe an approximate 2% improvement in SA, while RA shows only a slight improve- ment. This is attributed to the sparse MoE and top-1 routing strategy, where only one expert (ResNet18) 8) is activated for each input image. Since each expert is robustified during training and capable of handling classification tasks inde- pendently, the SA improvement is more noticeable. These results also demonstrate that RT-ER adaptable to various
A.4. Experiments with Pre-trained Dual-Model
In this section, we evaluate the dual-model's 's performance us- ing pre-trained MoE models under AutoAttack. The results are shown in Figure 6. For CIFAR-10, the attack strength
14

---

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach
set 8/255, while for TinylmageNer, it is 2/255.
We 50-step AutoAttack with step size of 2/255.
Stanfard Accuracy (%)
Jolbus Accurasy (%)
10.5
82
05
Acosenty (96)
Acturae (8)
=07 =0.
104 AiatiA
Epoch
Epoch
10
(a) SA
(b) RA
Del Modd Alet Suadid Seci
37
66
07
03
0 1
0.5 1 aasl
Figure 7. Performance evaluation of JTDMoE under different val- of the CIFAR-I0 test dataset. We the dual-model performance at a -==0. as the baseline, depicted by green squares in this figure. Standard accuracy (SA) and robust accuracy (RA) under a 50-step PGD attack are reported, with models trained us- ing a 10-step PGD attack. The results demonstrate that JTDMoE maintains effectiveness across different values.
(a) CIFAR-10
(b) TinylmageNet
Figure 6. Performance evaluation of the Dual-Model using pre- trained MoE models. The lack, combining standard MoE (ST) and robust MoE (RT-ER) from Table 3. The parameter a is varied from 0.5 to 1.0 in steps of 0.1; the Dual-Model relies entirely on the robust MoE.
evaluated under AutoAt-
geNet dataset (Deng al., 2009). The experimental setup follows the protocol described in Section 6, and the results are summarized in Table 10. RT-ER consistently outper- forms both AT and TRADES across all evaluation met- rics Mics-SA (+8%), RA (+12%), RA-E (+1%), and RA-R (00%))ddmmmnsrrtiiigg its effectiveness and efficiency in improving the robustness MoE models at scale.
The results indicate that the dual-model's SA decreases while RA increases as a grows. This trend aligns with structure of the dual-model: as the robust MoE's con- tribution increases (i.e., higher a). RA improves because the robust MoE dominates the final predictions, mitigating the effect of adversarial perturbations the standard MoE. These findings are consistent with the model performance under PGD attacks, as shown in Section 6.3.
Table 10. Robustness evaluation of TRADES, and RT-ER. RT- ER achieves substantial improvement of approximately 12% in RA and SA compared to conventional adversarial training, The evaluation is conducted on the ImageNet dataset using MoE model with ViT-based experts.
A.5. JTDMoE Experimental Results
Effectiveness of JTDMoE Under Different a. We
ImageNet
demonstrated the effectiveness of JTDMoE when = 0.7 in Section 6. Here, we evaluate the performance the dual-model for different values a = 0.5, 0.6, 0.7, 0.8 to show that JTDMoE remains effective across range of a. The standard MoE and robust MoE utilize ResNet18 as the experts. All other settings remain consistent with those in Appendix A.3. The results are illustrated in Figure 7.
Method
SA(%)
RA(%)
RA-E(%)
RA-R(%)
AT
60.32
44.64
43.06
70.24
TRADES
61.94
45.54
43.75
70.37
RT-ER
68.38
56.16
44.99
70.82
From Figure 7, we observe that SA consistently im- proves under different a during the training process, while RA remains stable or improves slightly. When a is small--ndiccatin that the robust MoE contributes less to the dual-model-the RA continues to increase, showcasing the model's potential for robustness. These results highlight JTDMoE's ability to improve the dual-model's 's performance, particularly in terms of SA, while maintaining robust accu- racy. This demonstrates the effectiveness and adaptability of JTDMoE across varying values a.
A.6. ViT Experimental results on ImageNet
Performance of RT-ER on the ImageNet Dataset. To evaluate the scalability of RT-ER to large-scale models and datasets, we conduct experiments using VIT on the Ima-
15


Reviews:
[{"review_id": "16iYxLv4gh", "reviewer": "Reviewer_K985", "summary": "The paper studies the adversarial robustness of mixture-of-experts (MoE) models in detail, investigating the susceptibility to adversarial attacks to both the router and the experts modules. Under some assumptions, the paper proofs that the perturbation on the entire model can be decomposed as the sum of the perturbations on the router inputs, and the experts inputs. Based on these, one can bound the Lipschitz constant of the entire MoE model. \n\nThe authors suggest the use of a dual model to improve the adversarial robustness, which is comprised by a model trained purely on a classification task, and a model trained with adversarial training. The paper shows that once can derive a certified robustness bound from the dual-model as well.\n\nThe paper presents experimental results on CIFAR-10 and TinyImageNet, showing that the proposed method performs better than vanilla adversarial training, using a ResNet-based MoE model.\n\n### Update after the rebuttal\nI thank the authors for clarifying in which scenarios Assumption 5.3 is realistic. I agree with their analysis in that regard.\n\nRegarding my two other concerns, the authors mention that they follow practices used in previously published papers (i.e. the choice of TinyImageNet, and the use of MoE models with as single MoE layer at the end). This argument is a bit weak, in my opinion. The fact that previous works have been published with a suboptimal evaluation method (in my humble opinion), doesn't mean that we should continue doing that. Nevertheless, the authors ran some additional experiments on the full ImageNet and using full MoE models (i.e. replacing the MLP in ViT with a MoE model), strengthening the evidence supporting their proposed method.\n\nGiven this, I have increased my score and I'm (slightly) leaning towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback and the opportunity to clarify and strengthen our submission. Below, we respond to each of your concerns in detail.\n\n## A concern regarding assumption 5.3 not holding in real practice\n\nWe would like to clarify that Assumption 5.3 is reasonable and holds in several practical scenarios. Specifically, we outline three cases where this assumption is satisfied:\n\n​\t1.**Sparse MoE after robust training**: Robust training techniques typically encourage large expert score margins. In our setting, a large margin in the expert scores implies that the differences between the scores of the top-k experts are well separated from the rest. This inherent separation makes it difficult for adversarial perturbations to change the top-k set within a realistic $\\delta$-ball. In this situation, the router becomes locally stable and consistently selects the same expert(s) for both the clean input $x$ and the perturbed input $x + \\delta$, thereby satisfying Assumption 5.3.\n\n​\t2.**Dense MoE**: In a dense MoE [6], all experts are activated for each input, and the routing function is continuous. Thus, Assumption 5.3 naturally holds.\n\n​\t3.**Soft MoE**: Similarly, soft MoE [5] use a continuous routing function with soft assignments. This inherent continuity ensures that small changes in the input lead to small changes in the routing outputs, thereby maintaining stability and satisfying Assumption 5.3.\n\nWe have added additional discussion and clarification in the revised version of the manuscript to further address this point.\n\n## Require large model (ViT) and dataset (ImageNet)\n\nWe would like to clarify that we have already included results using a ViT model on the TinyImageNet (see **Table 3** and **Figure 3**). Our work focuses on investigating the robustness of MoEs, and we adopt the ViT + TinyImageNet setting to evaluate our method. This setting has also been adopted in prior studies on the adversarial robustness of MoE, such as [7], making it a widely accepted benchmark for robust MoE research.\n\nTo further strengthen our empirical evaluation, we now additionally provide results for our proposed RT-ER using a ViT model on ImageNet. In particular, we observe a substantial improvement of approximately **12% in RA and 8% SA** compared with conventional adversarial training. The results are summarized below:\n\n| **Method** | **SA(%)**      | **RA(%)**      | **RA-E(%)**    | **RA-R(%)**    |\n| ---------- | -------------- | -------------- | -------------- | -------------- |\n| RT-ER      | 68.38$\\pm$0.17 | 56.16$\\pm$0.14 | 44.99$\\pm$0.16 | 70.82$\\pm$0.13 |\n| AT         | 60.32$\\pm$0.15 | 44.64$\\pm$0.14 | 43.06$\\pm$0.17 | 70.24$\\pm$0.14 |\n| Trades     | 61.94$\\pm$0.12 | 45.54$\\pm$0.17 | 43.75$\\pm$0.13 | 70.37$\\pm$0.11 |\n\n## A Key Distinction in MoE Architecture in Experimental Designs\n\nThank you for your comments. We would like to clarify that our primary focus is to investigate the *fundamental vulnerability* of MoE architectures to adversarial attacks in the context of **image classification**. Our architectural setup aligns with several recent works that adopt a similar design [2-4], allowing us to pinpoint the vulnerabilities of the experts without interference from additional layers. Importantly, the RT-ER and the JTDMoE strategies we proposed are designed to be agnostic to the underlying MoE integration scheme. We would also like to point out that the observation “the expert modules are more susceptible to adversarial attacks” is *independent of the specific placement* of the MoE layers—whether integrated within the classification head or embedded inside transformer blocks.\n\nTo support this claim, we have verified that our findings remain valid under the architecture proposed by Riquelme et al. [8], which replaces the MLP block with a MoE layer. After standard training, the MoE achieved **90.35% SA, 38.02% RA, 32.16% RA-E, and 64.97% RA-R**. Expert networks, being deeper and more complex than the router, are inherently more vulnerable to adversarial perturbations. Even though the experts do not directly produce the final output, their susceptibility can still impact the overall model, as adversarial effects propagate through subsequent layers. These results confirm that the insights and conclusions derived from our analysis are broadly applicable across diverse MoE integration strategies.\n\n[1] Puigcerver et al. On the adversarial robustness of mixture of experts.\n\n[2] Videau et al. Mixture of Experts in Image Classification: What's the Sweet Spot?\n\n[3]Chen et al. Heterogeneous Mixture of Experts for Remote Sensing Image Super-Resolution.\n\n[4] He et al. Mixture-of-experts for semantic segmentation of remoting sensing image.\n\n[5] Puigcerver et al. From sparse to soft mixtures of experts.\n\n[6] Zhang Dense vision transformer compression with few samples.\n\n[7] Lin et al. Towards Robust Vision Transformer via Masked Adaptive Ensemble.\n\n[8] Riquelme et al. Scaling vision with sparse mixture of experts.", "claims_and_evidence": "The paper successfully identifies how each component of an MoE model affects the robustness, theoretically (deriving an upper bound for each term) and empirically (observing that the models used in the experiments are more susceptible to attacks in the expert modules). \n\nThe proposed dual-model approach, and the joint training strategy to train them, obtain very successful results when compared to standard adversarial robustness training, both in terms of accuracy on the clean data and under adversarial attacks.", "methods_and_evaluation_criteria": "The evaluation criteria follows the standard one used in adversarial robustness works. Namely, studying the accuracy both on clean data and adversarially constructed inputs with standard methods such as PGD and AutoAttack.\n\nHowever, see the comments in \"Experimental Designs Or Analyses\" regarding my concerns with experimentation and the conclusions that one can draw from these.", "theoretical_claims": "I checked the proofs for both theorems 5.4 and 5.5. I could not spot any problem in the theorem themselves, but I do have a concern regarding one of the assumptions. In particular, the part relative to the router in Assumption 5.3 doesn't hold in real scenarios (including the ones in the paper's experiment), I believe.\n\nThe assumption states that the $i$-th output of the router is Lipschitz continuous with  $\\|a_{R_i}(\\mathbf{x} + \\mathbf{\\delta}) - a_{R_i}(\\mathbf{x})\\| \\leq r_{R_i} \\|\\|\\mathbf{\\delta}\\|\\|_p$. However, this assumption is not true for typical sparse MoEs, where $a(\\mathbf{x})_i = \\text{top}_k (\\text{softmax}(W \\mathbf{x}))$, at least not for all $\\mathbf{x}$.\n\nConsider a top-1 MoE, and a point $\\mathbf{x}$ that lies at a distance $\\leq \\epsilon$ of the decision boundary between the top-2 experts. Without loss of generality, let's assume that these are experts with indices 1 and 2, respectively, and let's assume that the top-1 routing weight is $a(\\mathbf{x})_1 = \\rho$. Due to discontinuity of standard sparse MoEs, no matter how small I pick $\\epsilon$, there exists a perturbation $\\mathbf{\\delta}$ such that:\n- $a(\\mathbf{x} + \\mathbf{\\delta})_1\\approx \\rho$, and $a(\\mathbf{x} + \\mathbf{\\delta})_2 = 0$\n- $a(\\mathbf{x} - \\mathbf{\\delta})_1 = 0$, and $a(\\mathbf{x} - \\mathbf{\\delta})_2  \\approx \\rho$\n\nNote that $\\rho$ can change depending on the exact definition of the router, but in any case it will be $\\rho > \\frac{1}{n}$, where $n$ is the total number of experts. Thus, completely unrelated to (and much bigger than) $\\epsilon$.\n\nEssentially, $\\mathbf{\\delta}$ is a perturbation that moves the input perpendicular to the boundary between expert 1 and expert 2, closer to one or the other depending on the direction.", "experimental_designs_or_analyses": "The experimental design is appropriate for the paper, in terms of evaluation protocol. \n\nHowever, the choice of the models used (a small ResNet with a single MoE linear layer in the classification head), and the datasets in which the experiments were conducted (CIFAR10 and TinyImageNet) raises some concerns about the relevance of the experiments.\n\nCompare this with (for instance) Puigcerver et al. (2022), which the paper refers to as one of the first works studying the adversarial robustness of MoE models, that used MoEs based on Vision Transformers, and trained on full ImageNet.\n\nIn addition to size of the models and datasets itself, there's a key distinction: this work uses only a single MoE layer _as a replacement of the classification layer_, while modern state-of-the-art transformer-based MoE models place MoEs as a replacement of the dense MLPs inside the transformer blocks, and not at the classification layer. This might shift completely one of the key observations from the paper: the fact that the expert modules are more susceptible to adversarial attacks, might be due to the fact that they directly affect the model's output.", "relation_to_broader_scientific_literature": "The key contributions are very relevant to the MoE community in computer vision and adversarial robustness. The paper does a good job referencing the main relevant papers from each topic, and making the relationship among them clear.", "essential_references_not_discussed": "Given my concerns about the (in)validity of the theoretical assumptions with non-continuous routers, I would suggest the authors refering to papers that try to amend this, such as \"From Sparse to Soft Mixtures of Experts\" by Puigcerver et al (2023), and the more recent \"ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing\" by Zhang et al. (2024).\n\nIt's probably too much to ask, but it would be interesting to repeat some of the analysis with one of these MoE approaches, to check if the findings still hold.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well motivated, structured, and written.\n- The theortetical proofs in the appendix easy to follow.\n- The proposed methods seem quite easy to implement, which can potentially widen the adoption of the proposed approach.\n- The paper contains ablation experiments tuning the hyperparameters of the dual-model strategy.", "other_comments_or_suggestions": "The tables feel a little bit too clutered with some many horizontal / vertical bars. I would suggest tidying them up a bit.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2tzCyzj8o", "reviewer": "Reviewer_vMs4", "summary": "The paper proposes: \n1. A loss function that specifically enhance the robustness of experts in MoE architecture; \n2. A dual-model strategy for robustness-accuracy trade-off; \n3. A joint-training strategy for dual-model. \nto enhance the adversarial robustness of MoE model.\nExperiments are conducted on CIFAR10 and TinyImageNet.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n## Contributions of Our Method\n\nWe appreciate the reviewer's comment. We would like to point out that aligning outputs with KL-loss and mixing clean and adversarial outputs are classical loss design techniques used in most of the adversarial training papers. We would like to clarify that our contributions are novel and **specifically tailored to the unique challenges of MoE architectures**. In particular:\n\n**MoE-Specific Vulnerability Analysis:** Our work is grounded in the empirical and theoretical observation that expert networks in MoEs are significantly more susceptible to adversarial attacks than the router. This insight is critical because, unlike standard neural networks, the MoE framework relies on dynamic routing where the experts' outputs directly affect the final prediction. Our contribution lies in isolating and addressing this vulnerability, which is not encountered in traditional architectures.\n\n**RT-ER – A Targeted Robustification Strategy:** While methods such as aligning outputs with KL divergence have been explored in classical settings, our RT-ER method is designed specifically for MoEs. For each input, we robustify an expert that is not selected by the router, and because the router's selection changes during training, this process iteratively reinforces the robustness of the entire expert network. This dynamic and efficient strategy is uniquely adapted to the MoE structure and its operational dynamics, setting it apart from standard adversarial training methods.\n\n**Dual-Model Strategy (JTDMoE):** Our dual-model strategy is not a mere combination of existing techniques but a novel design that integrates a standard MoE and a robust MoE in a unified framework. This approach allows us to achieve a favorable balance between clean and robust accuracy while maintaining efficiency. Unlike typical ensemble methods in classical networks, our dual-model is trained with a specifically designed bi-level training process and is accompanied by a rigorous theoretical robustness bound (Theorem 5.5) that quantifies how the interplay between the two models contributes to overall robustness.\n\n**Theoretical Contributions:** We provide new certified robustness bounds for both the full MoE model and the dual-model setup, offering insights into how individual components (especially the experts) impact overall robustness. These theoretical guarantees are tailored to the dynamics of MoEs and serve as a foundation for our proposed training strategies, further reinforcing the novelty of our contributions.\n\nIn summary, our method addresses core challenges in MoE training (namely robustness, performance, and efficiency) by proposing tailored strategies (RT-ER and JTDMoE) and accompanying theoretical guarantees that are specifically designed for the MoE setting. These contributions collectively advance the state-of-the-art in adversarial robustness for MoEs and are not simply an assembly of existing techniques from classical neural networks.\n\n## Results on Large-Scaled Datasets\n\nThank you for your comments. Our work primarily focuses on investigating the robustness of MoE. To evaluate our proposed method, we adopt the **ViT + TinyImageNet** and **ResNet + CIFAR-10** settings—both of which are widely used benchmarks in robust MoE research. Notably, this evaluation setup has also been employed in recent studies, such as Lin [1] and Zhang [2], further supporting its relevance and acceptance within the community. Although our current experiments focus on CIFAR-10 and TinyImageNet, our method is designed to be scalable. The computational cost of robustifying one additional expert is minimal relative to the overall training budget, ensuring that the approach can be applied efficiently to larger datasets and more complex models.\n\nTo further strengthen our empirical validation, we now additionally provide results for our proposed method using a **ViT model on the full ImageNet dataset**. The results are summarized as follows:\n\n| Method               | **SA(%)**      | **RA(%)**      | **RA-E(%)**    | **RA-R(%)**    |\n| -------------------- | -------------- | -------------- | -------------- | -------------- |\n| RT-ER                | 68.38$\\pm$0.17 | 56.16$\\pm$0.14 | 44.99$\\pm$0.16 | 70.82$\\pm$0.13 |\n| Adversarial Training | 60.32$\\pm$0.15 | 44.64$\\pm$0.14 | 43.06$\\pm$0.17 | 70.24$\\pm$0.14 |\n| Trades               | 61.94$\\pm$0.12 | 45.54$\\pm$0.17 | 43.75$\\pm$0.13 | 70.37$\\pm$0.11 |\n\nThese results demonstrate that our method (RT-ER) consistently outperforms AT and Trades on the ImageNet. Notably, RT-ER achieves **12% (10%) improvement in RA** and **8% (6%) improvement in SA** compared with AT (Trades). This underscores the scalability and effectiveness of our approach, even on large-scale vision benchmarks like ImageNet.\n\n[1] Lin F et al. Towards Robust Vision Transformer via Masked Adaptive Ensemble.\n\n[2] Zhang Y et al. Robust mixture-of-expert training for convolutional neural networks.", "claims_and_evidence": "The analysis of vulnerability of different part of MoE model is clear and convincing.", "methods_and_evaluation_criteria": "The proposed method contains three parts, but in some degree, more like a combination of different methods. And these robustness enhance methods, like aligning outputs with KL-loss and mixing clean and adversarial outputs, are similar with existing methods for classical neural network. The above factors decline the novelty of the proposed method.", "theoretical_claims": "None", "experimental_designs_or_analyses": "The experiment is conducted on small-scaled datasets, such as CIFAR10 and TinyImageNet. But the MoE architecture is designed for large-scaled datasets, where needs large model but control the computational overhead. Therefore, I think the author should contains more experiment results on larger-scaled datasets, like ImageNet-1K or ImageNet-21K.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMVzVvMT0p", "reviewer": "Reviewer_nXYP", "summary": "This paper proposes a novel adversarial training algorithm to improve the robustness of MoE models based on pilot studies on MoE attacks. They further interpolate the robust MoE and the unrobust MoE by linear interpolation to balance clean accuracy and robust accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for raising the seven insightful questions. We appreciate your thorough review and address each concern in detail below.\n\n1. ## Why Attack is not Weak?\n\n   ​\tWe compared our self-implemented AutoAttack with the official library. The perturbations generated by both methods are identical, and the results on the CIFAR-10 match. Second, regarding the concern that our PGD attack takes the end value, we note that many other studies adopt similar attack settings, such as [1,2]. Additionally, we conducted further experiments on TinyImageNet, both by using the final value and by selecting the highest loss point. The attack outcomes were nearly identical, and the loss values remained smooth in the later iterations.\n\n2. ## Why RA-E Attack Remains Effective?\n\n   ​\tWe would like to clarify that our evaluation assesses the vulnerability of both the router and experts by measuring accuracy relative to MoE. Regarding your concern about the RA-E attack’s effectiveness under varying router scores, we emphasize that this attack primarily targets the experts. As a result, the perturbations remain largely independent of the router’s behavior. Empirically, our experiments show that in **98%** of cases, attacked images are routed to the same expert(s), ensuring that the RA-E attack remains effective.\n\n3. ## Why Using KL Divergence in Eq.2?\n\n   ​\tThe goal of our paper is to explore a potential approach to further enhance the robustness of MoE. Regarding the use of KL divergence in the second term of Eq. 2, we chose this formulation because applying AT to the second expert would force the additional expert’s predictions toward the ground truth, potentially leading to overfitting.\n\n   ​\tWe conducted experiments comparing our method against two alternative formulations you suggested. Our results showed that RT-ER achieved similar performance to (ii) while improving RA by 2% compared to (i).\n\n4. ## Inclusion of Full MoE Results\n\n   ​\tIn Section 4.2, we compare the performance of MoEs using top-1 and top-2 routing strategies to demonstrate that our method generalizes across different routing mechanisms.\n\n   ​\tTo further address your request, we provide additional results for a Dense MoE and an MoE using a top-3 router on the CIFAR-10:\n\n   | Method | SA(%) | RA(%) | RA-E(%) | RA-R(%) |\n   | ------ | ----- | ----- | ------- | ------- |\n   | Dense  | 79.23 | 70.50 | 76.51   | 72.62   |\n   | Top-3  | 79.18 | 70.11 | 76.24   | 72.48   |\n\n5. ## Why $\\alpha$≥0.5 Guarantees Robustness of the Dual-Model?\n\n   ​\tWe provide theoretical evidence supporting the conclusion that$\\alpha \\geq 0.5$ is necessary to guarantee the robustness of the dual-model.\n\n   ​\tEq. (9) establishes a bound on the certified robustness radius:\n   $$\n   \\lVert \\delta \\rVert_p \\leq \\epsilon = \\min\\limits_{k \\neq y} \\frac{\\alpha \\left(F_R^{(y)}(x) - F_R^{(k)}(x)\\right) + \\alpha - 1}{\\alpha \\sum_i \\left(2 r_{R_i} + a_{R_i}({x}) \\left(L_{R_i}^{(y)} + L_{R_i}^{(k)}\\right) \\right)}.\n   $$\n   ​\tIn the numerator of Eq. (9), $\\alpha$ ranges from 0 to 1, and the maximum value of $\\left(F_R^{(y)}(\\mathbf{x}) - F_R^{(k)}(\\mathbf{x})\\right)$ is 1. If $\\alpha$ is smaller than 0.5, the numerator becomes negative,making the certified robustness radius undefined. Therefore, we conclude that $\\alpha \\geq 0.5$ is a necessary condition to ensure the robustness of the dual-model.\n\n6. ## Why only Robustifying the Second Expert?\n\n   ​\tDirectly robustifying all experts simultaneously would be computationally expensive and inefficient, making MoEs less appealing for large-scale applications. To balance efficiency and robustness, we propose RT-ER. For each input, RT-ER additionally robustifies an **expert not selected** by the router. Since the router dynamically selects different experts during training, the additional expert chosen for robustification also varies over time. This iterative process enables **RT-ER to progressively improve the robustness of the entire expert network** without significantly increasing computational costs.\n\n7. ## Why Not Simply Train a Robust Model?\n\n   ​\tCompared to simply training a robust model with $(1 - \\alpha) L_{clean} + \\alpha L_{rob}$, our method offers two key advantages:\n\n   1. **Better SA-RA Tradeoff**: The dual-model strategy in JTDMoE provides a better balance between SA and RA. Simply training often leads to conflicting gradients, limiting performance. We compare JTDMoE with a MoE of 8 experts trained using $\\alpha = 0.7$ and the combined loss. On CIFAR-10, JTDMoE achieved 92.29% SA and 74.62% RA, whereas the simply trained model reached only 87.48% SA and 67.45% RA. This shows that JTDMoE achieves a significantly better tradeoff.\n   2. **Reduced Training Costs**: Our approach eliminates the need for adversarial training in the standard MoE.\n\n   [1] Zhang Robust mixture-of-expert training for convolutional neural networks.\n\n   [2] Bai Improving the accuracy-robustness trade-off of classifiers via adaptive smoothing.", "claims_and_evidence": "I am concerned about the claims made in Section 4 and 5. See questions for details.", "methods_and_evaluation_criteria": "The method mostly make sense. See concerns regarding its implementation, evaluation and intuitions in questions.", "theoretical_claims": "The theorems are intuitive but I do not check their formal proof.", "experimental_designs_or_analyses": "I have major concerns regarding the design of experiments. See questions.", "relation_to_broader_scientific_literature": "This work combines adversarial robustness and MoE models. While both are extensively studied, I am not aware of prior studies on the adversarial robustness of MoE.", "essential_references_not_discussed": "The reference is sufficiently discussed.", "other_strengths_and_weaknesses": "The main strength of this work is to consider a novel perspective of MoE models. However, since MoE models are more commonly used in LLMs but not classical problems, maybe it's better to evaluate the proposed methods in LLMs as well.", "other_comments_or_suggestions": "Eq 2 has typos.\n\n\\icmltitlerunning is not properly set.", "questions_for_authors": "1. In Section 4.1, Why RA-E attacks are even better than attacking the full MoE, i.e., RA? This usually suggests that the applied attack is too weak (or badly applied) for the full MoE model. This might also explain the bad performance of adversarial training in Section 4.2. Further, Fig 2 shows the standard accuracy for AT is higher and the robust accuracy is lower, indicating a bad attack as well. I took a brief read on the code, and surprisingly the authors use a self-implemented AutoAttack, while the official library is easy-to-install and publicly available at https://github.com/fra31/auto-attack. This makes me question the effectiveness of the implementation. Further, the PGD attack code simply runs for a fixed number of iterations and takes the end value, while it should take the highest loss point during the full trajectory.\n\n2. In Section 4.1, RA-E and RA-R attacks a changed formulation of MoE; is the accuracy here still evaluated w.r.t. the original MoE model? I can hardly believe that since RA-E attack assumes constant router score during attack, this remains effective when the router scores are changed by the adversarial attack. More details should be provided.\n\n3. Eq 2 looks like applying AT on the full MoE and including TRADE on the second expert. This should be discussed or unified. Currently, there seems to be no evidence why such choice is better than (i) use AT on the second expert and (ii) use TRADE on the full MoE.\n\n4. Section 4.2 changes MoE to top-1 and top-2 routers. Could the authors include the results of the full MoE model, which is used by other experiments?\n\n5. Line 274, why does $\\alpha \\ge 0.5$ guarantee robustness of the dual-model?\n\n6. According to Line 280, robustifying every expert is essential. Why does RT-ER only improve the second expert but not all experts?\n\n7. The dual model effectively interpolates a clean model and a robust model; can't we simply train the robust model with $\\alpha L_{\\text{clean}} + (1-\\alpha) L_{\\text{rob}}$ to get the same effect? This is more common when one wants to do interpolation. In addition, the dual model is twice the size of a single model, thus the comparison in Table 3 is unfair.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an adversarial training method to improve the robustness of MoE models. The approach uses linear interpolation between robust and non-robust MoE models to balance clean and robust accuracy. Reviewers raised some concerns about the novelty of the paper and the significance of its theoretical contributions. They also noted that more results on larger-scale datasets are needed. In the rebuttal, the authors ran additional experiments, which strengthened the empirical evidence.
Decision: Accept (poster)