Mixture of Experts for Image Classification: What's the Sweet Spot?
Authors: Mathurin VIDEAU
Alessandro Leite
Marc Schoenauer
Olivier Teytaud
Venue: Accepted by TMLR ()

Abstract: Mixture-of-Experts (MoE) models have shown promising potential for parameter-efficient scaling across domains. However, their application to image classification remains limited, often requiring billion-scale datasets to be competitive. In this work, we explore the integration of MoE layers into image classification architectures using open datasets. We conduct a systematic analysis across different MoE configurations and model scales. We find that moderate parameter activation per sample provides the best trade-off between performance and efficiency. However, as the number of activated parameters increases, the benefits of MoE diminish. Our analysis yields several practical insights for vision MoE design. First, MoE layers most effectively strengthen tiny and mid-sized models, while gains taper off for large-capacity networks and do not redefine state-of-the-art ImageNet performance. Second, a Last-2 placement heuristic offers the most robust cross-architecture choice, with Every-2 slightly better for Vision Transform (ViT), and both remaining effective as data and model scale increase. Third, larger datasets (e.g., ImageNet-21k) allow more experts, up to 16, for ConvNeXt to be utilized effectively without changing placement, as increased data reduces overfitting and promotes broader expert specialization. Finally, a simple linear router performs best, suggesting that additional routing complexity yields no consistent benefit.

Published in Transactions on Machine Learning Research (10/2025)
Mixture of Experts for Image Classification: What's the Sweet Spot?
Mathurin Videau
mvideau@meta.com
Meta AT, TAU, INRTA, and LISN (CNRS & Unic. Paris-Saclay)
Alessandro Leite
adeste:iisoorruen..
INSA Rouen Normandy, University of Rouen Normandy, LITIS UR 4108
Marc Schoenauer
man.schoenoaerrGhnvii.f
TAU, INRTA and LISN (CNRS 8 Univ. Paris-Saclay)
Olivier Teytaud
Thales, CortAIz-Labs
Reviewed on OpenReview: https: openreview. net/ forum? id= Jgt
Abstract
Mixture-of-EEae Expertsss (MoB) models have shown promising potential for parameter-efficiett scaling across domains. However, their application to image classification remains limited, often requiring billion-scale datasets be competitive. In this work, we explore the integration of MoE layers into image classification architectures using open datascts. We conduct systematic analysis across different MoE configurations and model scales. We find that moderate parameter activation per sample provides the best trade-off between performance and efficiency. However, as the number of activated parameters increases, the benefits of MoE diminish. Our analysis yiclds several practical insights for vision MoE design. First, MoE layers most. ellectively seengghhen tiny and mid-sized models, while gains taper off for large- capacity networks and do not redefine state-of- theart ImageNet performance. Second, Last-2 placement hcuristic offers the most robust cross-architecture choice, with Every-2 slightly better for Vision Transform (ViT), and both remaining effective as data and model scale increase. Third, larger datasets (e.g., TmagoNet-21k) allow more experts, up to 16, for ConvNeXt to be utilized effectively without changing placement, as increased data reduces overfitting and promotes broader expert specialization. Finally, a simple linear router performs best, suggesting that additional routing complexity yields no consistent benefit.
1 Introduction
Recent advances machine learning, particularly in the domains natural language process- ing (VasP)(V et al., 2017; Kenton & Toutanova, 2019) and computer vision (Dosovitskiy et al., 2020), have been primarily driven by scaling up model size, computational budgets, and training data. Although these large-scale models demonstrraee impressive performance, they are often expensive to train and consume considerable energy resources (Strubell al., 2019). As result, the research community has become increasingly interested in exploring more efficient training and serving paradigms. One such promising solution is the use of sparse expert models, with Mixture-of-Expert (MoE) (Shazeer et al., 2017; Lepikhin et al., 2020) emerging as a popular variant.
MoE models introduce sparsity by partitioning the set of parameters into multiple parallel sub-models, called "experts." During training and inference, the gating part of the models routes input examples specific expert(s), ensuring that each example only interacts with subset of the network parameters. As the

---

Published in Transactions on Machine Learning Research (10/2025)
computational cost is partially correlated with the number of parameters activated for a given sample rather than the total number of parameters, this approach facilitates the scaling-up of the model, while keeping computational costs under control, making it an attractive option for a wide range of applications.
Despite their success in various domains (Hihn & Braun, 2021; Costa-jussà et al., 2022; Zoph et al., 2022; Fedus et al., 2022), the application of MoF models in image classification and computer vision, in general, remains limited, and it often requires very large datasets (Riquelme et al., 2021; Mustafa et al., 2022; Komatsuzaki et al., 2022) to be competitive against state-of-the-art approaches. In this work, focus leveraging the potential of MoE models for image classification ImageNet- 1k and ImageNet-21kk 21k (Russakovsky et al., 2015). We study the efficiency of integrating MoE within two renowned architecturos, ConvNext (Liu et al., 2022) and Vision Transformer (ViT) (Touvron et al., 2022). We conduct a series of experiments considering various architecture configurations. Likewise, we investigate the impact of varions components, including the number of experts and their sizes, the gate design, and the layer positions, among others. Our experimental findings indicate that optimal design is contingent on the specific network architecture, and that consistently situating the MoE layer within the final two even blocks invariably yields substantial improvements for moderate model size. Nevertheless, when scaling up the approach to large models and datasets close to the state-of-the-art, the benefits of using Mixturo-of-Expee
for image classification gradually vanish.
Contributions. In this work, we provide a systematic study of applying Mixture-of offpture-f-EExertt (MoE) models to image classification tasks, focusing on the ImageNet-1k and ImageNet-21ke benchmarks. Training follows supervised image classification in two setups, either on ImageNet- 1K or ImageNet 222. Unlike prior efforts that emphasize large-scale deployments with vast compute resources, we explore MoF. integration within mid-sized ConvNeXt and Vision Transformer (ViT) architectures, identifying effective design principles for efficient training and inference. Our analysis yields the following practical guidance for vision MoE design:
When MoE helps (scale): MoE strengthens liny and mid-sized models, but gains diminish for large-capacity networks, and do not redefine state-of-the-art ImageNet performance.
Placement heuristic: A Last- configuration is the most robust cross- architetcce choice, with Every-2 slightly better for ViT. These placements remain effective when scaling data or model size.
Scaling with data: With larger datasets (e.g,, ImageNet-21k)) more experts can be used effectively, up to 16 for ConyNeXt, without changing placement. Increased data reduces overfitting and enables broader expert utilization.
Routing choice: A simple linear router works the best, indicating that added routing complexity brings no consistent benefit.
2
Related Work
The Mixture-of- Experrss (MoE) model, introduced by (Jacobs et al., 1991), partitions complex tasks into hopefully simpler sub-tasks handled by expert models, whose predictions are combined to produce the final output. This framework has been successfully integrated into various neural network architectures, particularly transformers for NLP tasks (Shazeer et al., 2017). Given this success in NLP, interest has increased in applying MoF to the diverse and complex computer vision tasks.
The transformative potential of MoE architectures is underscored by their successful integration across various domains, with the Vision Transformer (VIT) being prime example. Riquelme et al. (2021) introduced V-MoE, MoLE-augmented ViT model, for image classification tasks massive dataset containing hundreds of millions of examples. They showed that V-MoE not only matches the performance of prior stato-of-the-art architectures but also requires half the computational resources during inference. Lou el al. (2021) presented a sparse MoE MLP model based on the MLP-Mixer architecture C (Tolstikhin et al., 2021). While the MLP- architocture does not display performance accuracy, the MoE-enhancod version surpassed its dense counterpart in experiments on ImageNet and CIFAR. Hwang al. (2023a) demonstrated the effectiveness of SwinV2-MoBs MoF-based model built. upon Swin Transformer V2 architecture. They reported superior accuracy in
2

---

Published in Transactions on Machine Learning Research (10/2025)
downstream computer vision tasks. Similarly, Puigcerver et al. (2024) introduced soft MoE mechanism, demonstrating improved performance and training speed compared to classical MoE on billion-scale datasets. More recently, Han et al. (2024) proposed ViMoE, which investigates expert placement strategies by fine- tuning DINOv2 models. In contrast, our work focuses on training MoE architectures from scratch, cnabling a systematic study of their behavior across both ConvNeXt and ViT backbones. This distinction allows us lo provide complementary insights into how MoE mechanisms contribute to compute efficiency and performance in vision models.
3 Sparsely Activated MoE
Based on conditional computations, MoE aims at activating specific parts the model depending on the input. The core idea is to assign experts to different regions of the imput space, thereby increasing the model capacity by augmenting the number of paramcters without incurring significant computational overhead (Jacobs et al., 1991).
According to Shazeer et al. (2017), an MoE' includes a router G and a set of experts E. The router learns a sparse assignment between the input and the experts, while the experts process the inputs like standard neural network modules do. Let æ be the imput, and let (Es)eet,,) be the N experts of the MoE layer. Following (Shazeer al., 2017), the output of this MoE layer is given by:
Σ G(x)) Ex(x)
(x)
where, G is a convlxl gate employing a softmax function, and G(z); is its 1th output. To promote sparsity in the MoE layer, the number of participating experts is restricted to < N for each imput: Topı (x) contains the indices of the k highest G(x); Note that involving is > 1 experts increases the computational cost compared to the sparse (k= = 1) counterpart. We tested several gate designs, but none of them clearly outperformed the simple convix1 gate (Sec. 4.3.5).
However, enforcing this MoE design is insufficient to yicld an effective MoE. To exploit compute capacity optimally, we encourage uniform usage of each expert by using a load balancing auxiliary loss proposed in (Shazeer et al., 2017), which promotes balanced expert assignments and equal importance experts (for more details, see Section Shazeer et al. (2017)). We also use Batch Prioritized Routing (BPR) introduced by (Riquelme et al., 2021).
3.1 Vision Transformer
The Vision Transformer (ViT) (Dosovitskiy et al., 2020) represents a paradigm shift in computer vision. It moves away from traditional convolution- based architectures in favor of the Transformer's attention mechanism, initially pioncered for natural language processing tasks (Vaswani al., 2017). In the VIT model, an image segmented into fixed-size and non-overlapping patches. These patches are linearly embedded into vectors and then passed through series of Transformer blocks. Each Transformer block is predominantly made up of two main components: multi-head self-attention mechanism and multi-layer perceptron (MLP). Within the context of VIT, the MLP ratio (denoted mlp_ratie ratio in the following) refers to the ratio of the hidden dimensions of the MI.P to the embedding dimensions. This ratio is pivotal, as it determines the capacity and computational cost of the MLP component. We replace some predefined MLP block by a MoE, as depicted in Fig. 1a. Each MoE operates at the patch level: the experts are spatially distributed. We consider (a) a standard ViT architecture (ViT-B (Base)) and (b) a variant with experts (ViT-B-8).
3.2 ConvNeXt
ConvNeXt (Liu et al., 2022) has modernized the classical convolutional neural networks (CNN) design by incorporating insights from Vision Transformers (ViT). The central component of the ConvNeXt architecture is a block containing a x depth-wise convolution followed by a x convolutional layer (Fig. 1b-left). This block is replicated across four stages, each with decreasing resolution. In this work, we replace some predefined
3

---

Published in Transactions on Machine Learning Research (10/2025)
7+7,
Attention, 384
dwr 7x7, 96
IN
Attention, 384
LN
Gatc
Gate
1x1, 384
Linear, 1.536
GELU
Emprto
Eppei
BELU
Lionn
1x1, 96
Linear, 384
(b) Left: ConvNeXt original base block. Right: The modified MoE CouvNeXi block. The green link repre- sents the added skip connection. DW: Depth-wise. LN: Layer normalization.
(a) Left: VIT base block. Right: Our modified MoE VIT block. LN: Layer uormalization. GELU: Gaussian Error Linear Units.
Figure 1: Vision Transformer and ConvNext architectures.
ConvNeXt block, namely the multilayer perceptron, with MoE block, and introduce skip connection, illustrated in Fig. 1b-right. Consequently, each expert becomes fully convolutional block operating at the feature map level, resulting in spatially distributed experts. Hence, this design not only increases the network capacity but also allows each expert to specialize in specific spatial locations and engages multiple experts per image. We examine three variants: (a) ConvNect-T (Tiny), ConvNeXt-S (Small), and ConvNeXt-B (Basc). These architectures primarily differ in depth and width; and (b) ConvNeXt-S-44 and ConvNeXt-B-44 which variants with experts.
4 Experiments ImageNet
This section presents the results obtained by the architectures described above (that incorporate MoE' into ConvNeXt and VIT architectures), trained on ImageNet-1k or pre-trained on ImageNet-21k datasets. It includes the experimental setup, the results on the ImageNet-1k validation set, and the impact of various MoE configurations.
4.1 Experimental Setup
All the results presented in this section rely MoE models trained the ImageNet datasets. Although the context is different due to the changes in network structures presented in Secs. 3.1 and 3.2, we set the training hyperparameters similar to (Touvron et al., 2022) for VIT and (Liu et al., 2022) for ConvNeXt (a grid search looking for better settings did not bring significant improvement).
Furthermore, when working with the ImageNet-1k dataset, we use a strong data-anggennattio pipeline, including Mixup (Zhang et al., 2018), Cutmix (Yun al., 2019), RandAugment (Cubuk et al., 2020), and Random Erasing (Zhong et al., 2020), over 300 epochs. Likewise, utilize drop path, weight decay, and expert-specific weight decay regularization strategies. However, for the pretraining phase ImageNet-21k,, we exclude Mixup and Random Erasing from the data augmentation pipcline because they did not improve the results, as also reported in (Touvron et al., 2022; Tu et al., 2022). We pre-train the model for 90 epochs, adhering to the original ConvNeXt approach (Liu et al., 2022) and, consistently with (Liu et al., 2022) and (Touvron al., 2022), the final results of ImageNet-Ik 1k are obtained through finc-tuning for 30 epochs for ConyNeXt and 50 for ViT. Comprehensive details of all the hyperparameters are provided in Tab. 10 in App. A.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Accuracy for different ImageNet-1K trained models and the MoE strategies "Every 2" and "Last 2" (see text). "Top k" corresponds to the number of experts involved. Throughput is measured on V100 GPUs, following (Touvron al., 2021). For ImageNet-lk, non-isotropic CouvNeXt models feature an mlp_ratio which contributes to their improved popp-eee--amplle efficiency compared to their dense counterparts.
ratio of 2,
#Params (x10")
Per sample
Throughput (im/s)
IN-IK Accuracy
Architecture
FLOPs
#Paramsact
ConvNeXt-T (Liu es al., 2022)
28.6
28.6
1.3G
814
82.1
ConvNeXt-TT44 Last Top
34.5
25.6
4.2G
768
82.1
ConvNeXt-8 (Lin al., 2022)
50
8.7G
466
83.1
ConyNeXt-S-4 Last 2 Top
56.1
47.3
8.5G
442
83.1
ConvNectt-B (Liu et al,, 2022)
88.6
88.6
15.4G
299
833.8
ConyNext-BB-t Last. Top
83.4
ConvNeXt-S (iso.)
22.3
22.3
4.3G
1100
79.7
ConvNeXt-S-8 (tso.) Last Top
22.3
4.3G
1031
80.3
CoavNeXt-BB (fso.)
82.4
16.9G
82.0
ConyNeXi-B-ss (iso.) Last 2 Top
115.4
82.4
16.9G
303
81.6
VIT-S
22.0
22.0
4.6G
1083
79.8
VIT-S-8 Last 2 Top 2
38.6
25.0
5.3G
892
80.5
ViT-S-8 Every Top 2
71.7
33.1
6.9G
724
80.7
VIT-B
86.6
17.5G
82.8
VET-B-8 Every 2 Top 2
284.9
129.9
26.3G
227
82.5
Table Accuracy for ImageNet-21KK pre-trained models and different adaptations with ImageNet-1K; IN-1K is standard fine-tuning, Linear prob is the training of linear layer for the output probability layer (everything else being frozen), and 0-shot is the direct application of the pre-trained model to ImageNet-1K. "Every 2" and "Last 2" are the corresponding MoF strategies (see text). "Top A corresponds to the number of experts involved.
Architecture
PParass
Per sample +Paramsect
FLOPs
IN-1K Accuracy
Linear prob
0-shot
(x10%)
Accuracy
Accuracy
ConvNeXi-T (Liu al., 2022)
28.6
28.6
4.5G
82.9
81.8
44.3
ConvNeXt-T-ae Last Top
70.0
28.7
4.5G
83.5
82.3
44.6
S (Liu al., 2022)
84.6
83.2
Con-NeXt-S-8 Last 2 Top
91.6
50.3
8.7G
84.9
83.6
45.3
Ne (Liu et al., 2022)
88.6
88.6
15:4G
86.8
84.9
45.8
Conv-Nett--8
Top
15.4G
85.7
VIT-S
22.0
22.0
4.6G
82.6
81.7
44.2
VIT-S-8 Every Top 2
71.7
33.1
6.9G
83.0
81.9
44.7
ViT-B
86.6
80.0
17.3G
85.2
84.0
45.7
ViT-B-8 Every 2 Top 2
284.9
129.9
26.3G
85.2
84.0
45.6
4.2
Base results on ImageNet
Tab. 12 presents the results obtained on ImageNet-lke validation set by a model that has been entirely trained on ImageNet-1k, for isotropic architecture (e.g., ViT, ConvNeXt iso.) and a hierarchical architecture, namely ConvNeXt. We see some significant improvements in accuracy compared to the "no-MoF." results for small model sizes, especially for anisotropic models.
Tab. 2 presents the results of models that are pre-trained ImageNet-21k,, and tested the same ImageNet- Ik validation set than above. Here, MoE does bring some improvement for moderate numbers of activations per sample. However, this improvement decreases for large numbers of activations per sample, as depicted in Fig. 2.
MoE Strengthens Moderate Model Size; Not the Frontier
MoE integration provides clear gains at moderate scale, with tiny and small models benefiting most. As data scale grows, these relative improvements strengthen. Yet as model capacity rises, these benefits diminish, showing that MoE enhances mid-sized models but does not redefine SoTA performance.
5

---

Published in Transactions on Machine Learning Research (10/2025)
90.0
T0p 18-33 (Ng
LIN-KK
83-
82.5-
model
82-
VIT
00.0
81
No
No
Yes
75.0
Yes
80
Pareto
Pareto
3x1
60
30
10'
10
10*
#Aclivations invvlved
#Activations inclived
Figure 2: Left: Pareto-front for ImageNet21k, x-axis = number of activations per sample. VIT models have been presented MoE versions only after additional pretraining, and are therefore not presented. MoE seems to be Pareto optimal for a number of activations per sample below 90M. Right: Pareto-frov froeto-front for VIT models on JFT-300M. Overall, MoE is never validated for number of activations per sample above 100M.
4.3 Sensitivity with Respect to Design Choices
This section presents different sensitivity studies that investigate the specifics of the design choices of the MoE layers. We explore the impact of the position of these layers inside the architecture (Sec. 4.3.1), the effect of varying the number of experts (Sec. 4.3.2), and finally, we assess the necessary design changes when transitioning from the ImageNet-Ik to ImageNet-21k dataset (Sec. 4.3.4) and the impact of the routing archilecture choices (Sec. 4.3.5).Ideally sensitivily analyses would explore the joint effects of mult factors (such as the number, placement, and size of experts), but exhaustive multi-axis exploration quickly becomes computationally intractable. Nevertheless, we partially address such interactions by analyzing the combined impact of expert size and expert count in ?? highlighting how these factors jointly influence efficiency and accuracy.
4.3.1 Impact of the Position of MoE Layers
Table 3: Comparative results for different positions of MoE layers: ImageNet-lk training on ConvNeXt-1 and VTT-S, all employing experta.
Params
Architecture
MoE
-10
Sample
FLOPs
IN-IK
Tonv-Next-T
MoE
28.6
28.6
4.5G
82.1
every stage
54.3
17.0
3.8G
81.8
ConvNext-T-8
17.4
25.5
4.0G
82.1
last
4.1G
last 2
46.3
25.6
4.2G
82.1
CouvNeXt-8
(4so.)
22.3
22.3
8.7G
Con-NeXi-S-8
every
96.7
22.3
8.7G
79.6
(Ss0.)
last
38.9
22.3
8.7G
80.3
Vrr-s
no MoE
22.00
22.0
4.2G
79.9
VIT-S-8
every
last
38.6
25.0
6.9G
80.5
Tab. displays the effects various placements of MoE layers, comparing the following configurations: (a) Every MoE layer replaces every second layer; (b) Stage: a MoE layer replaces the final layer of each stage, resulting in four MoE layers throughout the network for ConvNeXt; (c) Last MoE layer replaces the final layer of each of the last two stages; (d) Last 3: Same "Last 2", but with additional MoF layer

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Comparative results for different numbers Table 5: ImageNet-21k training on Conv.eett-T em- of experts. ImageNet-lk training using the "Last 2" ploying 8 experts and an MLP ratio of 4 unless speci- fied explicitly.
strategy.
Paramss
Por
IN-
MoF.
Params
Per samples #Paramss.ot
FLOPs
IN-1K
Architecture
# experts
Top-k
6am- ples
strategy
(x10")
Top
(x10%)
&Params, Top
no MoE
28.6
28.6
4.5G
82.9
acc.
every
97.5
28.6
4.5G
82.9
ConyNeXi-T
no MoE
28.6
28.6
82.1
stage
78.2
28.6
4.5G
83.5
last
70.0
28.6
4.5G
83.5
64.3
26.5
82.1
last 2
70.0
28.6
4.5G
83.5
ConyNeXi-T
47.4
25.5
82.1
47.4
25.5
82.0
mlp ratio
46.3
25.6
4.2G
83.4
46.3
25.5
81.7
VIT-S
no MoE
22.0
22.0
79.9
2
29.1
25.0
79.8
VIT-S
38.6
80.2
2
38.6
25.0
80.5
2
57.5
25.0
80.2
inserted in the middle stage 3. As shown in Tab. 3, "Last 2" strategy the most robust choice: it performs well across all architectures. contrast, "Every performs worst for CouvNeXt architecture, and the best for ViT. Consequently, in the following, all experiments use "Last 2" for ConvNeXt and "Every 2 for ViT.
MoE placement matters
While VIT benefits most from insorting MoE layers at every second block and ConvNeXt from placing them in the last two stages, the Last 2 stralegy offers simple, broadly effective rule of thumb that consistently delivers strong performance across architectures.
4.3.2 Influence of the Number of Experts
Tab. 4 presents the performance of ConvNext and ViT architectures on ImageNet-IK training with the "Last 2" strategy MoE layers. It provides comprehensive view of how the number of experts influences the size (parameter count) of the models and their Top-1 accuracy on this dataset.
The experimental results suggest that four experts yield the best performance for ConyNeXt, while eight experts are optimal for ViT. This showcased by the Top-1 accuracy rates, which are consistent at 82.1% for ConvNeXt with four experts and reach peak of 80.5% for ViT with eight experts. However, we note that increasing the number of experts to 16 has detrimental effect on both architectures. Specifically, for ConwNeXt, the top-1 accuracy slightly drops to 81.7%, and for VIT, the performance plateaus at 80.2%.
These findings highlight that while MoE layers can enhance model performance, there is delicate trade-off to be struck in terms of the number experts. Exceeding the optimal number can lead to suboptimal results, negating the potential benefits of the MoE integration. Furthermore, in our detailed analysis, described in Sec. 4.3.3, we explore the interplay between the number and the size of each expert. Our investigation revealed that, for isotropic networks such as ViT and ConvNeXt iso., reducing the size of experts adversely affects performance, while for ConvNeXt, it has no significant impact. Morcover, during the course of our experiments, we observed that while the Top-1 configuration was superior for ConvNeXt, the Top-2 configuration for ViT was comparable or even better than the ConvNeXt Top-1 configuration, leading us to employ Top-1 for ConvNeXt and Top-2 for ViT (cf. Tab. 4).
Expert Count under Moderate Data Scale
On ImageNet- Ik, ConvNeXt models reach peak performance with experts, while ViT benefits from up to This contrast stems from architectural width: ConvNeXt's broad final stages saturate quickly, limiting the value of additional experts, whereas ViT's narrower last stage tolerates more experts.

---

Published in Transactions on Machine Learning Research (10/2025)
4.3.3
Impact of Expert Size and Number
80
80
IMNETV2
Acmuray in the tast set.
IMNETR
70
IMNETSK
60
50
Acauracy
40
30
30
20
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Number of exporte
MCP-racio of expoxto
(b) Experts' MLP-ratio (E (2,4]) vs. performance. Points' size: # experts (E (1,4,8, 16)).
(a) # experts (E
161) vs. performance. Points' size: MLP-ratio (E (1,2,4)).
Figure 3: Exploring the interplay between the size and count of experts in MoE layer for ConvNeXI-T on ImageNet-1K. Baseline results (without MoF) are denoted by dotted lines. For this small dataset, MoE does not bring much improvement (see Fig. 2 for bigger datasets).
Two critical tunable parameters in the MoE layers are the number of experts per layer and the size of each expert. The latter defined by the mlp_ hip_rrtto,______ which scales the hidden dimension of the forw-forward network (FFN) (Sec. 3.1). Fig. illustrates the sensitivity of ConvNeXt's performance to variations in these parameters, with the number of experts ranging from 4 to 16 and the mlp ratio from I to 4. By default (original network), this ratio is set to 4, meaning that values below 4 correspond to smaller per-sample FFN capacities than in the original network.
To assess generalization, we report top-1 accuracy across three dataset categories: (i) in-distribution (ID) datasets ImageNet Imageees-ImageNNee and ImageNet (Recht et al., 2019); (ii) out-of-distributioo (OOD) datasets ImageNet-- R (Hendrycks al., 2021a) and ImageNet-Sketdl (Wang et al., 2019); and (iii) adversarial datasets ImageNett A (Hendrycks et al., 2021b). The objective is to determine whether performance benefits more from using many smaller experts fewer larger ones.
Our experiments on ConvNeXt reveal that while MoE' layers can slightly improve or preserve performance on ID datasets, they tend to degrade performance on OOD datasets. In particular, increasing the mumber of experts to 16 consistently reduces accuracy across all dataset lypes. Regarding expert size, smaller experts appear marginally more robust to increases in expert count. Overall, the best tradcoff for ID is achieved with to experts and an mlp ratio of 2 4. Based on these findings, we adopt configuration of experts ImageNet-1K. For ConvNeXt isotropic and VIT architectures,
with an mlp mal__ratio of 2 for experiments however, reducing the mlp ratio consistently harms performance, and in the case of ViT, it also introduces training instabilities. Also, OOD performance is typically degraded in Fig. 3.
Balancing Expert Number and Width on ImageNet-lk
On ImageNet-lk, 1k, the best trade-off is achieved with 4-8 experts and mlp ratirrttooo 44; increasing 16 experts consistently reduces accuracy and OOD robustness. Increasing number of experts while reducing their size helps reducing overfitting.
4.3.4 Results on ImageNet-22
Tabs. 5 and 6 present the results of models trained on ImageNet-2lk, reporting ImageNet- 1k accuracy after fino-tuning. We analyse the effect of MoE layer positioning, expert size (Tab. 5), and expert count (Tab. 6). The results demonstrate that, when large volumes of data are accessible, a greater number of experts can be effectively deployed. Notably, the use of sixteen experts, which sub-optimal for ImageNet-Ik, does
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: ImageNet-21k training on ConvNeXt-T: comparison between various numbers of experts, with an
MLP ratio of
# experts
PPrrams (x10")
Per samples
FLOPs
IN-1K
*ParamSact
Top 1 acc.
MoF
28.6
28.6
4.5G
82.9
64.3
28.0
4.5G
83.5
117.1
28.
4.5G
83.6
211.6
28.6
4.5G
83.4
not negatively affect performance on ImageNet-21k. This suggests that increasing the number of experts, together with the volume of data, could lead to valuable enhancements.
In terms of MoE layer configuration, data showed no performance improvements with the addition of more layers, and the "every 2" strategy again yielded the poorest results for ConvNeXt-based architecture.
Scaling Expert Count with Larger Pretraining
On ImageNet-21k, ConvNeXt scales up to 16 expert,s and ViT also benefits from more. Increased data reduces overfitting, allowing more experts. This highlights the link between dataset scale and MoE capacily. The optimal layer positioning remains unchanged.
4.3.5 Impact of the Routing Architectures
We investigated three distinct routing architectures for our study:
Table 7: Performance of MoE-Integrated Models vs Non-MoE Baselines: ablation study the gating network architecture. MoE layers placed on the last two even layers.
"Conv"; A straightforward x convolution, similar to a linear gate.
ConvNeXt-S
"Cos":
A gate that utilizes cosine similarity for routing, detailed in (Chi al., 2022). It is two-layer archi- tecture, with one linear layer projecting the features to a lover-dimensional space and another performing cosine similarity against some learned latent codes.
Gate
IMV2
IMA
IMSK
MoE
82.1
70.8
24.2
47.2
33.8
Conv
82.1
71.0
23.8
46.2
32.7
Cos
81.9
70.6
22.3
45.6
33.0
ConvNeXttS (iso.)
Gate
IN-1K
IMV2
IMA
IMSK
"L2": This is identical to the "Cos" gate, except that it employs the L2-distance for similarity instead of cosine.
MnF:
Conv
80.3
68.8
13.5
46.6
34.4
Cos
79.7
68.4
12.2
45.3
33.0
L2
80.1
68.6
13.3
46.1
34.3
We conducted training the ImageNet-IK dataset using var- tous small networks, each employing different routing mech- anisms as displayed earlier. The results, shown in Tab. 7, indicate that the simple convolutional (conv) configuration gen-
VIT-S
Gate
IN-1K
IMV2
IMA
IMR
IMSK
MaF
79.8
69.1
19.8
43.4
29.7
Conv
Cos L2
79.7
67.8
15.9
41.6
29.3
erally yields slightly better performance in most cases. However, when evaluating these models on out-of-distribution (OOD)
80.2
68.8
19.0
41.9
29.2
datascts, the results are more varied. The inclusion of OOD
evaluation is crucial, an elfective routing mechanism should ideally facilitate an expert decomposition that generalizes well. Notably, the MoE models that showed superior performance ImageNet-II -K also displayed enhanced robustness in the OOD evaluations.
Simplicity Wins in Routing Architectures
Across both ConvNeXt and VIT, straightforward linear gate achieves the best overall results. More complex routing adds no clear benefit and often underperforms, even on OOD data.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Robustness of MoE w.r.t. to domain change, for two different pretrainings, both fine-tuned on
ImageNet- Top-1 accuracies on various datasets.
Model
IN-1K
IMV2
TMA
TMR
IMSK
Imagenet 1k trained
CouyNeXI-T
82.1
70.8
24.2
47.2
33.8
ConvNeXt-T-4
82.1
71.0
23.8
46.2
32.7
83.1
72.5
31.3
49.6
37.0
ConvNeXt-SS
83.1
72.2
30.2
49.0
37.3
ConvNeXI-B
83.8
73.4
36.7
51.3
38.2
Co--NeXt-B-4
83.5
72.8
33.9
48.6
36.6
ViT-S
79.9
68.8
19.8
29.7
ViT-S-8
80.7
21.1
30.9
ViT-B
82.8
72.1
32.4
51.2
36.9
VTT-B-8
82.5
71.5
32.0
46.6
35.2
ConvNeXt-S (i80.)
79.7
68.6
13.0
46.4
34.0
Con-NeXt-S-8 (iso.)
80.3
68.8
13.5
46.6
34.4
ConvNeXtt-B (
82.0
71.1
21.2
50.0
38.1
ConvNeXt-B-8 (iso.)
81.6
70.6
19.1
48.5
35.7
Imagenet21k pretrained
CouvNeXt-T
82.9
36.2
51.1
ConvNeXt-11-8
83.5
72.8
32.6
51.3
40.8
ConvNeXi-8
84.6
74.7
44.8
37.5
43.6
ConvNeXt-S-8
84.9
75.5
44.4
55.7
45.5
ConvNeXI-B
85.8
76.0
54.6
62.0
48.8
ConwNeXt-B-8
85.7
76.3
51.8
59.6
48.4
VIT-S
82.6
72.6
38.9
50.8
39.0
ViT-S-8
83.0
72.7
35.3
51.0
39.4
VIT-B
85.2
76.1
56.0
61.5
46.9
VTT-B-8
85.2
75.4
48.1
45.9
5 Discussion
5.1
Hierarchical vs Isotropic Models
First, note that the best MoE designs typically apply experts only to the last layers. VIT operates at the same resolution at each layer after the input images have been cut into patches. This isotropic design implies that MoE is applied in layers that are not particularly large. On the other hand, with hierarchical models such as ConvNeXt, applying MoE to the last layers considerably increases model size. Indeed, as presented in Tab. 6, employing cight experts doubles the size the network despite using only two layers of MoE.
5.2 Positions and Numbers of MoE Layers
There are two prevalent strategies in the literature concerning the position of MoE layer: "Every 2" and "Last 2". In the context of hierarchical architectures such as ConvNeXt, the "Every 2" strategy dramatically increases the number of weights while simultancously yiclding inforior results. However, for ViT "Last 2" and "Every 2n are two viable strategies. This is line with the results V-MoE (Riquelme et al., 2021) on large-scale datasets. Overall, our results confirm that "Last 2" is a solid starting point, as consistently yields positive outcomes across all tested architectures. However, it may not always be the optimal choice for every architecture. For instance, in the case of ViT, "Every 2" demonstrates better results. Regarding the number of experts, we noted that model performance quickly saturates with the number of experts, reproduced in Tab. 6, and detailed in Fig. 3.
5.3 Robustness Evaluation
We evaluated MoE models, trained fine-tuned on ImageNet-Ik, 1k, against range of robustness benchmark datasets, such ImageNet-A (IMA) (Hendrycks et al., 2021b), ImageNet-R (IMR) (Hendryeks et. al., 2021a),
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table ViT-B trained on ImageNet-1k with different hyperparameters. In particular, for the top row (gray) follow the training recipe from Deit (Touvron et al., 2021) and for the bottom vron et al., 2022).
1.50
imagenet2kk magenettk JF3300M
1.25
follow Deit III (Tou-
Accurocy improvement with MOD
1.00
0.75
Architecture
Params
Per sample &Paramaact
IN-1K
0.30
(x10%)
Acc.
ViT-B-8 (Touvron et al., 2021)
86.6
86.6
81.8
0.00
impact
VIT-B-8 Every 2 Top
284.9
129.9
82.2
-0.25
VIT-B
(Touvron et al., 2022)
86.6
86.6
82.8
-0.30
ViT-B-8 Every 2 Top
78
82
84
80
284.9
129.9
82.5
Accuracy without MOE
Figure Improvement from MoE (y-axis) vs. base- line accuracy without MoE (x-axis) for image classi- fication, across pretraining sizes. Larger pretraining yiclds greater gains; higher baselinc accuracy reduces impact. Dashed lines show maximal improvement per accuracy level. Results are drawn from Tabs. 2, 11 and 12 (in App. B).
ImageNet-SSketch (IMSK) (Wang et al., 2019), and ImageNet V2 (IMV2) (Recht et al., 2019), the latter being used as a measure of overfitting.
Tab. 8 reports Top-1 accuracy for all datasets. It shows that MoE models tend to underperform their dense counterparts in ImageNet-1k. However, note that MoF models better for the robustness metric; they were actually alrcady better ImageNet-Ik. So, these results actually emphasize the success of MoE when the model is sufficiently small to be improved by MoE, as discussed in the previous section: the domain generalization holds only in the regime in which MoE improves the base results.
5.4 On Datm-Augmentatton
As shown in the previous section, MoE is sensitive the amount of data at hand. This sensitivity stems from the inherent structure of MoE, where increasing the number of experts results in each expert operating a smaller data fraction. Consequently, each expert tends to learn specific data clusters, as discussed by (Chen et al., 2022).
Modern training methodologies often employ robust data augmentation techniques to enhance model gener- alization. However, the interplay between these augmentations and the data clusters learned by the MoE model remains relatively uncharted. For instance, techniques like Mixup (Zhang et al., 2018), which blend images, could blur the distinctions between these clusters. Similarly, the Random Resize method, with its aggressive magnification capability, may distort the original data clusters. Understanding the precise influence of these augmentations on the MoE model's learning process is essential, as it holds implications for the model's ultimate performance. Some evidence of this can be seen in Tab. 9. While one training recipe might show MoE outperforming its dense counterparts (as indicated in the top rows of the table), a different, more effective recipe could lead the opposite outcome (Bottom rows of the table).
5.5 Alternate Views
With the ImageNet benchmark increasingly reaching saturation, the focus in contemporary computer vision models has shifted towards computational efficiency and scaling prowess. An essential question arises: Does the MoE model enhance this aspect in image classification? Our experiments with ImageNet reveal an intriguing insight: integrating MoF. into models like ConvNeXt and ViT enhances performance, but this
11

---

Published in Transactions on Machine Learning Research (10/2025)
Figure 6: The top row features the ConyNext-T-4 model, while the bottom row showcases the ViT-S-8 model, both of which have been trained on the ImageNet1k dataset. The images displayed have been generated by upsampling the gating output, retaining only the top expert for clarity. Each distinct color represents a unique expert. These specific images are samples from the ImageNet validation set, extracted from the last third stage of the ConyNeXt-T-4 and the final layer of the ViT-S-8 models, respectively.
improvement tends to plateau in models with over 100M parameters. At first glance, this might seem to contradict findings from some papers (Riquelme et al., 2021; Mustafa et al., 2022). However, with additional context, this observation aligns with the existing literature, as evidenced by results in Figs. 2 and 4, and Tab. 11 (App. B). Specifically, as the base dense network grows larger, the performance gap between MoE and its dense counterpart narrows.
Exploring this further in large-scale datasets, such as JFT billion (Zhai al., 2022), reveals similar findings. For instance, the vision transformer model SoViT, with 400M parameters (Alabdulmohsin al., 2023), exhibits comparable performance to the 15B V-MoE (Riquelme et al., 2021) when both use ViT- based architectures. Similarly, when considering the JFT-3B dataset (Zhai et al., 2022), the 5.6B LiMoE-H model (Mustafa et al., 2022) and Coca-Large (Yu et al., 2022) lead to analogous results when evaluated on a per-sample parameter basis. inglr,stingly, in the vision-language modeling domain, Lin et al. (2024) observe that increasing the number of experts for the image modality does not lead to diminishing returns, aligning with our findings regarding the efficacy of experts in scaling performance.
To summarize, while the MoE model does not appear to redefine the slate-of-the-arl when matched against dense models both ImageNet and billion-scale datasets, its real strength emerges in enhancing smaller models. By integrating MoE, these compact models can achieve better performance, pushing their capabilities closer to the forefront of current benchmarks. Yet, it is essential to acknowledge that, in the broader context, MoF might not drive significant. advanemeents in the overall state-of-the-art. Moreover, shape optimization as demonstrated SoVIT, could further improve performance and could even compound with the benefits of MoE, enabling smaller and more capable models. This nuanced perspective is further enforced in the appendix, with the data showcased in Tab. 11 in App. B.
Distribution
5.6 Model Inspection
0.15
Propartion
The objective of this section is to examine the routing component of the model. We will present the top expert for each routing layer. Note that when considering the top-2 configuration, we only report, for this visualization, the top-1 of these two experts.
0.10
0.05
0.00
2
3
5
Visual inspection: Fig. 6 provides a visualization of the portions of images that are assigned to the different experts. This reveals that some
Value
Figure 5: Distribution of experts
12

---

Published in Transactions on Machine Learning Research (10/2025)
100411
14
08
Fonkh
Fraeict
Pypess
0.6
(a) ViT-B deepest MoE. No absolute positional bias (b) ConvNeXt-B last MoE layer. A clear spatial par- is observed.
litioning of experts emerges.
Figure 7: Average spatial distribution of experts trained on ImageNet-222 22..
experts specialize in specific clements of image, such animals, buildings, or objects. This trend is especially apparent when considering the top-1 MoE, but the distinction becomes less clear with the top-2 MoF. Tn both instances, the clusters formed by each expert are challenging to interpret, and the spatial locations of each expert tend to lack clarity and can be quite ambiguous. However, when directly examining the spatial distribution, a clear spatial decomposition emerges for ConvNeXt models (cf. Fig. 7). In particular, Fig. 7b shows that certain experts focus specifically on the image borders or the center. This behavior may be explained by the strong local spatial bias imposed by convolution, in contrast the attention based models, which do not exhibit such patterns.
Number of experts involved per image: We quantified the average number of experts contributing to processing an image. In the initial layers, the majority of experts are typically engaged for cach image. With fewer than eight experts, most are utilized, even in deeper layers. For instance, in the ViT model, with eight experts, an average of seven experts contributed per image, while in the ConvNeXt model, there are six. As we increase the number of available experts per MoE layer, the number of contributing experts per image also rises. For example, in ConvNeXt, with 16 experts, an average of 10.5 experts are involved, and with 32 experts, this number increases to 15.8. This suggests that each expert is likely focused on limited number of image patches. This observation is confirmed by the cumulative distribution function (CDF) of each expert per image, as depicted in Fig. 8. On the other hand, when directly examining the routing distribution across experts, we observe a well-balanced assignment, indicating that the load-balancing loss is effectively fulfilling its role, as depicted in Fig. 5.
Correlation between experts and labels: Fig. (top) presents the correlation between the number of expert occurrences and individual classes in ImageNet- The class IDs in ImageNet- are organized in such a way that adjacent classes often share similar attributes. For instance, class IDs ranging from 151 to 268 are all dedicated to different dog breeds. From Fig, 9, note that the occurrences in the first three MoE layers do not match the ImageNet-1k classes, with most experts appearing uniformly across different classes. However, in the last three MoE layers, there is a noticeable trend of more experts aligning with specific ImageNet-1k classes. This distinction especially pronounced between experts focusing on animals (classes up to ID 397, such as experts 0 and 6 in Fig. 9-(e)) and those centered on objects (classes beyond ID 397, like experts and in Fig. 9-(e)). As the number of experts increases, this class-specific alignment still holds, though it remains challenging to clearly define the roles of these experts since they often span several teadd classes rather than forming light, contiguous groups that would fit each class, ralustrated in Fig. 10.
Expert Similarity: order to quantify the similarity between experts, we use a mothod derivod from the Jaccard index. For each image, we count the occurrences e; of each expert E, The similarity between two Σ EEinE o4tc--la--91 where E, O E, represents the images D E ctt
EEEE experts, E, and Ej, is defined as S₁ E,,E,
where both E, and E, appear together, while E, U E. refers to all images where either E, or E, present at least once. The absolute value term lq; - eyl acts as corrective factor to account for scenarios where expert may be deployed for minimal number of patches while the other is utilized for the majority,
13

---

Published in Transactions on Machine Learning Research (10/2025)
1.0
10-
1.0
0.8
Proportion
0.6
Proportion
0.6
Proportion
0.6
0.4
0.2-
0.2
0.0
150 Number patches
50
100
0.0
100
150
50
100
150
Number of patches
Number of patches
(a) experts
(b) 16 experts
(c) 32 experts
Figure 8: Cumulated Distribution Function of the number patches used per image for ConyNeXt-S trained on ImageNet-21k, with 8, 16, and 32 experts. Each colour corresponds to one expert. Sample detailed interpretation: for experts, the "red" expert sees between and 30 patches (out of 196) for 80% of the images which it is active. In particular (and even more when there many experts), expert frequently covers limited part of the image. For instance, if the y-axis is close to at x-axis=100 patches, covering more than 100 patches over 196 is rare: experts don't specialize much. As the number of experts increases, this tendency of experts being used only on a few patches increases too. For 32 experts, most of them focus on less than 30 patches; 80% of the time, they are active.
(8)
(b)
(c)
(d)
(e)
(f)
Figure 9: Routing analysis of VIT-S-8, trained on ImageNet-Ik. Top row: per-class occurrence of the experts, with the abscissa indicating the class ID and the ordinate representing the expert index (8 experts). Bottom row: similarity score between the experts. The graphs are arranged from left to right from the MoE layer closest to the input (a) to the layer closest to the output (f).
14

---

Published in Transactions on Machine Learning Research (10/2025)
0.46
1.31
0.20
2.45
0.24
42
0.34
8:12
6:16
0.49
92
0.39
2:43
0.24
3:074
2.45
0.33
8.12
0.45
1.18
6:27
3-054
0.31
9.24
0.14
0.074
0.32
2.32
9.45
0.30
5:34
0.30
1n0
200
300
400
500
800
700
800
900
1000
2
10
12
20
22
24
26
28
50
100
200
400
500
600
700
800
900
1000
Figure 10: ConvNext-S trained ImageNet-21k with 16, and 32 experts trained on ImageNet-21k routing analysis. The right columns display the frequency of each expert across different classes, where the x-axis represents class IDs, and the y-axis denotes the expert number. The left column provides a similarity score between experts.
indicating a lack of synergy between the two experts. In the bottom panel of Fig. 9, we observe distinct pattern in the utilization of experts across the stages of the MoE. In the initial layers, there is prevalent local image features across all the we analyze the network deeper, the frequency of expert co-occurrence diminishes. This shift indicates a transition in focus from local image elements to more semantically rich aspects, with individual experts concentrating on specific semantic attributes unique to certain images.
co-oceurrence of multiple experts, which highlights concentration images. Conversely,
Are experts well partitioned and clustered? For a general task such as ImageNet- there no clear, natural decomposition into experts, making it difficult precisely assess if expert decomposition is better than another, aside from comparing final accuracy. This comparison, however, provides limited insight into the quality of the experts. The literature (Chen et al., 2023; Mittal et al., 2022; Mohammed et al., 2022) suggests that, without explicit enforcement, MoE models struggle to separate tasks or find natural clusters beneath them. For instance, when a lask separation was explicitly incorporated into the loss, there was a marked improvement in both accuracy and the ease with which experts could be pruned for specific tasks (Chen et al., 2023). Similarly, Mittal al. (2022) found that the enforced decomposition resulted substantial gains, while the absence of such enforcement left the model unable to discover optimal partitions. Consequently, make three observations ImageNet. First, the number of experts increases, there is
15

---

Published in Transactions on Machine Learning Research (10/2025)
corresponding rise in the number of experts actively involved. Even with 32 experts, almost half of the experts are used per image in the last MoF layers. This seems counterintuitive since ImageNet is centered around objects; at least for deep layers, it should plateau at some point. Instead, experts are not well aligned with classes and spread widely, as depicted in Fig. 9. Second, an analysis of the cumulative distribution function (CDF) of experts' engagement (Fig. 8) reveals prevalent trend in which most experts are assigned less than patches of an image, equating less than 10% of the total image area, 50% of the time. Moreover, these patches are not necessarily contiguous. Third, when we look at the visualization of each expert, like in Fig. interpretation of each expert's 's role becomes challenging: An expert does not seem to adhere closely to an object, or to a part of the location of an object.
observe concentration of experts in the image for MoE layers near the output. The
Lack of Coherent Expert Partitioning
Qualitative and quantitative analyses reveal no consistent expert partitioning in vision MoE, partic- ularly as the number of experts increases. With few experts, routing weakly aligns with ImageNet classes, but the scale grows, expert allocation becomes diffuse, many experts activate per image, each covering only a small portion of patches, and long-range consistency across class IDs becomes less clear. ConvNeXt shows some spatial biases, whereas ViT shows none; in both, assignments remain well load-balanced yet not semantically disentangled, suggesting that the observed gains stem from added capacity rather than interpretable specialization.
6 Conclusion
Implementing effective Mixture-of- Experts (MoE) models for image classification tasks remains challenging and open problem. Our experiments show that, particularly in large-scale models, MoE for image classification offers limited gains over state-of-the-an methods in accuracy, robustness, generalization, or per-sample efficiency across both convolutional networks and vision transformers. This contrasts with NLP tasks, where MoE improves performance by enhancing capacity for knowledge storage and retrieval. In image classification, which relics more on feature extraction and extrapolation, MoE models show only modest benefits, mainly in smaller models with fewer activated parameters. These findings suggest that the effectiveness of MoF in image classification tasks is highly context- and scale-dependent, with advantages diminishing model capacity grows.
Acknowledgements
The authors thank the reviewers for their insightful and constructive feedback, which has substantially improved the quality of this work. AT. acknowledges support from the French National Research Agency (ANR) under Grant No. ANB-22-CCPJ1-0099-00
References
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Gelting VIT in shape: Scaling laws for compute-optimal model design. arXim:2305. 13035, 2023. 12, 21
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned- and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In TEEE/CV Conference on Computer Vision and Pottern Recognition, pp. 11828 11837, 2023. 15
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. Xrx:2:2208, 02813, 2022. 11
Zewen Chi, Li Dong, Shaohan Huang. Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 1-336000346133 2022.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv:2207.040 04672, 2022. 2
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In IEEE/CVF Conference on Computer Vision and Pottern Recognition Workshops, pp. 702- 202, 2020.
A. Dosovitskiy, L. Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkorcit, and N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020. 1,
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5233 5270, 2022. 2
Xumeng Han, Longhui Wei, Zhiyang Dou, Zipeng Wang, Chenhui Qiang, Xin He, Yinglei Sun, Zhenjun Han, and Qi Tian. Vimoe: An empirical study of designing vision mixture-of-experts. arXie preprint 14.:2410015792. 2024. 3
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of- distribution gencralization. In (EEE/CVE International Conference on Computer Vision, pp. 8340-8349, 2021a. 8, 10
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b. 8, 10
Heinke Hihn and Daniel A Bramn. Mixturecofvaaaatooaa-sppeet for continual learning. arXi::1100.12667, 2021. 2
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Ho Yuen Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixture-of-experts at scale. In D. Song, M. Carbin, and T. Chen (eds.), Machine Learning and Systems, volume 5, pp. 269-287, 287, 2023a. 2, 21
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale. Machine Learning and Systems, 5: 269 287, 2023b. 20
Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79 87, 1991. 2,
Jacob Devlin Ming-Wa Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pro-training of deep bidirectional transformers for language understanding. In NAACL-HLLT, volume 1, pp. 4171 4186, 2019. 1
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeycling: Training mixture-of-expert from dense checkpoints. 0rXiv:2212.05053 2022. 2
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv:2006. 166660000.16688, 2020. 1
Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Tyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient carly-fusion pre-training with mixture of modality-aware experts. 27X1::2407.21770, 2024. 12
17

---

Published in Transactions on Machine Learning Research (10/2025)
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVE /CV Conference Computer Vision and Pottern Recognition, pp. 2022. 2, 3, 4, 5, 20, 21, 22
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Cross-token modcling with conditional computation.
arXiv:2109.0P8 00000000,,0.....0000000000 2021. 2
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. is a modular architecture enough? Advances in Newral Information Processing Systems, 35:28747-28760, 28760, 2022. 15
Muqeoth Mohammed, Haokun Liu, and Colin Raffel. Models with conditional computation learn suboptimal solutions. In Can't Believe Il's Not Beller Workshop: Underslanding Deep Learning Through Empirical Falsification, 2022. 15
Basil Mustafa, Carlos Riquelme, Joan Puigeerver, Rodolphe Jenatton, and Noil Houlsby. Multimodal contrastive learning with LIMoE: the language-image mixture of experts. orXiv:2206.08 02770, 2022. 2, 12
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In 12th International Conference on Learning Representations, 2024. 3
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet In International Conference on Machine Learning, pp. 5389-5400, 2019. 8, 11
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Newral Information Processing Systems, 34:8583-8595, 2021. 2, 3, 10, 12, 21
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3))211-255) 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jell Dean. Outrageously large neural networks: The sparsely-gated mixturs-of-experts layer. 10X1:00000,0638, 2017. 1, 2,
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv:1906. 900,.024,, 2019.
Lya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MLP-Mixer: An all-MLP architecture for vision. Advonces in neural information processing systems, 1122261-222722 2021. 2
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021. 5, 11
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit III: Revenge of the ViT. In 1 77th European Conference Computer Vision, pp.
2022. 2, 4, 11, 20, 21, 22
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. European Conference on Computer Vision, pp. 459-479. Springer, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017. 1,
18

---

Published in Transactions on Machine Learning Research (10/2025)
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019. 8, 11
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXi::2205.01917, 2022. 12
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference Compuler Vision, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12104-12113, 2022. 12, 21
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. 4, 11
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation.
AAAI Conference Artificial Intelligence, 34(07):13001-1300; 2020. 4
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yamping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. 2iv:22202 28002000.00906, 2022.
19


Reviews:
[{"review_id": "yJHxExMog8", "reviewer": "Reviewer_WVPz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n-  (S1) The paper studies a question that has been asked before and continues to be asked - Do MoEs make a difference in the vision domain? The main finding that MoEs can make a difference but need nuance to understand their benefit in visual backbones is notable\n\n- (S2) The paper conducts a thorough analysis on classification tasks to support their main assertion on trade-offs involved with MoEs. The experiments pretrain networks on a dataset like ImageNet (21K) or JFT and then fine-tuned on a ImageNet1K task to obtain evaluation metrics.\n\n- (S3) The paper tests the robustness of trained models that on corrupted classification datasets and shows that using MoEs can be beneficial in this setting\n\n# Weaknesses\n\n- (W1) The paper determines the best architectural choice of integrating MoEs empirically on a in-domain dataset. This choice maybe brittle as I am unsure how the best strategy shown in the paper can extend to other image-based tasks\n\n- (W2) Related to above, the paper conducts its evaluation on classification tasks alone where a network is adapted based on fine-tuning. However, a dominant approach currently is to pretrain a backbone and conduct frozen features-based evaluation on downstream tasks. This can shed additional light on the trade-offs involved with MoEs\n\n- (W3) The paper lacks details on the pretraining task. What is the nature of the pretraining task?\n\n- (W4) (nit) Given the closed nature of JFT dataset, I would encourage the authors to consider using a public large scale dataset to help the community have a shot at reproducing their work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the use of (sparse) Mixture-of-Experts (MoE) layers in image-based backbones that are aimed at solving image tasks via fine-tuning. The main contribution is that the MoE layers are beneficial when used with small-to-moderate sized backbones (~80 million parameters) with the benefit of using  MoE layers diminishing beyond a certain point in terms of parameters count. The paper cconducts an thorough empirical analysis with ImagetNet and JFT datasets to support the main finding described above.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Q1) One approach to test the strength of a backbone is to use other image datasets for solving tasks like classification, detection, semantic segmentation etc. Given the hypothesis that MoEs ConvNext can specialize spatially, it would be interesting to check performance on detection or pixel-level tasks. \n\n- (Q2) Related to Q1 and W2, it would be very interesting to see how the hypothesis would hold under a frozen feature evaluation protocol instead of fine-tuning.\n\n- (Q3) Some of the notations used in the paper could be introduced in text. Specifically, things like \"ViT-S-8\" at a first glance may confuse a reader. Please include a section on naming in preliminaries, if possible\n\nI look forward to the authors' rebuttal. Note that the questions raised above are asked in the spirit of helping improve the paper. I am eager to hear back from the authors and get their perspective on the questions above.", "broader_impact_concerns": "No new concerns other than what might be noted in literature on vision-based ML research", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "stUquSSCdu", "reviewer": "Reviewer_C3G4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths\n\nThe paper offers actionable design insights for practitioners, where inserting MoE layers in the last two blocks for ConvNeXt and every second block for ViT yielded the most robust improvements. For example, the finding that using 4 experts for ConvNeXt or 8 experts for ViT is optimal, whereas pushing to 16 experts is counterproductive on ImageNet-1K, provides clear guidance for practitioners.\n\n#Weaknesses\n\nThis paper provides limited novelty over the V-MoE paper, which has already explored various configurations of MoEs for vision transformers. The scale at which the experiments are conducted are much smaller than the V-MoE paper as well, where the current work uses ViT-S and ViT-B on ImageNet-21k while the V-MoE paper goes up to ViT-H on JFT-300M. Also, the SoViT paper indicates that shape optimization and training on larger data (JFT-3B) is the key to improving the performance, but the current work does not consider shape optimization and trains on a small dataset (ImageNet-21k), so it is premature to draw conclusions such as “MoE for image classification offers limited gains over state-of-the-art methods in accuracy, robustness, generalization, or per-sample efficiency”.\n\nWhile the paper identifies what configurations work best, it offers less insight into the underlying reasons. For example, the heuristic “MoE in the last two layers” is empirically supported but there is no explanation as to why this is. The Discussion section notes that later layers are larger in a ConvNeXt so experts there yield more benefit, and that ViT’s uniform structure means MoE layers are “not particularly large” anywhere. This reasoning is plausible but a deeper analysis would strengthen the work. The authors also conclude that vision MoEs give limited gains because vision tasks rely more on feature extraction than memorization. However, this claim is somewhat speculative and is not directly validated by experiments. The paper would be more compelling if it included analyses of the learned experts or gating patterns. For instance, did different experts specialize in certain image features or classes? Was the load balancing loss effective in utilizing all experts uniformly?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a systematic study on incorporating Mixture-of-Experts (MoE) layers into modern vision models such as ViT and ConvNeXt for image classification tasks such as ImageNet-1k and ImageNet-21k. Key findings of the study indicate that gains diminish as MoE sparsity increases. Using too many experts or activating too large a fraction of parameters shows little benefit or even slight degradation. Furthermore, a comparison of gating mechanisms found that a standard learnable softmax gate slightly outperforms more complex routing schemes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should provide more explanation for certain design decisions, such as the use of MoE in the last two layers, and also the reasoning behind using Top-1 routing for ConvNeXt vs Top-2 for ViT.\n\nThe authors should include a brief analysis of how the experts are being used. For example, report the final load distribution across experts for a representative MoE model. It would also be interesting to know if experts in the ConvNeXt-MoE specialize spatially or by feature type. Since each expert is a convolutional block, do certain experts handle particular image regions or patterns?\n\nThe authors should mention any training stability measures used. For example, did they need to tune the auxiliary loss weight or use expert dropout to avoid collapse?. Also, elaborating on how the models were trained (data-parallel vs model-parallel for experts) would be useful for others attempting to reproduce or extend the work.\n\np.2 There is a typo where it says “ImageNet-2k” instead of “ImageNet-21k”.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xy0ZSoxtah", "reviewer": "Reviewer_t2so", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper addresses a timely and practically relevant topic: manually optimizing MoE for image classification without requiring billion-scale datasets.\n\n- It conducts a wide-ranging empirical analysis, covering multiple axes of design choices (e.g., layer position, number of experts, routing methods).\n\n### Weaknesses:\n\n- The paper suffers from significant structural and writing issues. The experimental findings are not clearly organized or summarized, making it difficult for readers to extract the key insights efficiently.\n\n- The experimental design lacks rigor in selecting candidates for evaluated design aspect. For example, when studying layer positions, the selected configurations (e.g., “every 2,” “last 2,” “last 3”) appear arbitrary and biased, with no comprehensive enumeration or justification.\n\n- There is limited analysis of the interaction between different design choices. For instance, the effect of expert count is analyzed independently from layer position, though these factors are likely interdependent.\n\n- While the paper reports many observations, few are distilled into actionable or generalizable insights. As a result, the overall contribution is limited.\n\n- The MoE inspection and visualizations, while informative, lack clear conclusions or takeaways, especially regarding the interpretability and functionality of the experts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a systematic empirical study on integrating Mixture-of-Experts (MoE) into image classification architectures, namely ConvNext and Vision Transformer (ViT), using the ImageNet-1K and ImageNet-21K datasets. The authors examine various design choices—including the number of experts, placement of MoE layers, and routing mechanisms—and report the performance-efficiency trade-offs associated with these decisions. The work aims to offer practical guidance for applying MoE to classification tasks with open datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following changes are suggested to improve the clarity and rigor of the paper:\n\n1. Improve organization and clarity of writing. As an empirical study, the paper would greatly benefit from a clearer structure such as:\n\n- State the motivating questions;\n\n- Summarize key findings upfront;\n\n- Explain experimental setup concisely;\n\n- Present results in focused sections;\n\n- Discuss insights and practical implications.\n\n2. Justify design decisions more thoroughly. For example, explain why “every 2,” “last 2,” and “last 3” configurations were selected for MoE placement. If a full sweep is impractical, at least justify the sampling.\n\n3. Consider interactions between factors. Rather than isolating one design choice while keeping others fixed, analyze how combinations of choices (e.g., number of experts vs. MoE placement) affect performance and trade-offs.\n\n4. Refine insights and reduce redundancy. Remove observations that are not actionable or insightful. Instead, focus on highlighting trends or guidelines that could inform future MoE designs.\n\n5. Revisit the model inspection section. The visualizations are interesting but currently inconclusive. Consider integrating them with quantitative measures or user studies to make stronger claims about expert behavior.\n\nA good and very relevant reference is [1]. The authors can learn from [1], and also need to justify the difference from [1] with larger contribution.\n\n[1] Han X, Wei L, Dou Z, et al. ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts[J]. arXiv preprint arXiv:2410.15732, 2024.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)