MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks
Authors: Xingkui Zhu
Yiran Guan
Dingkang Liang
Yuchao Chen
Yuliang Liu
Xiang Bai
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
computer vision
mixture of experts
weight initialization
fine-tuning
Primary area: machine_vision
TL;DR: We introduce MoE Jetpack, a framework that effectively converts existing pre-trained dense checkpoints into sparsely activated MoE models, substantially improve convergence speed and accuracy for vision tasks.

Abstract: The sparsely activated mixture of experts (MoE) model presents an effective alternative to densely activated (dense) models, combining improved accuracy with computational efficiency. However, training MoE models from scratch requires extensive data and computational resources, a challenge that limits their widespread adoption. To address this, we introduce MoE Jetpack, a framework designed to fine-tune the abundant and easily accessible dense checkpoints into MoE models. MoE Jetpack incorporates two key techniques: (1) **checkpoint recycling**, which initializes MoE models with dense checkpoints to accelerate convergence and enhance accuracy, minimizing the need for extensive pre-training; (2) the **hyperspherical adaptive MoE (SpheroMoE) layer**, which optimizes the MoE architecture to enhance fine-tuning performance and efficiency.
Experimental results indicate that MoE Jetpack doubles the convergence speed and enhances accuracy by 2.8% on ImageNet-1K. On smaller datasets, it achieves up to 8-fold faster convergence and over 30% accuracy gains, highlighting its efficiency.
The code is available at https://github.com/Adlith/MoE-Jetpack.

MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks
Xingkui Zhu" Yiran Guan"
Dingkang Liang Yuliang Liu Xiang Bai
Yuchao Chen
Huazhong University of Science and Technology
fadlith, yiranguan, dkliang, ylliu, xbai]@hust... cn
Abstract
The sparsely activated mixture of experts (MoE) model presents an effective ternative to densely activated (dense) models, combining improved accuracy with computational efficiency. However, training MoE models from scratch requires extensive data and computational resources, challenge that limits their widespread adoption. To address this, we introduce MoE Jetpack, a framework designed to fine-tune the abundant and easily accessible dense checkpoints into MoE models. MoE Jetpack incorporates two key techniques: (1) checkpoint recycling, which initializes MoE models with dense checkpoints to accelerate convergence and enhance accuracy, minimizing the need for extensive pre-training: (2) the hyper- spherical adaptive MoE (SpheroMoE) layer, which optimizes the MoE architecture to enhance fine-tuning performance and efficiency. Experimental results indicate that MoE. Jetpack doubles the convergence speed and enhances accuracy by 2.8% on ImageNet- K. On smaller datasets, it achieves up to 8-fold faster convergence and over 30% accuracy gains, highlighting its efficiency. The code is available https: / htt///gittut Tiggtt..b..... com Adlith/Moo Beee/MM--Jetpcck.
Introduction
Increasing model scale is key factor in boosting deep learning performance [1, 2, 3]. However, as models expand size, their computational demands surge, resulting in considerable slowdowns during both training and inference phases. A promising approach that decouples model size from computational costs is the sparsely activated mixture of experts (MoE) [4, 5, 6]. Unlike densely activated models (referred as dense models hereafter) [7, 8] that apply all network parameters to each input, MoE dynamically activates distinct parts of the model based on the input tokens. This allows for model scaling without substantially increasing the FLOPs thereby maintaining training and inference speeds during model upscaling. Recent advancements have seen successful implementations of MoE across various domains [11. 12, 13].
Despite their potential, MoE models face significant adoption challenges primarily due to the lack of pre-trained models. Unlike dense models, which benefit from vast collection of pre- trained resources available through platforms such as Hugging Face [14] and Timm [15], most MoE models must be trained from seratch with randomly initialized weights. This process demands substantial computational power and large datasets, limiting MoE research to a select few teams with the necessary resources. Consequently, our research aims reduce the training time and data
"quual contribution. Corresponding author.
FLOPSs means the floating point operations per second. The vanilla design of MoE does not inherently provide runtime advantages and requires additional parallelization strategies [9, 10] for acceleration. In our implementation, offer effective matrix multiplication method for parallelization, detailed in Appendix C.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pre-traincd
Spunsely Activated Mol:
ImageNet-
Dense Model
Emroodeed Patches
CIFAR-100
CIFAR-10
Embedked
VEM
Lx
Lx
Norm
Atlention
Ex
Ex
STL-10
129
R15
Food-101
485
N6
51.3
Exp.
Exp-
Exp-
46.
VIT (213)
MoE
Flowers
DTD
Soft MoE
Jetpack
MoB: Jetpack
Pets
(a) Equip Dense Checkpoint with MoE Jetpack
(b) The Performance Comparison Different Datasets
Figure 1: (a) MoE Jetpack converts dense checkpoints into initialization weights for MoE models, facilitating faster convergence and improved performance while maintaining equivalent FL.OPs. Here, Exp. represents individual experts, F. denotes the number of experts, and 7. indicates the total number of layers. (b) Performance comparison among ViT trained from scratch, pre-trained and fine-tuned VIT, Soft MoE [6] trained from scratch, and MoE Jetpack across multiple datasets.
requirements for MoE models by leveraging the pre-trained knowledge from dense checkpoints. We will specifically investigate whether utilizing dense checkpoints can enhance the accuracy and convergence speed of MoE models during fine-tuning.
In this paper, propose MoE Jetpack, points into MoE models. As illustrated in Fig. ((a), MoE Jetpack leverages the sunk cost of dense pre-training boost MoE performance and expedite convergence. It comprises two key techniques. Firstly, checkpoint recycling is used to initialize MoE models from dense checkpoints. Unlike sparse upcyeling [16], which simply duplicates the Multilayer Perceptron (MLP) to create experts, checkpoint recycling utilizes diverse dense checkpoints and strategic weight selection, providing flexibility and generating sigher-quality initialization weights for MoE models. The second tech- nique is the hy hyperspherical adaptive MoE (SpheroMoE) layer, which presents an optimized MoE architecture that facilitates the seamless integration of dense checkpoints and enhances fine-tuning performance. Existing MoE architectures, such as Switch Transformers [4] and Soft MoE [6], are not designed to incorporate pre-trained dense checkpoints, often resulting in optimization inefficiencies and seecseicaalizooonn during fine-tuning. The SpheroMoE layer mitigates these challenges by nor- malized token mixing, expert regularization, and adaptive dual-path mechanism, ensuring smoother integration and improved performance.
approach for fine-tuning pre-trained dense check-
By equipping dense checkpoints with MoE Jetpack, as illustrated in Fig. 1(b), the fine-tuned MoE models achieve significantly higher accuracy. Comprehensive evaluations across various image classification datasets of different scales further demonstrate the effectiveness of MoE Jetpack. In summary, our contributions are as follows:
We introduce checkpoint recycling, which pioneers the sampling of dense checkpoints to initial- ize MoE experts, enhancing initialization flexibility, diversifying experts, and eliminating the computational burden MoE pre-training.
We develop the spheroMot: layer, optimized for fine-tuning dense checkpoints into MoE architec- tures, alleviating optimization challenges, and preventing the over-specialiizaton of experts.
2 Background
In this section, we recap the core concepts needed to understand the MoE Jetpack, specifically focusing on the sparsely activated mixture of experts (MoE) and the routing mechanisms. Existing MoE models are typically derived from dense Vision Transformer (VIT) by replacing certain multilayer perceptron (MLP) layers with MoE layers. Each MoE layer consists of Router function, Router(2: 6 gatte). which directs input tokens to a set of "experts", where each expert is parameterized as MLP(; (:; 0,). Although the experts in MoE models are typically similar, the routing mechanisms vary and are critical to performance. Several routing algorithms have been developed, including top-k [17], BASE and Sinkhorn-B.. layers [18, 19], Hash layers [20], Expert Choice routing [2 and soft routing [6].

---

The most commonly used mechanism is top-k routing, which reduces computational overhead by selectively activating only the top-k experts most relevant the input tokens. The routing decision is computed as follows:
Router(æ; Ogate) = top-k (softmax(MMP((;; (gate)))
(1)
and the output is aggregated by combining the contributions of the selected experts:
y = x
Router(x;; (a;tter ) Expert(2; 8z).
REF
where denotes the weights, is the set of activated experts, and [R] = K. However, top-k routing presents challenges such imbalanced expert utilization, token dropping, and scalability issues.
more balanced and representative approach, Soft MoE [6]. effectively addresses these challenges through implicit soft assignments and serves experts, Soft MoE computes weighted combinations of all input tokens for each expert. Given the input tokens X Rmndd where m is the number of tokens and is their dimensionality, the learnable matrix & E [R project the tokens into x slots, where is the number of experts and s is the number of slots per expert. The projection is calculated as X softmax x X. Here, X E R (e))xd represents the transformed inputs that weighted combinations of X. Each MoE layer includes e expert functions {fr Rd ] with each expert handling s slots. The intermediate outputs Y E g(e-s)xd are obtained by applying the experts to the slots: Yi, fi().,) for i E (1....,ed } and j E (1,...,8) s}. Finally, the output tokens Y E Im are reconstructed by combining the expert outputs: Y = softmax(X4)
baseline. Instead of assigning tokens to specific
3 MoE Jetpack
In this section, we present the overarching concept of the MoE Jetpack. is divided into two phases: checkpoint recycling dense checkpoints to initialize MoE models and fine-tuning MoE models using the hyperspherical adaptive MoE (SpheroMoE) layer.
3.1 Checkpoint Recycling
Checkpoint recycling is a foundational phase in the MoE Jetpack framework, transforming pre- trained dense checkpoints (predecessors) into high-quality initialization weights (successors) for MoE models. This approach leverages the rich pre-trained knowledge from predecessors through weight reuse, boosting the performance and convergence speed of successors. The recycling procedure involves sampling a portion of the weights from the predecessors' multilayer perceptrons (MLPs) to construct experts, ensuring expert diversity and adaptability in expert size to meet varied needs.
To define the process checkpoint recycling (as illustrated in Fig. 2(a)), consider predecessors with N layers, where each layer L; has a feature dimension of d and hidden dimension of 4d. The object is to transform this predecessor into a successor MoE model S, which also has N layers, each denoted L4, but with reduced feature dimension d' < d. Following the Soft MoE architecture [6], the successor comprises two segments: a dense part with Nj layers and an MoE part with N₂ layers, where N1 N2 No Formally, the successor model represented as:
S (I (4N)))
Inspired by Weight Selection [22], our recycling process ensures consistency in feature dimensions. We explore four primary strategies to guide the recycling of checkpoints:
Importance-Based Weight Sampling (default): We select weights across hidden units and feature dimensions to construct initialization weights for diverse experts. For feature dimension selection, maintaining consistency across all layers is essential [22]. We achieve this by calculating the mean output features across the layers and selecting the top-d' dimensions based on these averages:
N
On
top-d' = argsort(O)]::)
1=1
3

---

st
Hidden
Dimension @ Selection
Expert Query
Hypersphere
Similarity Logils
Similarity
Impul Key
Canescan Noise
fofimax experts
softmax for tokens
Layer Norm
Expert
41
Drop
Combine
Dispunal
Eigperta
Attention
Layer Norm
Esperty
Inpul
Oulput
E Export
Tokens
E Expert
Tokens
(a) Checkpoints Recycling
(b) SpheroMoF Layer
Selects hidden units and dimensions from the dense checkpoint MLP, transforming pre-trained weights into diverse experts to initialize MoE models. (b) SpheroMoE Layer: Uses attssnattention to dispatch tokens slots and combine slots back into tokens. Input tokens are projected as keys, and random query is initialized to compute token-slot similarity in hyperspherical space. Experts then process their assigned slots.
Figure (a) Checkpoint Recycling:
where O is the output features of layer I,, and Ô Raxd the average output features. For hidden units, sampling is performed independently for each layer based on the magnitude of activations, as the arrangement of units does not impact the output. Activation values from batches of images used to form a probability distribution, and units are sampled accordingly:
An
H) EMCH Av"
hsuccessor P(h(H).
|hsacccsssee = 4d',
(5)
where An is the activation value of hidden units h. and H is the set of all hidden units. This method selects the most important weights for the successor model while promoting diversity among experts through the sampling process.
This strategy groups frequently co-activated hidden units into one expert. We construct a co-activation graph by counting the co-activations of hidden units in the testing procedure. Each unit vertex in the co-activation graph, and the edges represent their co-activation frequency. Formally, let G (V. E) be the co-activation graph, where V represents the hidden units and E represents edges with weights indicating co-activation counts. Using the Metis graph partitioning [23], we get several subgraphs:
Co-Activation Graph Partitioning:
G G. G (V..E. VOVV V = 0 for i 1=1
(6)
Experts are formed by the combination of sub- sub-graphs. This method leverages the natural grouping of hidden units, ensuring each expert captures specific functional subset of the predecessor model.
Uniform Weight Selection: Weights selected uniformly across dimensions and hidden units. For a predecessor with feature dimension d and a successor with dimension d', weights are chosen as:
(
(k)
i d i (0..... (O....,d'' 1}.
predoosssor!
d'
This method ensures an even distribution of the pre-trained weights across the successor MoE.
Random Weight Sampling: Weights randomly selected from the predecessor model. Let S be random subset of feature dimension indices:
S O.....d 1 |S| d'
(8)
Then, the weights for the successor are chosen as:
W (i)
W ()
jeS, iE to..... d' 1}.
(9)
saccessor
predecessor

---

Through the ablation in Sec. 4.3, Importance-Bact Weight Sampling is identified as the default method for recycling dense checkpoints to initialize MoE models.
Notably, the computational overhead introduced by Checkpoint Recycling is virtually negligible. Methods such as Random Sampling and Uniform Selection operate with minimal additional pro- cessing, as they directly select experts from the dense checkpoint without further computations. While Graph Partitioning and Importance-Based Sampling involve a preliminary inference step to determine neuron importance or co-activation patterns, the time required is minimal. For instance, performing inference on subset of 30, 000 images with an RTX 4090 takes less than 5 minutes. This efficient process ensures that Checkpoint Recycling significantly enhances model initialization while introducing almost no additional overhead.
3.2 SpheroMoE Layer
Following the initialization of MoE weights through Checkpoint Recycling. the next step is fine-tuning on downstream datasets. To enhance performance and sta- bility, we designed the hyperspherical adaptive MoE (SpheroMoF) layer (Fig. 2(b)), introducing three key improvements: SpheroMoE Routing to alleviate opti- mization challenges, Expert Regularization to prevent over-specialization, and Adaptive Dual-path MoE (Fig. 3) for better performance and efficiency. Additionally, the pseudo-code detailing these features' implementation be found in Appendix C.
Input Tokens
Slow
Token Dispatcher
less slots
Fast more slots
Core Dispatcher
Univ Dispatcher
Core Slots
Univ Slots
SpheroMoE Router
Heavy
Experts
Slot Combiner
Light Exports
Core Combiner
As shown in Fig. 2(b), the pro- posed hyperspherical MoE (SpheroMoE) routing mech- anism utilizes cross-attention [24] to distribute inputs across experts. Each expert receives an input slot, a weighted average of all input tokens. To maintain con- sistency between dense checkpoints and MoE layers M = (14) input tokens X E Rxxaxd (where b represents the batch size, n represents the token length, and d represents the input dimension) are layer nor- malized inherited from dense checkpoints, resulting in Xnor- Queries Q E (ex(s))) are randomly initial- ized and similarly normalized to align with Xnom- pro- ducing Quen- The layer normalization process ensures the consistency of distributions between the MoE model, input queries, and the pre-trained dense model. The nor- malized X are projected form keys K Rbxnxd for the cross-attention mechanism.
SpheroMoR Routing:
Univ Combiner
Output tokens
Figure 3: The Adaptive Dual-path
MoE structure enhances the SpheroMoE Router by adapting it into a dual-branch system, designed to optimize computa- tional efficiency and model performance. This configuration directs high-impact to- kens to a core path with fewer but larger experts while routing less critical tokens to a universal path equipped with a greater number of smaller experts.
To reduce numerical instability, Quomm and K are projected onto hyperspherical space using L2 normalization, ensuring that the resulting dot products reflect cosine similarities rather than unbounded magnitudes. This confines values within a stable range, preventing softmax saturation and enabling more balanced attention distributions, which improves model generalization. The similarity between Qnomn and K computed, yielding similarity logits (exx)x)) = Quem K". Input slots X (exx(x for experts are formed by a softmax operation along the n dimension of S:
exp(Sifk)
Xnam-
(10)
M=1 (Sp(S
Each expert processes its corresponding input slots X; independently, generating outputs Y, These outputs are then weighted by (after applying a softmax operation along the (e x s) dimension) to aggregate the experts' contributions, producing the final output Y E Roxxxd of the MoE layer:
(xp(S.jk) ΣS exp(S
Y.
(11)
5

---

In summary. SpheroMoE routing leverages layer normalization, hyperspherical projection, and cross- attention to effectively distribute imputs across experts, ensuring numerical stability and consistency with pre-trained dense models for improved optimization.
'oo improve generalization, SpheroMoF regularizes routing and expert behavior, preventing experts from over-specializing on specific inputs or outputs from depending excessively on certain experts. For the former, we introduce learnable sofumax temperatures Tiispatch and T Tombinee to precisely control token dispatch and slot combination, enabling smooth transitions between broad and focused attention. Since token dispatch distributes tokens across slots, while slot combination aggregates slot outputs back into tokens. these dual temperatures provide flexible control to each process. Both temperatures are initialized high to promote broad distribution of attention, preventing early specialization. As the training process, these temperatures adaptively decrease, allowing experts focus more precisely on relevant features and specialize where advantageous. Additionally, we added a certain level of normal noise to the similarity logits S, which improves generalization. For the latter, we utilized stochastic expert dropout, where each expert i is randomly deactivated with probability It ensures that single expert becomes a crutch for the entire output, promoting a more balanced utilization of all experts. These techniques form an expert regularization strategy that maintains expert versatility and mitigates over-fitting, ensuring the MoE model performs robustly on downstream datasets.
Expert Regularization:
To mitigate computational redundancy for less critical tokens and concentrate resources on essential ones, SpheroMoE Routing directs input tokens into core and universal slots based on importance. Building on this, the Adaptive Dual-Path structure is designed to assign each slot type to distinet pathway for optimized processing, as illustrated in Fig. 3. The core pathway consists of limited number of core experts with larger parameter counts, specifically configured for processing high-importance tokens. In contrast, the universal pathway comprises a larger set of small experts, each with one-fourth of the parameters of core experts, optimized for handling less critical slots, This dual-path configuration improves resource allocation by focusing computation on key tokens, thereby preserving model accuracy and enhancing processing efficiency.
Adaptive Dual-path MoE:
4 Experiments
4.1 Experimental Setups
Models. We validate our approach using the Vision Transformer (ViT)[7) and ConvNeXt[8] models. Specifically, we initialize the weight of V-JetMoE-T and C-JetMoE-F by transforming the dense checkpoints of VIT-S and ConvNeXt-T (pre-trained on ImageNet-21I and sourced from timm) through checkpoint recycling. As detailed in Sec. 3.1, V-JetMoF-T retains the dense layer structure of ViT-T in its first half, while the latter half is equipped with SpheroMoE layers. Each SpheroMoE layer consists of 2/2 core experts and N universal experts, where N is the number of imput tokens. Further details are in Appendix A.
Datasets. We evaluate MoE Jetpack on 8 image classification datasets, including ImageNet- 1K [25], CIFAR- 10, CIFAR- 100 [26], Flowers [27], Pets [28], STL-10 [29], Food-101 [30], and DTD [31]. These datasets encompass a diverse range of classification challenges, including object classification, fine-grained species recognition, and texture classification.
Baseline Implementation. We follow the implementation details outlined by Xu et al. [22] for comparisons of the dense models. For the MoE models, we employ Soft MoE [6] as the baseline and have replicated across all datasets. Our MoE Jetpack and Soft MoE utilize the same training strategies as the dense models to ensure comparison fairness. All implementations were executed using the MMPretrain framework [32] on RTX4090. More information can be found in Appendix
4.2 Main Results
Tab. 1 compares the performance of the MoE Jetpack with Dense ViT models and Soft MoE models on various image datasets using ViT-T (a) and ConvNeXt-F (b) architectures. All models maintain approximately the same number of FL.OPs. The columns in Tab. I are defined as follows:
Dense: Refers to dense models trained from scratch on each specific dataset.

---

Table 1:
Performance comparison on visual recognition tasks with VIT-T and ConvNeXt-F,
Dataset ())
Dense
Dense (2lk)
Soft MoE [6]
MoE Jetpack
Dense
Dense (21k)
Soft MoE [6]
MoE: Jetpack
ImgNet-1k
73.9
75.6
77.1
79.9 (+2.8)
76.1
76.4
79.1
80.5 (+1.4)
Food-101
79.6
86.9
82.0
89.5 (+7.5)
86.9
89.0
88.7
90.7 (+2.0)
CIFAR-10
92.4
97.0
92.9
97.9 (+5.0)
96.6
97.4
97.3
98.2 (-0.9)
CIFAR-100
72.3
81.4
75.9
88.4 (1
81.4
84.4
82.8
88.5 (15.7)
STL-10
61.5
83.4
67.7
95.3 (+22.66)
81.4
92.3
79.4
98.7 (+19.3)
Flowers
62.4
81.9
70.8
95.4 (+224.6)
80.3
94.5
83.3
98.6 (+15.3)
Pets
25.0
68.6
45.5
84.3 (+388.8)
72.9
87.3
77.4
94.9 (+17.5)
DTD
49.4
62.5
51.3
69.1 (117.8)
63.7
68.8
64.7
79.5 14.8)
(a) VIT-T
(h) ConvNeXt-F
Dense (21k): Represents dense models initialized with ImageNet-21K pre-trained weights, followed by fine-tuning on the target datasets.
Soft MoE: Reports the results of Soft MoE models trained from scratch on each dataset.
MoE Jetpack: Shows the performance of SpheroMoE models, initialized using checkpoint recycling with pre-trained dense checkpoints from ImageNet-21K, followed by fine-tuning on downstream datasets.
The MoE Jetpack, benefiting from the pre-trained knowledge embedded in dense checkpoints, consistently surpasses the performance of both Soft MoE models trained from seratch and dense models with ImageNet-21 initialization. These results underscore the effectiveness of MoE Jetpack.
4.3 Ablations
We perform ablation studies to assess the impact of various components and hyper-parameten within the MoE Jetpack. By default, we use a VIT-T model with the SpheroMoE layer integrated from layers 7 12, comprising 98 core experts and 196 universal experts (detailed in Appendix A). The Checkpoint Recycling method transforms dense checkpoints of ViT-S and ViT-T, pre-trained on ImageNet-21k, into initial weights for our V-JetMoB-T model.
Effect of MoE Jetpack Components. We conducted the ablation of two key components of the MoE Jetpack three datasets. As shown in Tab. 2, integrating Checkpoint Recycling with the Soft MoE baseline significantly improves performance across all datasets, with a mean accuracy increment of 10.2%. The SpheroMoF layer further enhances performance, achieving of 87.9%. These results demonstrate the efficacy both components, especially when used together, highlighting their synergistic effect in boosting performance.
accuracy
Checkpoint Recyeling vs. Sparse Upeyeling. To compare the four checkpoint recycling strategies men- tioned in Sec. 3.1 and the method of using duplicated MLPs to construct experts in Sparse Upcycling [16], conducted experiments on ImageNet. For fairness, also employed SpheroMoE layer in the Sparse Up- eycling. The results, summarized in Tab. 3, show that Importance-Bast Based Sampling achieves the highest performance, demonstrating its
Table 3: Checkpoint Recycling Sparse Upeycling
Method
Construction
ImageNet
Sparse Upcycling [16]
Copy
79.1
Random Sampling
79.5
Checkpoint Recycling
Uniform Selection
79.6
Graph Partitioning
79.8
Importance- based Sampling
79.9
Table 2: Ablation Study on MoE Jetpack Components.
Soft MoE [6] Checkpoints Recycling SpheroMoE
ImageNet CIFAR-100 Flowers
Mean Acc.
Baseline ViT-T
73.9
72.3
62.4
69.5
77.1
75.9
70.8
74.6 (45.1)
78.4
84.7
91.2
84.8 (415.3)
79.9
88.4
95.4
87.9 (+18.4)

---

Table 4: Effectiveness of SpheroMoE with the Dual-Pate Structure.
Model
Experts
ImageNet
FLOPs (G)
Soft MoE
197 core: 197, univ: 0
78. 4
1.2
SpheroMoFF
79.6
1.2
SpheroMoE
core: 98, univ: 196
79.9
1.1
Table 5: Comparison of Model Variants with Different Configurations
model
Weight Init.
MoB Layers
Expert Number
Param (M)
FLOPs (G)
CIFAR- 100
ImageNet
VIF-T Soft MoE-T [6]
6
1.1
72.3
73.9
7:12
197 197
354
1.2
75.9
77.1
Soft MoE-S [6]
7:12
1412
4.5
77.5
80.3
VITT
6
1.1
81.4
75.5
V-JerMoB-T
V
11:12
core: 98, univ: 196
92
1.1
87.4
V-JetMoE-T
9:12
core: 98, univ: 196
179
1.1
87.8
V-JetMoE-T
5:12
core: 98, univ:
352
1.2
86.7
V-JetMoE-T
7:12
core: 32, univ: 64
89
0.8
87.8
V-JetMoE-T
7:12
core: 64, univ: 128
175
88.0
V-JetMoE-T
7:12
corc: 98, univ: 196
265
1.1
88.4
79.9
V-JerMoB-S
7:12
core: 98, univ: 196
1058
4.3
89.9
82.4
effectiveness in leveraging critical weights to enhance model performance and convergence speed. Additionally, Checkpoint Recycling is highly flexible, allowing the construction of experts of varying sizes to meet different needs, feature not provided by sparse upcycling.
Effect of Dual-Path Structure. The Tab. presents an ablation study on the effectiveness of the dual-path structure in SpheroMoE. When configured with 197 core experts only (no universal experts), SpheroMoE achieves higher accuracy on ImageNet-18 compared to Soft MoE with the same number of experts. Introducing the dual-path structure with 98 core experts and 196 universal experts (each with one-fourth of the parameters of core experts) further enhances accuracy to 79.9 while reducing the computational cost 1.1G FLOPs. This result highlights the efficiency of the dual-path structure, which allows SpheroMoE to allocate resources adaptively and achieve better performance without increasing the overall FLOPs.
To evaluate the effectiveness of the Adaptive Dual- path MoE structure introduced in Sec.3.2 and to identify the optimal ratio between core and universal ex- perts, we conducted ablation on the CIFAR-100 dataset. With a fixed to- tal number of experts, varied the ratio of core experts to find the ideal
Core Experts Ratio.
5/6
Core Expert Ratio
2/3
1/2
1/3
balance between performance and re- source allocation. As shown in Fig. 4, the highest accuracy is achieved when core experts constitute approximately 1 1/3 of the total experts.
1/6
85.5
86.5
87.5
88.5
Cifar- 100 Accuracy
Figure 4: CIFAR-100 accuracy across different ratios of core (dark) to universal (light) experts, highlighting optimal performance at 1/3 core ratio.
MoE, Jetpack Configurations. This part evaluates the impact of various MoE Jetpack configurations on model performance, as summarized in Tab. 5. The experiments focus on the placement of SpheroMoE layers, the number of experts per layer, and the base size of converted dense checkpoints. Results indicate that more SpheroMoE layers generally enhance performance, though placing it before layer slightly hurt the performance. Consequently, SpheroMoE layers were incorporated into layers 7-12 Additionally, models with more experts exhibit improved accuracy, highlighting the benefits of increased expert specialization and diversity. Models converted from larger dense checkpoints demonstrate superior performance. These findings suggest that MoE network performance can improved by increasing number MoE layers, incorporating more experts, and utilizing larger base models.

---

80
ImageNer- Accuracy
2
Faster
CIFAR Accuracy
80
60
Faster
60
40-
40
20
V-etMoE-T
20-
V-JetMoE-T
Randimm Init MoE
Random Init Moc
100
150
200
250
300
50
100
150
200
250
300
Epoch
Epoch
Figure 5: Comparison of convergence speeds using MoE Jetpack versus training from scratch ImageNet (left) and CIFAR- 100 (right). MoE Jetpack achieves target accuracies significantly faster, demonstrating 2x speed increase ImageNet and an 8x increase CIFAR-100.
Inpus Image
Expert
Expent
Core Expert
Universal Expert
Mail Layer 07 caw
Maii: Layer uple
Maii
Layd
Malll
Mall:
5438 Luyer cole
Muul: Suyer upa
12.
10.
4
Expert
7.5
2-
200 339
(n) Attention Maps Different Experts
(b)
different exports the output tokens
Figure (a) The attention maps generated by five experts in response to the input image, highlighting the experts' specialization. (b) These line charts show varying contributions of core and universal experts, with core experts' influence peaking in later layers, emphasizing their detailed feature refinement, contrasted with the consistent input of universal experts.
4.4 Analysis
We analyze the impact of MoE Jetpack on convergence speed for MoE models fine-tuned ImageNet- 1K and CIFAR-100 dataset. Additionally, we offer insights into expert attention patterns and the contribution of each expert the output tokens.
Accelerating MoE Convergence with MoE Jetpack. The effect of MoE Jetpack on convergence speed is illustrated in Fig. 5 for ImageNet (left) and CIFAR-100 (right). In both cases, models with MoE Jetpack achieve target accuracy significantly faster. For ImageNet, MoE Jetpack enables the model to reach approximately 77% top-1 accuracy within 150 epochs twice fast training from scratch. This acceleration is even more pronounced on smaller datasets like CIFAR-100, where MoE Jetpack achieves 76% top-1 aceuracy by around 40 epochs, eightfold improvement the baseline. These results underscore MoE Jetpack's 's efficiency in accelerating convergence, reducing fine-tuning time and computational demands.
We visualize the attention maps experts in Fig. 6(a). which illustrates that different experts focus different parts of the input image. This diversity in attention suggests that each expert specializes in capturing unique aspects of the input, enhancing the model's ability to represent features comprehensively, The specialization allows the MoE model to combine multiple perspectives, resulting in more robust and detailed understanding of the input.
Intuition of Expert Attention Patterns.
Contribution of Each Expert to Final Results. Fig. 6(b) demonstrates the varying contributions of core and universal experts across different layers of the MoE model. Core experts show an increasing influence in the later layers, emphasizing their role in refining specific and highly relevant features. Additionally, the contributions among core experts are markedly uneven, some experts can impact output tokens 1 7x more than others, reflecting greater specialization and diversity in their focus areas.

---

In contrast, universal experts maintain a relatively consistent contribution level, indicating a more uniform integration of broader contextual information throughout the network. This hierarchical structure, balancing the specialized refinement by core experts with the generalized understanding provided by universal experts, enhances the model's overall performance and robustness.
Related Work
Sparsely activated Mixture of Experts (MoE). Scaling Laws [33] indicate that increasing model parameters can enhance performance. However, traditional densely activated models (dense mod- els) [7, 8] activate all parameters for every input, resulting in high computational costs as models scale. In contrast, MoE models [12, 34, 35, 36] activate only a subset of parameters for specific input tokens, enabling efficient scaling to trillions of parameters with sublinear increases in computational costs [37, 5, 4]. To optimize input token allocation among experts, various routing mechanisms have been developed. BASELayer [18] formulates token-to-exxe exepert allocation as linear assignment problem, while BC-CF2 [21] propose expert choice routing, soft routing methods like SMEAR [38], and Soft MoE [6] implicit soft assignments involving all tokens. However, few studies explore leveraging dense model checkpoints to accelerate MoE training [16].
Knowledge transfer with pre-trained models. Knowledge transfer occurs between identical or distinct models. Pre-training followed by fine-tuning is well-established for identical models, utilizing large datasets through supervised learning (e.g.. ImageNet21K [25]. JFT-300M [39]) or self- supervseee methods (e.g.. BERT [40]. CLIP [41], MAE [42], DINO [43], EVA [44, 45]). These approaches produce foundation models with broad applicability, and subsequent fine-tuning con- sistently improves performance. For distinct models, knowledge distillation [46] trains a smaller student model mimic the larger teacher model, enhancing efficiency. Additional strategies include weight pruning [47, 48, 48, 49, 50], which removes redundant parameters, and weight selection [22] initializes smaller model with a subset of weights from a pre-trained larger model.
Research on transferring knowledge from dense checkpoints to MoE models is limited. MoEfica- tion [51] partitions& dense model into MoE components, while Sparse Upcycling [16] replicates a dense model multiple times to form a MoE model. Our MoE Jetpack recycles important weights from larger dense checkpoints to initialize experts of various sizes, combining the flexibility knowledge transfer across distinct model types with the efficiency of transfer between identical model types.
6 Conclusion
In this paper, we introduced MoE Jetpack, novel framework for fine-tuning pre-trained dense checkpoints into Mixture of Experts model. Our approach leverages checkpoint recycling, which inherits the knowledge of open-source dense checkpoints and the hyperspherical adaptive MoE (SpheroMoE) layer to enhance fine-tuning performance. These innovations contribute to improved convergence speed and model accuracy. The MoE Jetpack significantly improved various visual tasks while maintaining computational efficiency.
The limitation of our approach lies in its reliance on the quality of pre-trained dense checkpoints; inadequately trained or poorly generalized dense models may hinder performance improvements. Additionally, while our experiments focused on visual tasks, further research is needed to validate the generalizability of MoE Jetpack across other domains, such natural language processing and reinforcement learning. Future work may address these limitations, further enhancing the scalability and robustness of the framework, and broadening MoE applicability across diverse tasks.
Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant Nos. U2341227 and 622261044 104).
10

---

References
[T] A. Krizhevsky, I. Sutskever, and G. E., Hinton, "Imagenet classification with deep convolutional neural networks," Proc. of Advances in Neural Information Processing Systems, vol. 25, 2012. 1
[2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proc. of IEEE Inil. Conf. on Computer Vision and Pattern Recognition, 2016, pp. 770-778.
[3] X. Zhai, Kolesnikov, N. Houlsby, and L. Beyer, "Scaling vision transformers," in Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2022, pp. 12 104-12 113. 1
[4] W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," Journal of Machine Learning Research, vol. 23, no. 120, pp.
1-39, 2022. 1, 2, 10
[5] C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, "Scaling vision with sparse mixture of experts." " Proc. of Advances in Neural Information Processing Systems, vol. 34, pp. 8583-8595, 2021. I, 10
[6] J. Puigcerver, C. R. Ruiz, B. Mustafa, and N. Houlsby, "From sparse to soft mixtures of experts," in Proc. of Intl. Conf. on Learning Representations, 2023. 1, 2, 3, 6, 7, 8, 10
[7] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition scale," in Proc. of Intl. Conf. on Learning Representations, 2020. 1, 6, 10
[8] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2022, pp. 11 976--1 1 986. 1, 10
[9] Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, Z. Wang el al., "M Mixture- of-experts vision transformer for efficient multi-task learning with model-accelerator co-design," Proc. of Advances in Neural Information Processing Systems, vol. 35, pp. 28 4814-288457 457, 2022.
[10] S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, "Deepspeed-moes Advancing offexereof-eeppetts inference and training to power next-generation ai scale," in Proc. of Imll. Conf. on Machine Learning. PMLR, 2022, pp. 18 332-18 346.
[11] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, F. Bressand et al., "Mixtral of experts," arXiv preprint wxXiv2401.04088 2024.
[12] D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu et al., "Deepseekmoe: Towards ultimate expert specialization in cexture-of-expertt language models," arXiv preprint arXiv:2401 .06066, 2024. 1, 10
[13] B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Zhang, M. Ning, and L. Yuan, "Movallava: Mixture of experts for large vision-language models," arXiv preprint arXhv:2401.1594,, 2024.
[14] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., "Huggingface's 's transformers: ffate-of-the-ar natural language processing," arXiv preprint 191.v:191. 2019. 1
[15] R. Wightman, "Pytorch image models," http://gthhhbboom///gggaaayyyyoh 2019.
[16] A. Komatsuzaki, J. Puigcerver, J. Lee-Thory C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby, "Sparse upcycling: Training mixture-of-experts from dense checkpoints," in Proc. of Intl. Conf. Learning Representations, 2022. 2, 7. 10
[17] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q Le, G. Hinton, and J. Dean, "Outrageously large neural networks: The sparsely-gates -atted mixture-of-experts layer," in Proc. of Intl. Conf. on Learning Representations, 2016. 2
11

---

[18] M. Lewis, S. Bhosale, T. Detimers, N. Goyal, and L. Zettlemoyer, "Base layers: Simplifying training of large, sparse models," in Proc. of Intl. Conf. on Machine Learning. PMLR, 2021,
pp. 6265-6274. 2, 10
[19] A. Clark, D. de Las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hecht- man, T. Cai, S. Borgeaud et al., "Unified scaling laws for routed language models," in Proc. of Intl. Conf. on Machine Learning. PMLR, 2022, pp. 4057-4086. 2
[20] S. Roller, S. Sukhbaatar, J. Weston et al., "Hash layers for large sparse models," Proc. of Advances in Neural Information Processing Systems, vol. 34, pp. 17 155-111 566, 2021. 2
[21] Y. Zhou, T. Lei, H. Liu, N. Du, Y. Huang, V. Zhao, A. M. Dai, Q. V. Le, J. Laudon al., "Mixture-of-experts with expert choice routing," Proc. of Advances in Neural Information Processing Systems, vol. 35, pp. 7103-7114, 2022. 2, 10
[22] Z. Xu, Y. Chen, K. Vishniakov, Y. Yin, Z. Shen, T. Darrell, L. Liu, and Z. Liu, "Initializing models with larger ones," in Proc. of Inll. Conf. on Learning Representations, 2024. 3, 6, 10
[23] G. Karypis, "Metis: Unstructured graph partitioning and sparse matrix ordering system," Tech- nical report, 1997. 4
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Proc. of Advances in Neural Information Processing Systems, vol. 30, 2017.
[25] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition. Icce, 2009, pp. 248-255. 6, 10
[26] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," 2009.
[27] M.-E. Nilsback and A. Zisserman. "Automated flower classification over large number of classes," in Indian conference on computer vision, graphics & image processing. IEEE, 2008, pp. 722-729.
[28] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, "Cats and dogs." in Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 3498-3505. 6
[29] A. Coates, A. Ng, and H. Lee, "An analysis of single-layer networks in unsupervised feature learning." " in Proceedings of the international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 201 1, pp. 215-223, 6
[30] L. Bossard, M. Guillaumin, and L. Van Gool, "Food- 101 111-mining discriminative components with random forests," in Proc. of European Conference on Computer Vision. Springer, 2014,
pp. 446-461. 6
[31] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures in the wild," in Proc. of IEEE Intl. Conf. Computer Vision and Pattern Recognition, 2014, pp. 3606-3613.
6
[32] M. Contributors, "Openmmlab's 's pre-training toolbox and benchmark." https://gituu..com/ open- mmlab/mmmpretrrin, 2023.
[33] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv2201..03661 2020. 10
[34] B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Zhang, M. Ning, and L. Yuan, "Moc-llava: Mixture of experts for large vision-language models," " arXiv preprint arXiv:2401.15947 2024. 10
[35] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. 1. Casas, E. Hanna, F. Bressand et al., "Mixtral of experts," arXiv preprint arXiv:2401.cm 04088. 2024. 10
12

---

[36] X. Wu, S. Huang, and F. Wei, "Mole: Mixture of lora experts," in Proc. of Intl. Conf. on Learning Representations, 2023. 10
[37] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, "Gshard: Scaling giant models with conditional computation and automatic sharding," " in Proc. of Inil. Conf. on Learning Reprexentations, 2020. 10
[38] M. Muqeeth, H. Liu, and C. Raffel, "Soft merging of experts with adaptive routing," arXiv preprint arXiv:2306.03744, 2023. 10
[39] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, "Revisiting unreasonable effectiveness of data in deep learning era," in Porc. of IEEE Inil. Conf. on Computer Vision, 2017, pp. 843-852. 10
[40] J. D. M.-W. C. Kenton and L. K. Toutanova, "Bert: Pre-training of deep bidirectional trans- formers for language understanding." in Proceedings of NAACL-IILT, 2019, pp. 4171-4186. 10
[41] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," in Proc. of Intl. Conf. on Machine Learning. PMLR, 2021, pp. 8748-8763. 10
[42] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, 2022,
pp. 16000-16 009. 10
[43] M. Oquab. T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. HAZIZA, F. Massa, A. El-Nouby et al., "Dinov2: Learning robust visual features without supervision," Transactions on Machine Learning Research, 2023. 10
[44] Y. Fang, Q. Sun, X. Wang, T. Huang, X. Wang, and Cao, "Eva-02: A visual representation
for neon genesis," " arXiv preprint arXiv:2303. 11003.11333 1131, 2023. 10
[45] Q Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, "Eva-clip: Improved training techniques for clip at scale," arXiv preprint arXiv:2303. 2023. 10
[46] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," 2014. 10
[47] S. Han, J. Pool, J. Tran, and W. Dally, "Learning both weights and connections for efficient neural network," " Proc. of Advances in Neural Information Processing Systems, vol. 28, 2015. 10
[48] S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefter, and J. Hensman, "Slicegpt: Compress large language models by deleting rows and columns," in Proc. of Intl. Conf. on Learning Representations, 2023. 10
[49] M. Xia, T. Gao, 7. Zeng, and D. Chen, "Sheared llama: Accelerating language model pre- training via structured pruning," in Proc. of Intl. Conf. on Learning Representations, 2023. 10
[50] Yu, K. Huang. M. Wang, Y. Cheng, W. Chu, and L. Cui, "Width & depth pruning for vision transformers," in Proc. of the AAAI Conf. on Artificial Intelligence, vol. 36, no. 3, 2022, pp. 3143-3151. 10
[51] Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou, "Moefication: Transformer feed-forwart layers are mixtures of experts." " in Findings of the Association for Computational Linguisties: ACL 2022, 2022, pp. 877-890. 10
[52] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization." in Proc. of Intl. Conf. on Learning Representations, 2018. 15
[53] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "Randaugment: Practical automated data augmentation with reduced search space," " in Proc. of IEEE Inil, Conf. on Computer Vision and Pattern Recognition workshops, 2020, pp. 702-703. 15
13

---

[54] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Pazz "mixup: Beyond empirical risk minimization," in Proc. of Intl. Conf. on Learning Representations, 2018. 15
[55] Yun, Han, Oh, Chun, Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," in Porc. of IEEE Inil. Conf. Computer Vision, 2019, pp. 6023-6032 6032. 15
[56] Z. Zhong, L.. Zheng. G. Kang, S. Li, and Y. Yang, "Random erasing data augmentation," in Proc. of the AAAI Conf. Artificial Intelligence, vol. 34, 07, 2020, pp. 13 000-13 008. 15
[57] C. Szegedy, V. Vanhoucke, S. Ioffe, Shlens, and Wojna, "Rethinking the inception archi- tecture for computer vision." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. pp. 2818-2826. 15
[58] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou, "Going deeper with image transformers." in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 32-42. 15
14

---

A Detailed Model Configurations
In this section, present the detailed model configurations for the main experiments in Sec. 4 in Tab. 6. We refer to pre-trained dense checkpoints predecessors and the derived MoE models as successors. We use ImageNet-21k pre-trainn pre-trained predecessor from timm with our Checkpoint Recycling algorithm to generate initialized weights for the successor.
Table 6: Configurations for Models.
Configuration
Successors
Predecessors
Model
V-JetMoF-T
C-JetMoF-F
ViT-S/16
ConvNNxt-T
F1.OPs (G) Initialization
1.1
1.1
1.1 ImageNet-21k
1.1 ImageNet-21k
Checkpoint Recyeling Checkpoint Recycling
MoE Layers
7:12
10:18
Core Expert Number
98
[98, 24] [196, 48]
Universal Expert Number
196
B Experiment Settings and Time Costs
In this section of the appendix, provide comprehensive description of the training settings used in our experiments. Tab. outlines the standard training configuration utilized across our experiments. Tab. 8 details the sptasct- training configurations, capturing variations in batch size, warmup epochs, total training epochs, and drop path rates for each dataset employed in our experiments.
Our experiments were conducted on RTX 4090 GPU. Training V-JetMoE-T on the CIFAR-100 dataset (60,000 images) required 2.5 GPU hours while training on the ImageNet-IE IK dataset (1,281,167 images) required 120 GPU hours. Training C-JetMoB-F on CIFAR-100 also required 2.5 GPU hours and 156 GPU hours on ImageNet- I. For V-JetMoE-S,, training on CIFAR- required GPU hours and 200 GPU hours on ImageNet- I. Compared to the original dense models (ViT-Tiny, ConvNeXt-Femtto. ViT-Small), our method achieves nearly equivalent training times.
For all the experiments presented in our paper, we required 3, 300 GPU hours for training. In total, we spent approximately 8, 000 GPU hours for exploration and validation of our work.
Table 7: Our basic recipe for model training.
Training Setting
Configuration
image resolution
224 x 224
Table 8: Hyper-parameter setting on ViT-T.
optimizer
Adam W[52]
base learning rate
10-3
Batch
Warmup Epochs
Training Epochs
Drop
Setting
weight decay
0.05
Size
Path Rate
optimizer momentum 31, 82 = 0.9, 0.999
C-10
512
50
300
0.1
batch size
4096
C-100
512
50
300
0.1
training epochs learning rate schedule
300
Pets
512
100
600
0.1
cosine decay
Flowers
512
100
600
0.1
warmup epochs
50
STL-10
512
50
300
0
warmup schedule
linear
Food101
512
50
300
0.1
randaugment [53]
(9, 0.5)
DTD
512
100
600
0.2
mixup [54]
0.8
INIk
4096
50
300
0
cutmix [55]
1.0
random erasing [56]
0.25
label smoothing [57]
0.1
layer scale [58]
1 10-6
15

---

C Implementation of SpheroMoE Layer
Algorithm 1: Simple implementation of SpheroMoE.
def parallel_ aewalloeeeepert_foowwadd.. experts) uun
Traditional MoE models use for loop to process each token through the experts.
By merging all expert weights into large matrix, implementation allows for single matrix multiplication operation each layer across all tokens and experts, replacing multiple individual operations.
einsum(x, experts. experts.weight_1.. ebe d1, d2 d1 -> b d2")
Dearrange(expertaa bias_1, "e d2
() d2")
x experts.act
x einsum(x, experts. weight_ _2, "bes d2, d1 d2 -> bbes d1")
rearrange (experts. rearrangeexxperts. bias_2, "e d1 - 0 O d1") return
def spheromoe. Mayeromoe_layer(X (x, Q, T, core_experts, mniveexpetss):
Performs the Spheromoe layer operation.
Parameters:
(tensor): tensor with shape (batch, token_num channel).
Q (tensor): tensor with shape (expert_num) slots_ped expert, _ee__exeet, channel).
T (tensor): the learnable temperature list for the softmax function.
core_experta, univ_experta (expert): expert weight for MoE layer.
Returns:
tensor: Output tensor after applying the Spheromoe layer operations.
X_norm layer_normi dim=-1)
Q_norm 12_norm(()
K 12_norm(K normmoK_prooeee project (X_norm))
Compute similarity logits.
logits einaum(K, Q_norm, "b a edd-bbee
create normal noise noise normal_noise noieelltiis self noise_mult
Apply softmax to similarity logits.
Dispatch = (Logi//Too)))) dim=1) noise
Combine softmax (logits/T[1), dim=[-1,-2]] + noise
Token dispatch.
X_hat einsum (Dispatch, X_norm, "b dd bnes __core = X_hat[: :core_mum pere__um, ;, :]
x univ X_hat[:, core_num:, ;, :]
Using core experts universal experts processes each slot.
Y hat stack([
laallee__xxert_forwaaee errallel_experttc. (otprtt__oowrrdx___iiit umiv_experts) dim=1)
core_experts),
# Expert dropout.
Y_hat expert_drop('' hat)
Token combine.
Y einsum(Combinn Y hat, "b n Abnes bead b dnd)
return Y
16

---

D Dynamic Allocation and Focus Regions of Experts in MoE Jetpack
In this section, we discuss the dynamic allocation and focus regions of core and universal experts across different layers of MoE Jetpack. We used the same test images in the main text, visualizing the focus regions of the most important (i.e., those with the highest output contribution) core and universal experts for each MoE layer in Fig. 7. The corresponding contribution values for these experts are listed in Tab. 9.
Our findings are follows: Initially, in the shallower network layers (MoE Layer 7 and 8), the core experts contribute less than the universal experts, and their focus regions are relatively dispersed. As the network deepens, in MoE Layer 9, the most important core and universal experts show similar contribution values and focus regions. With further depth (MoE Layers 10, 11, and 12), the dominance of the core experts becomes increasingly evident, with significantly higher contribution values than the universal experts. Core experts focus on prominent objects in the images and are inclined to capture global information.
These experts' dynamic allocation and different focus region tendencies are crucial to our method. Different experts have varying capabilities in extracting information at various granularities, and the network facilitates collaboration among these experts to produce the final output. This illustrates the effective utilization of expert diversity in the MoE model.
Most Important
Most Important
Most Important
Most Important Universal Expert
Core Export
Universal Expert
Core Expert
MoE Layer 07
MoE Layer
MoE Layer
MoE Layer 11
MoE Layer 09
MoBi Layer 12
Figure Visualization of the attention map identified by the most important core experts and universal experts across different layers (MoE Layer 07 to MoE Layer 12). The images show the regions deemed most relevant by each type of expert at each layer.
Table Contribution values of core and universal experts across network layers.
MoE Layer
Core Expert Contribution
Universal Expert Contribution
7
1.71
3.91
8
2.52
4.16
9
3.78
3.77
10
8.17
6.71
11
17.66
2.12
12
7.36
0.77
17

---

E
Broader Impacts
The proposed MoE Jetpack framework significantly enhances the accessibility and efficiency of MoE models by utilizing pre-existing dense checkpoints to substantially reduce the computational costs associated with training these models from scratch. This method not only minimizes the environmental footprint by decreasing the reliance on extensive GPU resources but also bridges resource gap, facilitating wider adoption and fostering innovation across the community. Additionally, our commitment to open-sourcing all experimental code promotes greater transparency and collaboration research. We have carefully considered the potential societal impacts of our method and believe it does not pose any significant ethical or fairness concerns, thereby ensuring its responsible application.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims the abstract and introduction accurately reflect the scope and contributions of the paper, they directly supported by the detailed theoretical analysis and empirical results in Sec. 3 and Sec. The paper also clearly differentiates between its current contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of out proposed MoE Jetpack in Sec. 6. We discuss the computational efficiency Appendix B.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all necessary information to reproduce the main results in Sec. 4.1. For more detailed experiment configurations, please refer to Appendix A and B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: Since the code has not been fully organized yet, we will make the code and model weights available as open source once the organization is complete.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. cccpublic//eeeeeceeeeceeeeeeeeee wwticc//g///////eeee (CodeSubmissionPe)
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the paper provides all necessary details for understanding the training and test processes. We conducted training and testing using publicly available datasets (like ImageNet and CIFAR), the specifics of which are listed in Sec. 4.1. For more detailed experimental configurations, such as hyperparameters, type of optimizer used, etc., please refer to Appendix A and B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not provide enror bars because our experiments used a fixed training and testing dataset and constant random seed (3407), which resulted in minimal variability (below 0. %) in network training outcomes across runs. Given this stability, we concluded that error bars were unnecessary.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed specifications of the computing resources used for experiments are listed in Sec. 4.1. For comprehensive information regarding the compute consumption and execution time for each experimental run, please refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp//p///////.../eee//// cc htte//ubll//tiiieeeeeeeeeeeeeeee csGuidel ines?
Answer: [Yes]
Justification: The research presented in this paper adheres all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification:
We discuss the Broader Impacts in Appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used in our paper, including code, datasets, and models, are properly credited to their original creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The new assets introduced in the paper, including models and code, have not yet been released. Documentation for these assets is currently preparation and will be provided upon their official release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
24

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "zK4sHjol0L", "reviewer": "Reviewer_7BZ9", "summary": "The paper proposes several ideas to convert dense checkpoints into MoEs for vision tasks. In particular, four alternatives are presented to reuse checkpoints of dense models to be use as an initial checkpoint for an MoE model for a later fine-tuning phase. Of the four alternatives presented, Importance-Based Weight Sampling is the one that works the best. Most importantly, all of them are reported to work better than Sparse Upcycling, at least when evaluated on ImageNet (see Table 3).\nThe second novelty that the paper presents is a variant of the SoftMoE router: the SpheroMoE router. The SpheroMoE has an additional projection on the inputs, and places the L2 normalization in different places, compared to SoftMoE. It also has a temperature parameter that is decayed over time. Similar to the SoftMoE router, the SpheroMoE computes slots as a weighted average of the input tokens (similar to attention).\nLast but not least, the paper introduces the Adaptive Dual-Path structure in the MoE blocks. The experts are divided in two sets: a set of light experts that process lots of slots, and a set of heavy experts that process fewer slots. \nThe paper presents results on variety of common vision tasks, such as ImageNet-1k (a.k.a. ILSVRC2012), CIFAR-10/100,  Food-101, etc. Sometimes the models are trained from scratch on these datasets, and others an initial (dense) checkpoint pre-trained on ImageNet-21k is used.", "questions": "- In Algorithm 1, the meaning of `K_project` and `inherit_layer_norm` are not clearly explained. Please, do so. In particular, what's the output size of the `K_project`? Is it the same as the embedding size of X?\n- In that regard, assuming that  `inherit_layer_norm` is doing some sort of LayerNorm with shared parameters, LayerNorm and L2-normalization are very similar, except for a $\\sqrt{n}$ factor and the additional $\\alpha$ or scale parameters of LayerNorm (and potentially the biases too). How was the use and positioning of the additional L2-norm decided?", "limitations": "No negative societal impact is particular to this work, in my opinion.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The different ideas presented to initialize a MoE from a dense checkpoint are new and are shown to work better as a previously proposed approach: Sparse Upcycling. The explanation of the methods is well-detailed in the paper.\n- A quite detailed implementation of the SpheroMoE router is given in the appendix, which will certainly widen the reproducibility and potential adoption of the paper.\n- Most of the presented ideas have been ablated on different experiments. In particular, the choice of Importance-Based Weight Sampling is justified in Table 3, experimenting on ImageNet, which is a quite rich dataset. The use of the SpheroMoE router is well justified in Table 2, and is showed to be superior to the SoftMoE router on 3 datasets (ImageNet, CIFAR-100, Flowers). \n- Overall, it's shown that using MoE Jetpack not only significantly accelerates the training on ImageNet-1k and CIFAR-100, compared to training from scratch, but also the models achieve a higher accuracy which would probably not be achieved from scratch, or at least not in a reasonable amount of time, in the case of CIFAR-100.", "weaknesses": "- All experiments are done with quite modest backbones: Tiny (T) and Small (S) vision tranformer backbones. The Sparse Upcycling paper, that the authors use a baseline, observes that the benefits of initializing from a dense checkpoint are much smaller when using larger backbones (especially when evaluating transfer to small downstream tasks), see Figure 3 and 4 from https://arxiv.org/pdf/2212.05055. So,  it's not clear at all wheather the proposed improvements will make a difference at much bigger scales.\n- SpheroMoE has far more experts that SoftMoE. It's not clear wheather the benefits come from the improvements or by using finer-grained & more experts, which has been proposed before (for instance, in https://arxiv.org/abs/2401.06066 and https://arxiv.org/abs/2402.07871).\n- The results for SoftMoE in Figure 1b and Table 1 are for models trained from scratch. Since a distinction is made for ViT on whether it's trained from scratch or pre-trained on ImageNet-21k, perhaps it would be fairer to present this distinction for SoftMoE as well in these main figure and table. This ablation is presented in Table 2, but only for 3 datasets.\n- Figure 2b and Equation 10 (and algorithm 1 in the appendix), do not quite match in one aspect: the L2-normalized X is also used to compute the content of the input slots of each expert, and not the original X as Figure 2b seems to depict.\n- SpheroMoE has a few differences with SoftMoE. It would be much better if the individual differences were ablated. For instance, what's the effect of the additional (linear?) projection on X used to compute the combine and dispatch weights? What's the difference between having a decayed temperature vs. having a learned temperature (or scale) as SoftMoE does?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our paper and for highlighting the novelty and effectiveness of our method. We will address each of the concerns you raised in detail.\n\n> **W1:** All experiments are done with quite modest backbones ... \n\nWe acknowledge the use of modest backbones due to limited computational resources, as our experiments were conducted on 8 NVIDIA RTX 4090 GPUs. Training a vanilla ViT-S model on ImageNet (1.28M images) took over two days, limiting our ability to explore larger models.\n\nThe Sparse Upcycling paper suggests that the benefits of dense checkpoint initialization diminish with larger models, likely due to diminishing marginal returns on performance for smaller tasks. However, our MoE Jetpack improves upon Sparse Upcycling by not only using dense checkpoints but also modifying the MoE architecture, which we expect to yield better results.\n\nDespite the constraints on large-scale experiments, our proposed checkpoint recycling and SpheroMoE layer present promising avenues. We conducted experiments within our means using modest backbones, which can still provide valuable insights and demonstrate the potential of our approach.\n\n> **W2:** SpheroMoE has far more experts...\n\nIn our ablation study, we ensured a fair comparison between SoftMoE and SpheroMoE by increasing the number of experts in SoftMoE to 6*197, rather than using the original count from the SoftMoE paper. However, we did not compare the models with an equal number of experts. The corresponding results are provided in the table below and will be included in the revised manuscript. When the number of universal experts (experts with 1/4 parameters) in the dual-branch structure is set to zero, SpheroMoE and SoftMoE have the same number of experts.\n\n|Model| Number of Experts | ImageNet 1k | FLOPs(G) |\n|-|-|-|-|\n|SoftMoE|197|78.4|1.2|\n|SpheroMoE|core: 197, univ: 0 |79.6|1.2|\n|SpheroMoE|core: 98, univ: 196|79.9|1.1|\n\n> **W3:** SoftMoE are trained from scratch in the main result...\n\nWe designed the main table to highlight the overall performance improvements from our two approaches: checkpoint recycling and SpheroMoE. Using SoftMoE trained from scratch as the baseline effectively demonstrates the combined effectiveness of checkpoint recycling and SpheroMoE. Therefore, placing the ablation study of SoftMoE and SpheroMoE, both initialized with checkpoint recycling, in Table 2 is more appropriate to illustrate the component effectiveness.  We conducted this ablation on three representative datasets, including ImageNet, and will aim to conduct more extensive experiments on additional datasets.\n\n> **W4:** Figure 2b does not quite match.\n\nSome details were missing in Figure 2. We will incorporate additional details in the subsequent versions.\n\n> **W5 & Q1:** SpheroMoE has a few differences with SoftMoE.  1) The effect of `key_proj`. 2) decayed temperature.\n\n(1) **Additional Projection:**\nIn cross-attention, queries (Q), keys (K), and values (V) are generated by applying linear projections. In SpheroMoE, projections are adjusted to align the activation distributions between the SpheroMoE layer and the pre-trained model.\n\n- **V Projection:** \nTo maintain alignment with pre-trained dense checkpoints, we use layer normalization from the pre-trained network to process input $X$ to produce $V$. The `inherit_layer_norm` in the pseudocode corresponds to `self.norm2` below.\n```\n    # vit block forward\n    x = x + self.mlp(self.norm2(x))\n```\n- **Q and K Projections:** For K, we used a scale-and-shift as `k_proj`: $K = \\alpha_k \\cdot X + \\beta_k$, while Q underwent layernorm, containing scale and shift adjustments. On ImageNet, these projections didn't help. However, on smaller datasets, these projections both accelerated convergence and enhanced final performance.\n\n(2)  **Decayed Temperature:** \nWe apologize for the limited explanation provided in the manuscript.   **SoftMoE** uses a single trainable scalar for scaling Q: $\\text{scale} \\cdot Q$. In **SpheroMoE**, the Decayed Temperature $T$ is represented as a list with two learnable parameters, which adjust the dispatch and combine logits, respectively:\n\n$Dispatch = softmax(logits/T[0] + noise, dim=1)$, \n\n$Combine = softmax(logits/T[1] + noise, dim=[-1,-2])$. \n\n$T$ is initialized to large values (1.5) to ensure a uniform distribution of attention among experts. As training progresses, these coefficients are optimized by backpropagation and gradually decrease. The ablation is in the table below.\n\n|SoftMoE|ImageNet 1k|\n|-|-|\n|Scale|78.4|\n|$T$|78.7|\n\n> **Q2:**  LayerNorm and L2-normalization are very similar. How ... decided?\n\nWhile LayerNorm and L2-normalization share similarities, they serve distinct purposes. LayerNorm adjusts the mean and variance, stabilizing activations, while L2-normalization scales input to unit norm. \n\nWe use layer_norm on the query (Q) to maintain a consistent distribution with the keys (K). \n\nL2-norm is applied for the following reasons:\n1. Preventing Overfitting: By projecting Q onto a hyperspherical space, L2-norm constrains the representation space.\n2. Stabilizing Softmax Outputs: L2-normalized Q prevents numerical instability, which is crucial in functions involving exponentiation, such as softmax. Unlike layer-norm, L2-norm directly controls vector magnitudes. Without L2-norm, **large logits can cause sharp distributions and large gradients**, destabilizing training. L2-norm addresses these issues, a capability that layer-norm lacks.\n3. Enhancing Training Stability: Our experiments show that applying L2-norm results in more stable gradients during training. Specifically, **the grad norm remains below 2 with L2-norm, while it can reach around 70 without it, leading to training instability.**\n\nOur experiments indicated that omitting L2-norm on Q significantly degrades performance while applying it to K has negligible effects.\n\n|l2norm position|cifar100|ImageNet 1k|\n|-|-|-|\n|Only Q|88.4|79.9|\n|Only K|87.5 |-|\n|Q & K|88.3|-|\n|w/o|87.5|79.2|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOFXwiBN6r", "reviewer": "Reviewer_qYPk", "summary": "This paper introduces MoE Jetpack, a novel method for fine-tuning pre-trained dense model checkpoints into mixture of experts (MoE) models. It leverages checkpoint recycling to accelerate convergence and enhance accuracy, and incorporates a hyperspherical adaptive MoE (SpheroMoE) layer for optimized integration and performance. Extensive evaluations show that MoE Jetpack not only speeds up the fine-tuning process but also achieves higher accuracy across various datasets while maintaining computational efficiency.", "questions": "Please refer to the concerns raised in the weaknesses section. It is essential to provide more experiments to demonstrate that MoE models using pre-trained weights deliver significantly better performance, and to verify that the proposed method has sufficient scalability.", "limitations": "The paper has discussed the limitations in the conclusion, specifically noting that the method is only applicable to visual tasks. There is no need to consider potential negative societal impacts for this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Tables and figures are informative, and the paper is generally well written.\n2. The technical contributions of the paper are substantial.\n3. The experiments comprehensively validate the effectiveness of the proposed methods.", "weaknesses": "1. The work consumes extensive computational resources to validate a complex algorithm on small-scale datasets. Specifically, all experiments presented required 3300 GPU hours for training and a total of 8000 GPU hours for exploratory purposes. However, the datasets primarily showcased, aside from ImageNet, are of small scale and outdated. Given the current emphasis on large models, these small datasets are not suitable for substantiating the claims of this paper. It is recommended to conduct more extensive experiments on ImageNet or similar large-scale datasets to validate scalability and justify the extensive use of pre-trained checkpoints.\n\n2. From an application perspective, the effectiveness of utilizing checkpoints in practical scenarios is questionable. For instance, the core result of the ViT-based models, where V-JetMoE-T with 264M parameters only achieves 79.9% accuracy, suggests that using pre-trained weights for MoE may not be as parameter-efficient as expected. Instead, using comparable computation models, like those on the ImageNet performance leaderboard, might be more advantageous. This critique underscores the need to demonstrate that utilizing pre-trained weights for MoE can surpass the performance of similarly scaled models trained conventionally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive comments on our paper. We appreciate the opportunity to address your concerns and clarify our work.\n\n> **W1:** The work consumes extensive computational resources on small-scale datasets.  It is recommended to conduct more extensive experiments on ImageNet or similar large-scale datasets...\n\nWe appreciate the reviewer's suggestion to conduct more comprehensive experiments on large-scale datasets. While we agree that such experiments would further substantiate the efficacy of our approach, our current hardware setup, consisting of 8 RTX 4090 GPUs, imposed significant limitations. The (3300 + 8000) GPU hours expended represent the maximum computational budget we could allocate, **with 70% of this time dedicated to ImageNet-1k.** The primary experiments and most of the ablation studies conducted on ImageNet-1k effectively demonstrate the robustness of our method. **ImageNet-1k has been widely recognized and utilized in recent literature as a standard benchmark for validating new methods [1,2,3,4].** Should more computational resources become available in the future, we plan to further validate our approach on additional and larger datasets.\n\nAdditionally, **the experiments conducted on smaller datasets, although not consuming a significant portion of computational resources, play a crucial role in illustrating the adaptability and generalization capabilities of our method.** In practical scenarios, many downstream vision tasks may not have access to extensive datasets like ImageNet-1k. Therefore, evaluating our approach across datasets of varying scales offers valuable insights into its performance under conditions of limited data availability. This not only highlights the versatility of our method but also its potential applicability in diverse real-world scenarios.\n\n[1] Xu, Zhiqiu, et al. \"Initializing models with larger ones.\" ICLR 2024. \n\n[2] Chen, Mengzhao, et al. \"Diffrate: Differentiable compression rate for efficient vision transformers.\" CVPR, 2023.\n\n[3] Pan, Zizheng, Jianfei Cai, and Bohan Zhuang. \"Stitchable neural networks.\" CVPR, 2023.\n\n[4] Trockman, Asher, and J. Zico Kolter. \"Mimetic initialization of self-attention layers.\" ICML, 2023.\n\n> **W2:** From an application perspective, the effectiveness of utilizing checkpoints in practical scenarios is questionable. ... V-JetMoE-T with 264M parameters only achieves 79.9% accuracy...\n\nThank you for your thoughtful comments on the parameter efficiency of MoE architectures compared to dense models, particularly those on the ImageNet performance leaderboard.\n\n**It is important to note that the primary consideration here is not merely the checkpoint recycling but rather the fundamental differences between MoE and dense architectures.** Moreover, the effectiveness of utilizing checkpoints in practical scenarios has already been demonstrated in the Sparse Upcycling paper [1], with specific results detailed in Appendix C. The scalability of MoE models has been evidenced in numerous prior works [1,2,3]. For instance, the Soft MoE L/16 128E model, with 13.1B parameters, achieved an accuracy of 89.2% on ImageNet. However, conducting experiments of this scale is far beyond our available resources.\n\nMoE models function under a sparse activation paradigm, which is quite distinct from dense models. Specifically, MoE models are designed to activate only a subset of parameters, thus optimizing computational resources and enabling scalability without a linear increase in computational costs. For instance, **the V-JetMoE-T model (265M parameters, 79.9% accuracy, 1.1G FLOPs) shares the same FLOPs as the ViT-T model (6M parameters, 75.6% accuracy, 1.1G FLOPs), their training and inference time are very similar.** This is a key advantage of MoE models: they can improve performance while maintaining comparable FLOPs, making them well-suited for scenarios that require efficient training and inference.\n\nFurthermore, when **given the same training budget, MoE Jetpack can often achieve superior performance compared with vanilla ViT.** For example, V-JetMoE-T reaches 79.9% accuracy with almost the same computational overhead as ViT-T, which only achieves 75.6% accuracy (considering the fine-tuning process). **A dense model with a similar parameter of V-JetMoE-T (256M) would require approximately 54G FLOPs, resulting in a significantly higher computational cost**—around 50 times that of V-JetMoE-T. Therefore, direct comparisons between the parameter counts of dense and MoE models may not be entirely fair.\n\nWe hope this explanation clarifies the distinctions and advantages of MoE models. Thank you again for your valuable feedback.\n\n[1] Komatsuzaki, Aran, et al. \"Sparse upcycling: Training mixture-of-experts from dense checkpoints.\"  ICLR 2023.\n\n[2] Puigcerver, Joan, et al. \"From sparse to soft mixtures of experts.\" ICLR 2024.\n\n[3] Riquelme, Carlos, et al. \"Scaling vision with sparse mixture of experts.\" NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KLPxbNrjZY", "reviewer": "Reviewer_ysa3", "summary": "This paper introduces MoE Jetpack, a novel method to efficiently fine-tune dense model checkpoints into sparsely activated mixture of experts (MoE) models, reducing the need for extensive data and computational resources. Key techniques include checkpoint recycling and the hyperspherical adaptive MoE (SpheroMoE) layer, which effectively leverage the pre-training costs of existing dense models and optimize the MoE architecture. Experimental results demonstrate significant performance gains and improved convergence speed across various datasets and network architectures.", "questions": "1.\tI am curious about the differences that might arise from initializing expert weights from models of different sizes. Is a larger model always better?", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper provides a thorough explanation of the methods, including various strategies for checkpoint recycling and detailed architecture of the SpheroMoE layer, offering a clear and replicable approach. The pseudo-code in the appendix is exceptionally clear and well-structured.\n2.\tThe idea of effectively decomposing pre-trained dense models into experts of varying sizes is very interesting. It significantly enhances the flexibility in utilizing the sunk pre-training costs of existing dense models.\n3.\tThe Adaptive Dual-path MoE employs experts of varying sizes within a single MoE model, effectively balancing speed and accuracy. This approach significantly enhances the flexibility and performance of MoE models.\n4.\tThe experiments cover a wide range of datasets and model architectures, ensuring that the results are robust and generalizable.", "weaknesses": "1.\tThe authors need to carefully check the formatting details of the paper:\ni) The order of Tables 2 and 3 is incorrect in lines 227 and 262.\nii) There is a missing space before the parentheses in Table 1.\niii) The numerical values within the parentheses in Table 2 need verification for correctness.\n2.\tThe paper misses some related work [2][3][4] and should discuss the proposed method in comparison with these approaches, such as the differences between the proposed Expert Regularization method and the load balancing loss [1] and router Z loss [2]. Are the methods proposed in this paper orthogonal to these existing methods?\n3.\tThe additional computational overhead introduced by checkpoint recycling should be detailed and quantified by the authors.\n\n[1] Fedus W, Zoph B, Shazeer N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[J]. Journal of Machine Learning Research, 2022, 23(120): 1-39.\n[2] Zoph B, Bello I, Kumar S, et al. St-moe: Designing stable and transferable sparse expert models[J]. arXiv preprint arXiv:2202.08906, 2022.\n[3] Xue F, Zheng Z, Fu Y, et al. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models[C]//Forty-first International Conference on Machine Learning.\n[4] Zhou Y, Du N, Huang Y, et al. Brainformers: Trading simplicity for efficiency[C]//International Conference on Machine Learning. PMLR, 2023: 42531-42542.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewers for their valuable comments and suggestions. We have carefully considered each point and provided our responses below.\n\n> **W1:** The authors need to carefully check the formatting details of the paper...\n\nWe acknowledge the formatting issues identified. We will correct the order of Tables 2 and 3, ensure the numerical values within the parentheses in Table 2 are verified, and address the missing space before the parentheses in Table 1. These corrections will be implemented in the revised manuscript.\n\n> **W2:** The paper misses some related work and should discuss the proposed method in comparison with these approaches...\n\nWe agree that a more thorough comparison with related work is necessary. We will expand the related work section to include these papers. In ST-MoE, the load balancing loss and router-Z loss are employed. The load balancing loss, however, is not applicable to SpheroMoE due to its use of a cross-attention-like mechanism that dynamically assembles the input for each expert. This is an advantage of the SoftMoE design, which inherently balances the load without needing additional loss functions.\n\nThe router-Z loss, on the other hand, operates on the logits of the gating mechanism. It aims to regulate the scale of the logits, encouraging them to remain as small as possible in absolute value, as larger logits can lead to larger gradients within the softmax activation function, potentially causing instability during training. This loss can indeed be applied to SpheroMoE to constrain the logits and enhance numerical stability.\n\nWe will conduct further experiments to investigate the impact of incorporating router-Z loss in SpheroMoE and include the findings in the revised manuscript. We appreciate the reviewer's valuable suggestion and will ensure that these aspects are thoroughly explored.\n\n> **W3:** The additional computational overhead introduced by checkpoint recycling should be detailed and quantified by the authors.\n\nWe understand the concern regarding the additional computational overhead introduced by checkpoint recycling. Checkpoint recycling involves four methods for weight selection: Random Sampling, Uniform Selection, Graph Partitioning, and Importance-based Sampling. The first two methods, Random Sampling, and Uniform Selection, **do not entail any additional computational processes**, as they directly select experts from the MLP of a larger network without further calculations.\n\nIn contrast, Graph Partitioning and Importance-based Sampling require a preliminary step. We first randomly select a small subset of images (30,000) from ImageNet 1k and perform inference using the dense checkpoint on these images. This step is necessary to obtain the intermediate activation values required for determining the graph partitioning and assessing the importance of individual neurons. For instance, when selecting experts for the V-JetMoE-T model, we use ViT-S to infer on the selected images. As this process does not involve optimizing the network, the computational overhead is minimal. Specifically, on an RTX 4090, **this inference process takes less than 5 minutes.**\n\nWe will include these details and a quantitative analysis of the overhead in the revised manuscript to provide a clearer understanding of the computational impact of checkpoint recycling. Thank you for pointing this out.\n\n> **Q1:** I am curious about the differences that might arise from initializing expert weights from models of different sizes. Is a larger model always better? \n\nThis is a highly insightful question. **While larger models generally contain more extensive pre-trained knowledge, initializing experts from them does not always yield better performance** in a fixed-size MoE network (e.g., V-JetMoE-T). In practice, experts selected from models of similar sizes (e.g., ViT-S) often perform better. This observation aligns with previous findings in weight selection [1, 2] and knowledge distillation [3, 4].\n\nLarger models, though more expressive, may possess a high degree of redundancy in their learned representations. When transferring weights to a smaller model, **the size disparity can lead to a loss of critical structural information,** thereby diminishing the benefits of the pre-trained knowledge. Moreover, **smaller models closer in size to the target MoE network often have more compatible feature representations,** which are easier to adapt and fine-tune within the MoE framework. This compatibility can lead to more efficient utilization of the model's capacity, resulting in better generalization and performance. Therefore, selecting experts from models of similar size helps maintain a balance between the richness of learned features and the adaptability of the network's capacity.\n\n[1] Xu, Zhiqiu, et al. \"Initializing models with larger ones.\" ICLR 2024. \n\n[2] Pan, Zizheng, Jianfei Cai, and Bohan Zhuang. \"Stitchable neural networks.\" CVPR 2023.\n\n[3] Hu, Chengming, et al. \"Teacher-student architecture for knowledge distillation: A survey.\" arXiv preprint (2023).\n\n[4] Pham, Cuong, Tuan Hoang, and Thanh-Toan Do. \"Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks.\" WACV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wbz7ghBDR8", "reviewer": "Reviewer_hQZu", "summary": "The authors propose MoE Jetpack, a method to transform dense checkpoints into a SoftMoE-like network. They introduce checkpoint recycling to utilize dense checkpoints as initial weights for MoE models. Additionally, the authors enhance the SoftMoE with the proposed Expert Regularization and adaptive Dual-path mechanisms. Extensive experiments and ablation studies are conducted to demonstrate the effectiveness of these proposed mechanisms.", "questions": "1.\tCould you compare MoE Jetpack with a Soft MoE pre-trained on ImageNet21k and fine-tuned on the evaluation dataset?\n2.\tCould you compare MoE Jetpack with a model pre-trained on ImageNet21k, upcycling to an MoE, and fine-tuned (only router and experts' MLP layer) on the evaluation dataset?", "limitations": "Limitation addressed in the paper", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "How to transform a dense checkpoint to an MoE is surely an important problem. The proposed checkpoint recycling method seems promising. Extensive experiments and ablations are given to prove the effectiveness of the proposed method.", "weaknesses": "The comparison in the main results appears to be unfair. From my understanding, MoE Jetpack is initialized with a pre-trained model and then fine-tuned on the evaluation dataset. However, the baseline (SoftMoE) is trained from scratch solely on the evaluation dataset. For instance, in the ViT part (Table 1a), MoE Jetpack is initialized with a ViT pre-trained on ImageNet-21k and then fine-tuned on ImageNet-1k, whereas SoftMoE is trained from scratch on ImageNet-1k. This unfairness is even more pronounced on smaller datasets like STL-10 or Pets, as MoE Jetpack already benefits from prior knowledge gained from ImageNet-21k.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1-3:** The comparison in the main results appears to be unfair...  MoE Jetpack is initialized with a pre-trained model and then fine-tuned on the evaluation dataset. However, the baseline (SoftMoE) is trained from scratch solely...  you should use a SoftMoE pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k as the baseline. The novelty of the proposed method is limited.\n\nThe reviewer's comments seem to reflect a misunderstanding of our methodology and experimental design. We would like to clarify our approach, the **checkpoint recycling and SpheroMoE layer**, and explain the rationale behind our experimental setup.\n\nThe background for our method lies in the lack of pre-trained weights for existing MoE models, where pre-training on large datasets like ImageNet-21k requires substantial computational resources. We proposed **checkpoint recycling, a method that leverages pre-trained weights from densely activated models to initialize MoE models, thereby avoiding the significant cost of pre-training.** Inspired by sparse upcycling [1], checkpoint recycling offers greater flexibility and efficiency, allowing for the creation of experts of arbitrary size and number. It is important to note that checkpoint recycling involves selecting and reconfiguring dense checkpoints into experts **without any training or optimization,** resulting in minimal computational overhead. Since checkpoint recycling is a cross-network knowledge transfer method, and specifically aims to circumvent the high computational cost of MoE pre-training, we did not pre-train MoE models on ImageNet-21k. \n\nTo demonstrate the effectiveness of checkpoint recycling with SpheroMoE, we used randomly initialized SoftMoE as the baseline in the main results. **To ensure a fair comparison, the influence of initialization was excluded in the ablation study presented in Table 2 of the paper, where the SoftMoE model also utilized the checkpoint recycling technique.** The comparison between sparse upcycling and checkpoint recycling is presented in Table 3 of the paper.\n\nSoftMoE was not designed to utilize dense checkpoints for initialization, and its architecture is **not optimal for the task of checkpoint recycling followed by fine-tuning. Thus, we proposed the SpheroMoE layer, optimized for this task.** SpheroMoE and SoftMoE differ in several significant ways:\n\n1.  Adaptive Dual-path: Given that checkpoint recycling can produce experts of any size and number, we can control efficiency and performance by using experts of varying sizes. Adaptive Dual-path is introduced to reduce expert parameters while increasing the number of experts, allowing SpheroMoE to achieve better performance with lower FLOPs and fewer parameters.\n2.  Expert Regularization: This includes a learnable softmax temperature T, expert noise, and stochastic expert dropout, which help mitigate overfitting in MoE models.\n3.  Extra Norm: To align the distribution between MoE layers and pre-trained models, we use the layer norm from the pre-trained models to constrain the QKV of SpheroMoE. This alignment is crucial for ensuring smooth convergence during fine-tuning, which would not be possible with the original SoftMoE structure. Additionally, to address the impact of the magnitude of randomly initialized queries on similarity calculations and softmax computation, and to introduce regularization to prevent overfitting, we project the queries onto a hyperspherical space using L2 normalization.\n\nThe corresponding ablation experiments are presented in the table below. We will revise the manuscript to include this table.\n\n|SoftMoE|Extra Norm|Expert Regularization|Adaptive Dual-path|ImageNet-1k|Cifar100|\n|-|-|-|-|-|-|\n|√| | | |78.4|84.7|\n|√|√| | |79.2|86.9|\n|√|√|√| |79.6|88.2|\n|√|√|√|√|79.9|88.4|\n\nIn summary, checkpoint recycling and SpheroMoE work synergistically, leveraging the pre-trained knowledge from dense checkpoints to efficiently fine-tune on downstream tasks without additional pre-training costs. This forms the core motivation behind our method. We hope this explanation provides a clearer understanding of our approach, and we look forward to further discussions.\n\n> **Q1:** Could you compare MoE Jetpack with a Soft MoE pre-trained on ImageNet21k.\n\n**The primary motivation behind our method is to avoid the substantial computational cost of pre-training MoE models on large datasets like ImageNet21k.** Our approach, checkpoint recycling, leverages existing dense checkpoints to initialize MoE models without training, thus bypassing the need for extensive pre-training.\n\nOur work aims to demonstrate that MoE models can achieve competitive performance on downstream tasks without the overhead of pre-training from scratch, which is particularly relevant when computational resources are limited. While comparing MoE Jetpack with a Soft MoE model pre-trained on ImageNet21k would indeed provide additional insights, it falls outside the core objective of illustrating the efficacy of checkpoint recycling.\n\nNevertheless, we are open to including it in future work, should computational resources permit. This would allow us to explore the full potential of MoE models.\n\n> **Q2:** Could you compare MoE Jetpack with a model pre-trained on ImageNet21k, upcycling to an MoE, and fine-tuned (only router and experts' MLP layer) on the evaluation dataset?\n\nWe appreciate the suggestion to explore this fine-tuning mode, which has also been mentioned in ST-MoE [2]. The results of our experiments are as follows:\n\n|MoE Jetpack|ImageNet-1K| \n|-|-|\n Full fine-tuing|79.9|\n|Only router and experts|78.1|\n\n[1] Komatsuzaki, Aran, et al. \"Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.\" ICLR 2023.\n\n[2] Zoph, Barret, et al. \"St-moe: Designing stable and transferable sparse expert models.\" arXiv:2202.08906.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to express our gratitude for your insightful feedback and suggestions, which have been instrumental in updating and enhancing our submission. We are particularly grateful for your recognition of the novelty of our approach (R 7BZ9, ysa3), the thoroughness of our experiments (R 7BZ9, qYPk, ysa3), the substantial technical contributions (R qYPk), and the clarity and quality of our writing (R qYPk, ysa3). Your positive comments on these aspects are greatly appreciated.

We kindly invite you to review our author rebuttal. For each point raised by the reviewers, we have provided detailed responses. We hope our responses adequately address your concerns. If you have any further questions, please do not hesitate to bring them to our attention. We sincerely appreciate your valuable feedback.

Best regards,

Authors

Meta-review: In the initial reviews, all reviewers acknowledged the novelty of this paper and the thoroughness of the experiments. However, one reviewer pointed out that essential experiments were missing, such as a comparison to a Soft MoE pre-trained on ImageNet21k and fine-tuned on the evaluation dataset, or a model pre-trained on ImageNet21k, upcycled to an MoE, and fine-tuned (with only the router and experts' MLP layers) on the evaluation dataset. During the rebuttal phase, the authors addressed these concerns with additional experiments, leading to positive feedback from all reviewers. AC has also carefully reviewed the paper, the reviews, and the rebuttal, and concurs with the reviewers' opinions. Therefore, AC recommends acceptance. It would be highly beneficial to include the additional results from the rebuttal phase in the final version.
Decision: Accept (poster)