Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts
Authors: Weilin Cai
Juyong Jiang
Le Qin
junweicui
Sunghun Kim
Jiayi Huang
Venue: ICML 2025 poster (2025)
Keywords: Mixture of Experts
Expert Parallelism
Shortcut Connection
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Expert parallelism has emerged as a key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the processing of increasingly large-scale models. However, the All-to-All communication inherent to expert parallelism poses a significant bottleneck, limiting the efficiency of MoE models. Although existing optimization methods partially mitigate this issue, they remain constrained by the sequential dependency between communication and computation operations. 
To address this challenge, we propose ScMoE, a novel shortcut-connected MoE architecture integrated with an overlapping parallelization strategy. ScMoE decouples communication from its conventional sequential ordering, enabling up to 100\% overlap with computation.
Compared to the prevalent top-2 MoE baseline, ScMoE achieves speedups of $1.49\times$ in training and $1.82\times$ in inference.
Moreover, our experiments and analyses indicate that ScMoE not only achieves comparable but in some instances surpasses the model quality of existing approaches.

Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts
Weilin Cai 1
Juyong Jiang'
Le Qin
Junwei Cui
Sunghun Kim
Jiayi Huang'
Abstract
et al., 2021; Mustafa et al., 2022: Jiang et al., 2024). Expert parallelism (Lepikhin et al., 2021: Fedus et al., 2022) has emerged as viable strategy to efficiently distribute MoE computations over multiple devices, synergizing with con- ventional parallelism techniques (Hwang et al., 2023; Singh et al., 2023) such as data parallelism (Rajbhandari et al., 2020; 2021) and model parallelism (Narayanan et al., 2021; Smith al., 2022).
Expert parallelism has emerged as key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the processing of increasingly large-scale models. However, the All-to-All communication inherent to expert par- allelism poses significant bottleneck, limiting the efficiency of MoE models. Although exist- ing optimization methods partially mitigate this issue, they remain constrained by the sequential dependency between communication and compu- tation operations. To address this challenge, we propose ScMoE, novel hhorteut-connecter MoE architecture integrated with an overlapping paral- lelization strategy. ScMoE decouples communi- cation from its conventional sequential ordering, enabling up to 100% overlap with computation. Compared to the prevalent top-2 MoE baseline, ScMoE achieves speedups of 1.49x in training and 1.82x in inference. Moreover, our experi- ments and analyses indicate that ScMoE not only achieves comparable but in some instances sur- passes the model quality of existing approaches.
Nevertheless, expert parallelism incurs substantial All-to- All communication overhead (Lepikhin et al., 2021; Fedus et al., 2022), which can contribute to approximately 50% of the total time in intra-node multi-GPUs or multi-nodes distributed environments (see Figure 1), thus forming a crit- ical bottleneck in scaling MoE models (Nie et al., 2022; Hwang et al., 2023; Mayer & Jacobsen, 2020; Smith et al., 2022). Despite existing optimizations such as hierarchical All-to-All (He et al., 2022; Nie et al., 2022) and pipelin- ing (Hwang et al., 2023; Zhang et al., 2023) strategies that mitigate communication delays and partially overlap com- munication with computation, the communication challenge persists due to the inherent sequential dependencies between these operations (Wang et al., 2023). To address this con- straint, intuitive idea is to reconstruct the inputs of MoE layer by incorporating not only the current-layer but also the preceding layyer representations through shortcut con- nection, thereby refining the conmunicaiion-compuaatoo dependencies and expanding the potential for their overlap- ping optimization.
1. Introduction
In recent years, Transformer- baased large language models (LLMs) have significantly propelled the fields of Natural Language Processing (Vaswani et al., 2017; Brown et al., 2020; Wei et al.. 2022b; Ouyang et al., 2022; Wei et al., 2022a; Chowdhery et al., 2023; Achiam et al., 2023), Com- puter Vision (Dosovitskiy et al., 2021; Liu et al., 2021), and Multimodality (Lu et al., 2019; Zhou et al., 2022: Zhang et al., 2021: Zhu et al.. 2023). The sparsely-gated mixture- of-experts (MoE) approach has been integral in increas- ing parameter counts and enhancing model performance across various modalities (Shazeer et al., 2017; Riquelme
In this paper, we propose the shortcut-connected MoE (Sc- MoE) architecture, which completely decouples communi- cation processes from the sequence of conventional MoE models. ScMoE architecture is initially built on the standard top-2 MoE (see Figure 2 (a)), which typically substitutes the multi-layer perceptron (MLP) module with a top-2 gating MoE module in every second Transformer block (refer to the Transformer block with the MoE module as "current layer", and the preceding one without the MoE module as "preced- ing layer"). Diverging from the top-2 approach, ScMoE utilizes top-1 MoE module to process preceding- rep- resentations via a shortcut connection, while employing a shared expert (an MLP module) to process current-layer representations. These two processes are independently managed in parallel, with their results integrated into the final output of the current layer. Furthermore, the ScMoE
'The Hong Kong University of Science and Technology (Guangzhou). Correspondence to: Jiayi Huang <hjy @ @hkust gz.edu.cn>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Shortcut-comnected Expert Parallelism for Accelerating Mixture of Experts
MoE-Top2
20
in both vision and language downstream tasks. We also conduct in-depth analysis and discussion of the ScMoE architecture, investigating the proposed shortcut connection and exploring opportunities for further development.
MoE-Topl
10.9
1 node (8 A30-PCle)
MLP
1.0
FP16. 165 TELOPS, Bendwishhh 54 68/5
MoE-Top2
5.6
MoE-Top1
In summary, our contributions are as follows:
I node (8x A800-NVLnnk F216 312 B 400 586.
MLP-1 0.6
MoE-Top2
10.0 2 nodes (16x A800-NVLnn)
We propose the connuutte MoE (ScMoE) ar- chitecture that breaks the conventional dependency be- tween communication and computation in distributed MoE models, bypassing the restrictions imposed on current communication optimization techniques.
MoE-Top1
53
Computation Communication
MLP
5
10
15
20
Time (ms)
The overhead of MLP and top-2/top-1 MoF in trans- former block of Swin -2- (Hwang et al., 2023) model, allo- cating one expert per GPU with expert parallelism. The All-to-All communication takes up 60% of total time on single node with A30 GPUs, but drops to 15% on 8x A800 due the latter's 6x higher bandwidth provided by GPU-to-GPU NVLink (Foley Danskin, 2017). Despite benefiting from NVLink, communica- tion still approaches 50% due to the lower- banddihhh inter-node Ethernet (Li et al., 2020) when scaling across multiple nodes.
Figure 1.
We develop an adaptive overlapping strategy for ad- vancing expert parallelism with the shorteut-connectt MoE, which significantly improves the efficiency of MoE models and ensures broad compatibility.
We conduct empirical evaluation and theoretical anal- ysis on our methods, confirming that our methods ac- celerate MoF. models while achieving comparable or even better model quality compared to existing meth- ods, and offer in in-depth analysis and discussion on the effectiveness of the proposed shorteut connection.
architecture can be extended to accommodate MoE models that employ various MoE placement frequencies, such as integrating an MoE module into every Transformer block.
2. Background & Related Work
To efficiently overlap the decoupled communication and computation within the ScMoE architecture, we implement an adaptive overlapping parallel strategy that dynamically schedules operators based on actual performance metrics. Compared to existing optimization strategies such as pipelin- ing (Huang et al., 2019; Hwang et al., 2023), our approach not only doubles the overlap duration, but also realizes complete overlapping of communication in scenarios where communication time does not exceed the computation dura- tion. Furthermore, our method essentially advances the MoE architecture in algorithm aspect, which is device-agnostic to improve the efficiency of MoE model, thus ensuring broad applicability across various hardware configurations and maintaining compatibility with current optimizations.
2.1. Sparsely-Gated Mixture of Experts
The sparsely-gated mixture-of-experts (Shazeer et al., 2017) (MoE) layer is composed of multiple multi-layer perceptron (MLP) sub-networks, termed "experts," and employs a train- able gating network to selectively activate a subset of these experts during each iteration. Given N expert networks (E,),, gating network G and input representation æ, the output of MoE module can be written:
N
MoF(x) = Σ G(x).E,(t).
(1)
Following the prevailing approach in existing MoE research, we use the noisy top-k softmax gating network to select l: experts for the computation, formalized by
The experimental results reveal that, compared to the stan- dard top-2 MoE, our proposed ScMoE architecture opti- mally accelerates training by 1.49x and 1.14x in 8 A30 PCIe and 8 x A880-NVLink scenarios characterized by high and low communication overheads, respectively, and ac-
G(z) = (To
(2)
H(x); if H()), E (opK(H(m))
TopK (H (x), k);
(opK(H(C
celerates inference by 1.82x and 1.21x. Moreover, we perform experiments on different configurations of the Sc-
-o, otherwise.
(3)
MoE architecture, including chortcut-comnected position and coefficient gating network. Considering the optimal ac- curacy and the relatively longer overlap duration, we favor selecting the intermediate representations between the At- tention and MLP modules in the preceding layer as the input for the gate-routed experts. In addition, ScMoE has been demonstrated through experiments and thcoretical analysis to attain or exceed the model quality of existing methods
H(x); = (xr W Wgate); + 6, 4 -SStanndddNormalll Softplus(( Wnoise)
(4)
(5)
where is tunable Gaussian noise, W gate and W Wnoise denote two trainable weight matrices.
Leveraging sparse output of G(z), this approach signifi- cantly increases the number of model parameters without
2

---

Shortcut-conneeted Expert Parallelism for Accelerating Mixture of Experts
Ourputs
Outputs
Outputs
Outputs
Decode
Decode
E1
E2
E3
E4
SE
E1
E2
E3
All- Combine
MTh
E1
En
Gate
Gate
All- Dispatch
Block-MoE
Block-MoE
Encode
Encode
Self-Attention
Attentinnnn
Gate
Gate
Davice
Devicop
MLP
MLP
Self-Attention
Self-Attention
Block-MLP
Block-MLP
Inputs:
Inputs,
Self-Attemtion
Self-Atrention
Inputs
Inputs
Figure 3. Illustration of scaling MoE transformer layer across mul- tiple devices with expert parallelism.
(a) Standard MoE (Top-2)
[b) Shared-Expert Expert MoE
Figure 2. Illustrations of the standard top-2 MoE architecture (a) and the corresponding shared-expert MoE architecture (b). "SE" in (b) denotes the shared expert.
par with
superior to existing approaches, leading to its growing adoption (Xue et al., 2024; Wu et al., 2023; Chen et al., 2024; Gou et al., 2023; Gao et al., 2024).
causing a proportional increase in computational demand. The value of ls can be set to 2 or even higher values. Opting for a larger k moves the model closer to the dense architecture, which generally results in higher prediction accuracy (Riquelme et al., 2021), but also leads to greater computational overhead.
2.2. Expert Parallelism
To facilitate efficient distributed training and inference of MoE models, expert parallelism is proposed to allocate unique experts to each distributed computing device such as GPU and TPU, and map tokens to their corresponding experts through All-to-All communication across partici- pating devices (Lepikhin al., 2021; He al., 2021; Nie et al., 2022). As illustrated in Figure 3, the workflow of MoE employing expert parallelism is segmented into the following sequential operations: gate routing, input encode, All-to-All dispatch, expert computation, All-to-All com- bine, and output decode. To enhance the efficiency, input encode is employed to aggregate the token data layout to contiguous format before All-to-All dispatch, and output decode is the inverse process after All-to-All combine. Fur- thermore, the integration of expert parallelism with other parallelisms (Hwang et al., 2023; Singh et al., 2023; Fedus et al., 2022; Zheng et al., 2022) has been explored to support the scaling of larger MoE models on extensive distributed systems. However, the All-to-All communication used for token transfer has been a primary bottleneck limiting the efficiency of distributed MoE models, as shown in Figure 1.
Figure 2 (a) illustrates the prevailing top-2 MoE architecture. Each Transformer block with MoE module, denoted by replaces light blue block and referred to "Block-Mote the MLP with a set of experts ("E1, E2, E3, E4") and gating network ("Gate"). Following prior work (Lepikhin et al., 2021; Du et al., 2022; Shen et al., 2023; Hwang et al., 2023; Lieber et al., 2025), the "Block-MoE" is interspersed with the conventional Transformer block, depicted as a gray block and referred options exist for the frequency of MoE module placement, including placing MoE module into every block (Jiang et al., 2024; Dai et al., 2024: Qwen, 2024; Databricks, 2024) or every four blocks (Zoph et al., 2022; Xue et al., 2024).
"Block-MLP" Additionally, various
Shared Expert. In contrast to the standard top-2 MoE architecture, the shared-expert MoE incorporates fixed dense MLP module to process all input tokens, combining its output with the result from the top-1 gating expert for each token, as illustrated in Figure 2 (b). Given the shared expert SEE, the output of the MoE module is formulated as:
3. Shortcut-connected MoE Designs
In the prevailing Transformer-basee model, the MoE mod- ule substitutes MLP to sequentially manipulate intermediate representations (Lepikhin et al., 2021; Du al., 2022; Shen et al., 2023), impeding the efficacy of existing optimization strategies (He et al., 2022: Nie et al., 2022; Hwang et al., 2023; Zhang et al., 2023) due to the limited interaction within the MoE module. To address the aforementioned lim- MoE (ScMoE)
MoE(x) SE(x) + Σ (()).
(6)
This method, originally proposed by DeepSpeed-MoFE (Ra- jbhandari al., 2022), activates the same number of experts as the standard MoE for computation while reducing dy- namic expert selection and communication volume. Exten- sive empirical results from DeepSpeed-MMr and subsequent studies (Dai et al., 2024: Qwen, 2024) demonstrate that the shared expert architecture achieves model quality that is on
itations, we propose the architecture, which enables optimization opportunities for compuaation-oommnicial overlap for expert parallelism.
3

---

Shortcut-comncetel Expert Parallelism for Accelerating Mixture of Experts
Block-MoE
BLock-MLP
SE
E2
E3
SE
E1
E2
E3
SE
E1
E2
E3
Block-MoE
Block-MLP
Block-MoE
Self-Attention
Self-Attention
Self-Attention
[N/2K
Block-MLP
oouput
Block-MoE
MLP
Gate
MLP
Gate
MLP
Gate
Blo
intermediate
Block-MoE
Self-Artention
Self-Atrention
Self-Attention
BLP
Inpul
(c) ScMoE (Pos-1)
ScMoE (Pos-2)
ScMoE (Pos-3)
Figure Illustrations of various ScMoE architectures with shorteut connections different positions the preceding layer: (a) "Pos-I" output, (b) "Pos-2" intermediate, and (c) "Pos-3" input. The red line indicates the transmission of the preceding-layed representations to the MoE via a shorteut connection. Details regarding pre-layer normalization and dropout procedures have been excluded for simplicity.
3.1. Architectural Design
Block-MLP:
H.M.P. HH + MLP
In this section, we introduce the shortcut-connected MoE (ScMoF) architecture. Unlike the prevailing MoE archi- tectures, illustrated in Figure 2, which focus on processing intermediate representations within the current layer (the Transformer block containing the MoE), the ScMoE pro- cesses representations from both the current and preceding layers. Specifically, ScMoE employs top-1 MoE module to handle representations from the preceding layer via shortcut connection, while a shared expert processes the current-layer representations. These two operations are con- ducted independently, with their outcomes integrated into the final output of the current layer, facilitating communica- tion and computation overlap between these two processes.
HMH - MultiHead (1)
MLP (Hi (ML
(10)
where 7C refers the output from the MoE sub-layer, H signifies the output from the Multi-Head Attention (MultiHead) sub-layer MultiHead
MaE (-) in the (1 + 1)- th Transformer block ("Block-MoE"), (E(+))) denotes the shared expert while E,,.., EN represent the N gate- routed experts. The gating network G(.) is referred to as Equation 2. H.M.P and H are the outputs of the MLP sub-layer MLP(1 (-) and the MultiHead sub-layer MultiHead (.)()) respectively, in the l-th Transformer block ("Block-MLP"), Note that we omit the pre-layer nor- malization and dropout for simplicity.
While the shared expert processes the same intermediate representations in the current layer as the prevailing MoE ap- proaches, we explore three distinct preceding-layee represen- tations ScMoE's s top-1 MoE process, as illustrated Fig- ure The configurations "Pos-1" (a), "Pos-2" (b), and "Pos- 3" (c) represent shortcuts connecting different positions of the preceding layer: output, intermediate, and input, respec- tively. Given that TAtten TSE, and TM represent the du- rations of Attention, Shared Expert, and MLP, respectively, the corresponding overlap durations are (a) TAtten + TSE, (b) TAtten + TSE + TMLP, (c) 2T + TSE + TMLP.
In our experiments involving three tions, models configured with "Pos-2" achieve the highest accuracy in both vision and language cases, while also en- suring substantial overlap duration. As a result, we favor selecting "Pos-2" for practical development. Moreover ,the "Pos-2" configuration is used to elucidate the overlapping strategy in Section 3.2. The specifics of the other two con- figurations can be inferred by analogy.
posi-
Additionally, our proposed ScMoE architecture can be adapted to support MoE models with varying MoE place- ment frequencies. As illustrated in Figure 5, the ScMoE architecture can be integrated into MoE models that incor- porate an MoE module into every Transformer block. With more frequent MoE placement, the potential overlap dura- tion for each MoE module is minimized, and the "Pos- 1" configuration has already fully utilized the computation duration for the overlap. Conversely, less frequent MoE placement extends the potential overlap duration for each (8) MoB module, which may lead to increased acceleration.
Using the "Pos-2" configuration as an example, this ScMoE architecture can be formulated as follows:
Block-MoE:
ScMB
-HMH + SE0+)
MM = H,M ML MultiHead (++1)
(H.ML))
Mob

---

Shortcut-comneeted Expert Parallelism for Accelerating Mixture of Experts
Qulputts
Outputs,
Block-MoE
Device
All-to-All Combine
Devicen
Block-MoE
SE
E1
E2
E3
E1
En
Block-MoE
Self-Attention
Self-Attention
Block-MoE
Self-Attention
@
All-to-All
Block-MoE
Gate
MLP
Dispatch
MLP
[N/2]X
Block-MoE
Gote
Gate
Block-MoE
Self-Artention
Self-Attention
SE
E1
E2
E3
Inputs
Inputs,
Block-MoE
Figure 6. An overview of advanced expert parallelism using our proposed ScMoE architecture and overlapping strategy. The red line represents the decoupled MoE stream and the numbers ! through 4 denote the potential locations for expert computation.
Block-MoE
Self-Artention
Block-MoE
Gate
Figure 5. Illustration showcasing the application of the ScMoF (Pos-1) 1) architecture to the MoE model, wherein the MoE module integrated into each Transformer block.
Standard MoE. M.
Standard McE Pipeline
M
Shared-Expert MoE:
3.2. Overlapping Strategy for Expert Parallelism
M
A
Computation
ScMoE
Overlopping (Ours):
Attentionn
As mentioned in the previous section, the MoE operations in ScMoE architecture are completely decoupled from the backbone network, enabling parallel execution across two independent streams: one for the shared expert process and the other for the MoE process. To enhance efficiency, we im- plement asynchronous All-to-All communication operators to enable the overlapping of communication and compu- tation within these streams, while computation operators are unable to execute concurrently due to the constraints on computing resources.
A
M
A
MLP
Gate
ScMoE + Overlopping Pipeline (Ours).
A
M
Communication
Shared Expert
G
D
A2A Dispatch
Expert
Combinee
Encode/Decada
Figure 7. The timeline of different MoE architectures with corre- sponding parallel strategies, including pipeline and our proposed overlapping strategy. In each timeline, the length of each operator represents its time cost, while multiple rows indicate the utilization of parallel CUDA streams. The standard MoE utilizes top-2 gating, whereas the shared-expert MoE and ScMoE activate one shared expert alongside one gate-routed expert.
Adaptive Operators Scheduling. We observe that operator execution times are influenced by the specific model and hardware configuration, necessitating the implementation of adaptive scheduling for operators.
MLP and one Block-MoE is
Tblock = min( Toomp
-Taispp + Teomp Toombh)
Following the execution order in the MoE stream, we can directly schedule the gate routing and encode operators at the earliest viable position while deferring the decode oper- ator to the latest position, thereby maximizing the potential duration for overlapping. Then, this challenge is distilled into the selection of optimal position for expert compu- tation among four possible locations 123A within the shared expert stream, as depicted in Figure 6.
K-1
min(| Σ
COMP: -Taispl + Σ COMP: Teombl)
i=k+1
(11)
TMlck
2
|(Trre
Teoott
-
(Tesep + Teomb))l (12)
Tblock
(Temp
Teomp
+ (Tdisp Teomb)-
(13)
To demonstrate the efficiency, we have illustrated the op- erational timelines of various MoE architectures alongside their respective parallel strategies Figure 7, exemplified by the selection of location ( for expert computation. Each timeline's 's operator length corresponds to its execution time, and the presence of multiple rows signifies the utilization of parallel CUDA streams.
Formally, we define the communication costs associated with AAll-to-All Dispatch" and "All-to-All Combine" as Taisp and Teomb. respectively. The variable I. is des- ignated to represent the specific location where expert computation is applied. Prior to the expert computa- tion, the computational costs are denoted as Tyre (COMP),... P while the costs following the expert computation are represented as Toost == (COM C1MP.). Consequently, the minimal aggregate time cost for each pair consisting of one Block-
The widely-used pipeline parallel strategy equally segments input tokens into smaller fine-grained chunks, enabling con- current computation and communication dispatched on dis-
5

---

Shortcut-comneeted Expert Parallelism for Accelerating Mixture of Experts
tinct GPU streams (Hwang et al., 2023; Zhang et al., 2023). Contrary standard MoE with pipelining (2nd timeline), our proposed ScMoE with the overlapping strategy (4th timeline) significantly reduces the total time. This reduction is attributed to the decrease in absolute communication time, similar to that in the shared-expert MoE (3rd timeline), and the overlap of communication with the computation dura- tion (T Atten= + TSE + TMLP), which extends beyond the overlap duration achieved through pipelining.
Table I. Test top-1 accuracy and end-to end speedup of train and inference (one iteration) for SwinV2-MoE-S (Hwang et al., 2023) models with various architectures pre-trained on ImageNet- IK for 90 epochs in the 8 A30-PCIc scenario, using standard MoE with top-2 gating as the baseline.
Model
ImageNet-1K (Aco@) 11)
Train (Speedup1)
Inference (Speedupt)
Standard top-2 MoE
79.33%
1
Standard top-1 MoB
78.95%
1.27
1.39x
Shared-Expert MoE
79.53%
1.24x
1.35x
Our strategy possesses the capability to fully overlap com- munication if the communication can be accommodated within the overlapping window. This advantage is not shared by the pipeline strategy as it cannot overlap the initial and terminal data transmissions (Huang et al., 2019; Narayanan et al., 2019). In cases where communication durations ex- ceed the available overlap duration, our strategy be augmented with pipelining (5th timeline), thus utilizing the expert computation duration to further hide communication.
Our ScMoE (Pos- I) Our ScMoE (Pos-2)
79.14%
1.36x
1.54x
79.38
1.43x
1.66x
Our ScMoE (Pos-3)
79.20%
x.49x
x.82x
In scenarios characterized by a high communication-to- computation ratio, where communication hard to be completely overlapped by computation, ScMoE architec- tures with extended overlap durations can achieve supe- rior speedup. Specifically, ScMoE (Pos-3), which has the longest overlap duration of TAtten Tse + TMLP,, achieves the highest acceleration, with a 1.49x speedup in training and a 1.82x speedup in inference. Furthermore, the three different ScMoE architectures result in minimal variations in accuracy, ranging from 79.14% 79.38%. Additionally, these methods, which utilize two activated experts, consis- tently outperform the standard top-1 MoE in terms of model quality, as the top-1 approach activates fewer parameters.
4. Experiments
4.1. Experimental Setup
To assess the effectiveness of our proposed overlapping strategy for enhancing expert parallelism, we conduct ex- periments three hardware configurations: 8xA30-PCIe, 8x A800-NVLLink, and 16x A800-NVILink (across 2 nodes). These configurations cover scenarios with both high and ratios. Furthermore, we evaluated our methods using both vision models (2 MoE) (Hwang et al., 2023) and language models (GPT2- MoE, GPT3-MoE, LLaMA2-MoE) (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023). Additional details on the experimental setup are provided in Appendix A.8.
low patpmuunniatioon-o-coouualllll
4.2.2. LANGUAGE MODEL
To demonstrate the effectiveness of our proposed ScMoE architecture in models with two prevailing MoE placement frequency designs, we perform experiments using the GPT2- MoE--Medium and LLaMA2-MoE models, positioning the MoE module in every second Transformer block for GPT2- MoE and in every Transformer block for LLaMA2-MoE, Specifically, we utilize the ScMoE architecture with the configuration of "Pos-2" on the GPT2-MoR. model, since this setup yields the lowest final validation loss in our exper- iments across various shortcut-conmected positions, as dis- cussed in Section 5.2. Furthermore, the ScMoE in LLaMA2- MoE adopts the "Pos-1" configuration, as this setup has already maximized the potential overlap duration in the scenario of MoE placement in every block.
4.2. Analysis of Model Quality and Efficiency
In this section, we assess the quality of the models with our proposed ScMoE architecture. Furthermore, evaluate the efficiency of ScMoE models in distributed scenarios, which are accelerated through our proposed overlapping strategy for enhancing expert parallelism. To maintain the same computational volume as the standard top-2 MoE, both the experimental -hpred MoE and our ScMoE utilize one shared expert and one gate-routed expert.
As shown in Table 2, our ScMoE models achieve the high- est average scores, with 38.69 in GPT2-MoE and 38.96 in LLaMA2-MoE, Furthermore, when integrated with our ScMoE, GPT2-MoE experienced an 11% improvement in training speed and an 15% improvement in inference speed compared to the standard top-2 MoF, in the x A800- NVLink scenario where communication accounts for 15% of the total MoE time. In LLaMA2-MoE, our ScMoE accel- erated training by 1.14x and inference by 1.21 x, demon- strating superior efficiency compared to other methods.
1. VISION MODEL
Table shows that ScMoE (Pos-2) and the standard top- 2 MoE attain comparable accuracy of 79.3%, while the shared-expert MoE delivers the highest accuracy, with a marginal increase of 0.2%. In 8 A30PPCe where commu- nication overhead accounts for 60% of the total MoE time, ScMoE (Pos-2) exhibits 30% speed improvement in training and 40% in inference compared to the standard top-2 MoE.
6

---

Shortcut-conneeted Expert Parallelism for Accelerating Mixture of Experts
Table Comparison of zero-shot evaluation and end-to-end speedup of training and inference (one iteration) ) for the pre-trained GPT2-MoF
and LLaMA2-Man MoE models with various architectures in the 8x A800-NVLiigg scenario, using standard top-2 MoE the baseline.
Model
Method
Train
Inference
HellaSwag
PIQA
WinoGrunde
BoolQ
ARC-E
OBQA
RACE
MathQA
AVG.(t)
Standard top-2
27.53
59.19
48.62
59.72
38.43
25.20
23.83
20.37
37.86
GPT2-MoE
Shared- Shared-Expert
1.04x
1.06x
27.23
59.09
51.22
60.00
38.85
26.60
25.07
20.57
38.58
Our ScMoE
1.12x
1.17x
27.70
39.25
52.09
60.76
39.23
25.40
24.98
20.10
38.69
Standard top-2 Shared-Expert
28.40
60.07
50.83
58.26
38.72
24.60
25.17
21.07
38.39
LLaMA2-MoE
1.06x
1.11x
29.08
60.01
50.91
60.92
25.45
20,80
Our ScMoE
1.14x
1.11x
29.09
51.38
57.25
38.89
26.40
27.08
21.07
38.96
Top?
Top2
9.1
Top2
Tpp2-P
21
Top2-P
8.8
Top2-P
12:3
15.0
Comp
Top1
Comp.
Top1
Comp.
Overlap
Qverlap
Ove rlap
Topl-P-
Comm.
Top1-P
Commm
Top1-P
Comm.
+SE1
Top1
+SE1
Tog1
7.2
+SE1
Tap1
5:0
ScMoE
12.7
ScMoE
ScMoE
6.8
15 Time (ms)
20
10
12
Time (ms)
Time (ms)
(a) Node (8x A30-PCI)
(b) Node (8x00NN-VLLinn)
(c) 2 Nodes (16xA800-NVLii))
Figure 8. Overhcad analysis for each pair of Block-MLP and Block-MoE within Swin model, deployed across three diffcrent distributed scenarios. "Topk" denotes the standard top-k MoF, while the one followed by the suffix "P" indicates using pipeline optimization as implemented by Tutel (Hwang et al., 2023). "TopI+SE1" refers to shared-expert MoE.
4.2.3. ANALYSIS OF OVERHEAD AND ACCELERATION
5. Discussion
In addition exhibiting the end-to-end speedup of ScMoE in Tables I and 2, we delve into a detailed analysis of the overhead and the acceleration effect with our overlapping strategy, which can be generalized to other MoE models.
The empirical results presented in Section 4.2 have demon- strated that our proposed ScMoE architecture facilitates effi- ciency optimizations without compromising model quality. Subsequently, we delve into a more thorough examination of the proposed shortcut connection, uncovering potential underlying reasons for its algorithmic effectiveness, and identifying opportunities for further development.
In the communication-intessiie 8x A30-CCle scenario (Fig- ure 8(a)), our ScMoE overlaps 70% communication time, resulting in 27% speed improvement over shared-expert MoE, a 42% improvement over the pipelined standard top-2 MoF, and a 15% improvement over the pipelined standard top-1 MoE. In the 8 x A800-NVLink scenario (Figure 8(b)), which features almost minimal communication overhead, our approach maintains its acceleration by fully overlapping.
5.1. Delve into the Proposed Shortcut Connection
5.1.1. ANALYSIS OF GATING BEHAVIORS
Firstly, we investigate the use the same MoE module to select the top-1 expert twice for processing each input token's current-layer and preceding- representations, respectively. As illustrated in Figure (a), we observe that the same gating network typically selects the same expert for the two representations of most tokens. As the training progresses, the token percentage of repeating selection ini- tially escalates, peaking at 98%, and then diminishes, with a significant drop manifested in the last MoE sub-layer.
In multi-node scenario (Figure 8(c)), with 16 A800- NVLink across two nodes, communication incurs more sig- nificant overhead than in the single-node 8 A800-NVLiikk scenario due to the (Li et al., 2020). Here, our ScMoE achieves complete overlap, resulting in a 24% speed improvement over the shared-expert MoF, 43% improvement over the pipelined standard 2
inter-node Ethernet
Next, we measure the L.2 distance (similarity) between each token's preceding-layer and current-layer representations. Figure 9(b) illustrates that, as training advances, the L2 dis- tance initially decreases with network depth, then increases, and ultimately reaches its maximum value in the final layer. Since the gating network is used to classify the representa-
In general, our ScMoE delivers significant acceleration over the standard top-2 MoE, and even outperforms the top- 1 MoE when communication exceeds approximately 20% of the total MoE time. Additionally, our ScMoE can fully overlap communication in scenarios where communication does not exceed an estimated 50% of the total MoE time.

---

Shortcut-conneeted Expert Parallelism for Accelerating Mixture of Experts
100
0.3
0.24
0.7
Taken Percentage (h)
90
0.22
L2 12-Distance
Gating Score
0.5
80
Gating
Epoch
0.18
70
Epoch 3D
0.3
0.16
Epoch
0.2
2-
60
5 9 Layer
13
15
5
7
11
13
15
17
3
5
9 Layer
11 13
17
5
11 13
15
17
Layer
Layer
(a) Repeated selection
(b) L.2-Distacce
(c) Preceding-layer score
(d) Current-layer score
Figure 9. Results from the analysis of the proposed shortcut connection. during the 90-epoch training (including 10-epoch warm-up) SwinV2-MoF-S model (Hwang al., 2023). Employing the same MoE. module to select the top-1 expert twice processing each input token's two representations from the current and preceding layers, respectively, (a) illustrates the percentage of tokens that retain the same expert selection across the current layer and preceding layer, (b) shows the L2 distance between these two representations. Using the DGMoE, which imposes constraint against repeatedly selecting the same expert, (c) presents the average gating score for the preceding- layer representations, (d) displays the average gating score for the current-layer representations.
tions, this similarity may lead to the repeated selection of the same experts, as evidenced by the correlation between the results Figure 9(a) and 9(b).
Furthermore, we trained an experimental MoE model in- corporating specialized MoE modules that select the top-1 expert twice for processing each input token's 's current-layer and preceding-laye representations. We observe that this experimental architecture achieves model quality equiva- lent to the standard top-1 1 MoE, despite incurring the same computational cost as the top-2 MoE. Interestingly, this architecture can achieve model quality comparable to the standard top-2 MoE by imposing constraint on the MoE module that ensures the selection of different expert for the current layer than for the preceding layer. We refer to this enhanced experimental architecture as DoubleGating MoE (DGMoE), with further details provided in Appendix A.2. With this constraint, we observe gating score behaviors are similar to those of the standard top-2 MoE (Riquelme et al., 2021), as illustrated in Figure 9(c) and (d).
(a) Swin-MoE-SS MoE--MM
(b) GPT2-MoE- MoE--MMM-MM---E MoE-Medium
Figure 10. Analysis of cosine similarity in intermediate represen- tations. The representations include the input to the first layer (denoted as 'In') and the outputs of the Attention (e.g.. '1A') and MLP/MoE (c.g.. "IM') sublayers within each Transformer block.
Moreover, analyze the similarities in the intermediate representations of the Swin-MoE-Smail and GPT2-MoE. Medium models, which use the standard top-2 MoE, as illustrated in Figure 10(a) and 10(b). is evident that the representations from adjacent Transformer blocks exhibit a cosine similarity close to 1.0, highlighting their high degree of similarity. Consequently, our proposed ScMoE archi- tecture assigns distinct experts the representations of each token (a shared expert for current-layer representa- tions and routed experts for preceding layer representationss, thereby preserving behavior akin to the standard top-2 and shared-expert MoE architectures and ensuring comparable model quality. Similar observations in LLaMA2-MoE and OLMoE (Muennighoff al., 2024), demonstrated in Appendix A.9, further confirm the generalizability of our ScMoE other models.
5.1.2. ANALYSIS OF SIMILARITY IN REPRESENTATIONS
Based the observations mentioned above, believe that the similarity between each token's preceding-laye and current-layer representations is crucial to understand- ing these outcomes and validating the effectiveness of our proposed ScMoE model. Assuming that the representations of the preceding layer and the current layer are identical, utilizing the same expert process these representa- tions is equivalent to employing single expert to process only the current-layer representations, thereby resulting in model quality comparable to that of the standard top-1 MoE. On the other hand, assigning distinct experts to the two rep- resentations each token is equivalent to activating two experts process the current-layer representations, thereby achieving model quality similar that of the standard 2.
In addition, we provide a theoretical analysis of our pro- posed ScMoE architecture in Appendix 1, elucidating the propagation of gradients to guarantee consistent training and preserve model quality.
8

---

Shortcut-conneetel Expert Parallelism for Accelerating Mixture of Experts
5.2. Configuration of ScMoE Architecture
Table 3. Comparison of the final validation loss of GPT-2 MoE pre-training across various MoE methods and configurations.
Coeflicient Gating Network. In contrast to the gate-routed expert, the shared expert is fixed to process all representa- tions without computing a gating score through the gating network. Therefore, some work (Qwen, 2024; Rajbhan- dari et al., 2022) employs a coefficient gating network CG to generate the coefficient for combining the outputs of gate-routed and shared experts. Specifically, the coefficient gating network a linear layer that uses the MoE module's input representation as its input to generate the coefficient.
MoF Method
Configuration
Final Validation loss (L)
Standard top-2 MoE
3.270405
Shared-Expert MoB
3.240592
Direct Add
3.236811
Our ScMoR (Pos-2)
CG-1
3.224763
CG-2
3.232943
Pos-1
3,237530
Our ScMoE (CG-1)
Pos-2
3,224763
Pos-3
3.241349
We conduct experiments on ScMoE using three distinct methods for combining the outputs of gate-routed experts and shared experts: (1) Direct Add, (2) CG-1, and (3) CG-2. The Direct Add method, indicated by Equation 6, involves directly summing the outputs from both the shared expert and the gate-routed expert. For each input token æ R", the MoE outputs using CG-1 and CG-2, which generate efficients for the combination, can be expressed as follows: CG-l:
Pos-2 (I.0 Pos-1)
3.225626
Pos-1 while the remaining modules employ Pos-2, config- uration referred as Pos-2 (LO Pos-1). This setup results in slightly higher loss compared to Pos-2 alone. Observa- tions of varying shortcut-connedt superior performance of Pos-2 suggests the model quality of ScMoE is not necessarily improved by connecting more similar or dissimilar intermediate representations.
positions reveal that the
coef = Sigmoid(Wca x), Weo E EiI".
(14)
Consequently, we select the ScMoE configuration of CG-1 and Pos-2 for the experimental GPT2-MoE model, with its evaluations presented in Table 2.
N
MoE(z) = coef- - (E(()))()+ +
Σ G(z),E(zz),
(15)
CG-2:
coef = so softmar(Wec. x), Woo E 22
(16)
5.3. Optimization for Memory-Limiee Limited Inference
Existing studies (Hwang et al., 2024; Yi et al., 2023) offload expert parameters to CPU memory in memory-limite infer- ence scenarios where GPU cannot store the full MoE model. These studies utilize information in preceding layers to pre- dict expert selection for the current MoE layer, enabling early expert migration from CPU to GPU and overlapping it with model computation. In contrast to existing speculative expert migration methods, we implement an expert offload- ing strategy with overlapping determinate migration, built upon our ScMoE that inherently advances expert selection to the preceding layer. The experimental results demon- strate that our expert offloading strategy reduces peak GPU memory usage by up to 60% and decreases expert migration costs by up to 75% through overlapping with computation. More details are shown in Appendix A.3.
MoE(x)) coef(0) SSE(z) + coeff1] E())
(17)
As shown in Table 3, the configuration of CG-1 achieves the lowest final validation loss among the three combination methods, all of which are set to Pos-2. Moreover, ScMoE models with three configurations consistently outperform both the standard 2 and the shared-expert MoE (config- ured with CG-2 according to (Rajbhandari et al., 2022)).
cooneeed Position. While the Pos-1 configura- tion can maximize the potential overlap duration when MoE is placed in every block, using different configurations of conneett-ooneeeed positions (Pos-1, Pos-2 or Pos-3) when placing MoE in every second block will result in variations in overlap duration and model quality. Therefore, we con- duct experiments to identify its optimal configuration.
6. Conclusion
As shown in Table 3, ScMoE (Pos-2) achieves the lowest
The inherent dependency between communication and com- putation in conventional distributed MoE models hinders parallel optimization techniques to improve execution effi- ciency. To address this, we propose a shorteut-connectee MoE (ScMoE) architecture, and develop a communication overlapping parallel strategy. Through empirical evaluation and theoretical analysis, our approaches demonstrate bet- ter execution efficiency while maintaining or exceeding the model quality of existing methods. In addition, we provide an insightful analysis and discussion of ScMoE.
final validation loss among the configurations tested, all of which utilize the CG-1 setup. This outcome mirrors the findings from the vision experiments detailed in Table 1, where Pos-2 also delivers the highest accuracy.
As illustrated in Figure 10(b), the input and intermediate representations of the first layer differ significantly from those of subsequent layers, with the MLP/MoE altering the representations more substantially than Attention. There- fore, we explore modifying the first MoE module to utilize

---

Shortcut-comnected Expert Parallelism for Accelerating Mixture of Experts
Acknowledgements
of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 2924 2224-2293,,-2 2019.
We thank the anonymous reviewers for their valuable com- ments. This work was supported in part by the Guang- dong Provincial Project (No. 2023QN10X252), the Guang- dong Basic and Applied Basic Research Foundation (No. 1223A1515110333 and GDIC. This research was con- ducted on the High-Performance Computing Platform of HKUST(GZ).
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Chal- lenge. arXiv preprint arXiv: A003v:/803..05500 0803.00437, 2018.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, Zeng, W., Yu, X., Wu, Y., et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of- Experts Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1280-1297, 2024.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Databricks. Introducing DBRX: A New State- of-the-Art Open LLM, March 2024.
URL
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 Technical Report. arXiv preprint ar 231v:23033007744 2023.
https://wwe www.wwwatabbcs...oom com/blog/ Antroducingg-drrr newpaaicwnnn-rr---eeee-ew-----ee
art open-llmm
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and
Kaiser, L. Universal Transformers, In International
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorsk, R., Choi, Y., and Hajishirzi, H. MathQA: Towards Interpretable Math Word Problem Solving with Operation Based For- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2357-2367, 2019.
Conference on Learning Representations, 2018.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In International Conference Learning Representations, 2021.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reason- ing about Physical Commonsense in Natural Language. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 34, pp. 7432-7439, 2020.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M. P., Zhou, Z., Wang. T., Wang, Y. E., Webster, K., Pellat, M., Robinson. K., Hellstee, K. S., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui, C. GLaM: Efficient Scaling of Language Models with Mixture-of- ofxture-of-Experts.. In Interna- tional Conference on Machine Learning, pp. 5547-5569. PMLR, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language Models Are Few-Shot Lear- ers. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Chen, S., Jie, Z., and Ma. L. LLaVa-MoLEE: Sparse Mixture of LoRA Experts for Mitigating Data Con- flicts in Instruction Finetuning MLLMs. arXiv preprint arXiv:2401. 16160, 2024.
Du, Z., Li, S., Wu, Y., Jiang, X., Sun, J., Zheng, Q., Wu, Y.. Li, A., Li, H., and Chen, Y. SiDA: Aparsity-Inspieee Data- Aware Serving for Efficient and Scalable Large Mixture- of of-Experts Models. Proceedings of Machine Learning and Systems, 6:224-238, 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. The Journal of Machine Learning Reseurch, 23(1):52325570, 2022.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings
Foley, D. and Danskin, J. and NVLink Interconnect. 2017.
Pascal GPU IEEE Micro, 37(2):7-17,
10

---

Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang. J., and Subrahmanian, V. Higher Layers Need More LoRA Experts. arXiv preprint arXiv:2402.08562, 2024.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. ALBERT: A Lite BERT for Self- supervised Learning of Language Representations, In International Conference on Learning Representations, 2019.
Gokaslan, A. and Cohen, V. OpenWebText Cor- http:://SkyionOO// OpenWebTexttCoppu 2019.
pus.
i
Lepikhin, D., Lee, H., Xu, Y., Chen, D.. Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. GShard: Scaling Giant Models with Conditional Computation and Auto- matic Sharding. In International Conference on Learning Representations, 2021.
Gou, Y., Liu, Z., Chen, K., Hong, L., Xu, H., Li, A., Ye- ung, D., Kwok, T., and Zhang, Y. Mixture of Cluster- Conditional LoRA Experts for Vision-Language Instruc- tion Tuning. arXiv preprint arXiv:2312.12379 2023.
Li, A., Song, S. L., Chen, J., Li, J., Liu, X., Tallent, N. R., and Barker, K. J. Evaluating Modern GPU Intercon- nect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect. IEEE Transactions on Parallel and Distributed Systems, 31(1):44-10,, 2020.
He, J., Qiu, J., Zeng, A., Yang. Z., Zhai, J., and Tang, J. FastMoE: A Fast Mfxture-of-Expert Training System. arXiv preprint vrXiv:2103.1326. 2021.
He, J., Zhai, J., Antunes, T., Wang, H., Luo, F., Shi, S., and Li, Q. FasterMoE: Modeling and Optimizing Train- ing of Large-Scare
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., al. Jamba: Hybrid Transforme-Maamba Language Model. In International Conference on Learn- ing Representations, 2025.
Dynamic Tra-irained Models. In Proceedings of the 27th ACM SIGPIAN Symposium Principles and Practice of Parallel Programming, pp- 120-134, 2022.
Huang, G., Liu, Z.,, van der Maaten, L.,, and Weinberger, K. Q. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF International Conference on Computer Vision, pp. 9992- 10002. IEEE, 2021.
Huang. Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M. X., Lee, H.. Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems, volume 32. pp. 103-1 12, 2019.
Lu, J.. Batra, D., Parikh, D., and Lee, S. ViLBERT: Pre- training Agsk-Agi Visiolinguistic Representations for
Tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Hwang, C., Cui, W., Xiong, Y., Yang, Z., Liu, Z., Hu, H., Wang, Z., Salas, R., Jose, J., Ram, P., et al. Tutel: Adap- tive offxturr-of-Expertt at Scale. Proceedings of Machine Learning and Systems, 5:269 287, 2023.
Mayer, R. and Jacobsen, H.-A Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools. ACM Computing Surveys (CSUR), 53(1):1-37, 2020.
Hwang, R., Wei, J.. Cao, S., Hwang, C., Tang, X., Cao, T., and Yang, M. Pre-gated MoE: An Algorithn-System Co- Design for Fast and Scalable tixture-of-EEpper Inference. In 2024 ACM/EEE Annual International Symposium on Computer Architecture (ISCA), pp. 1018-1031. IEEE, 2024.
Merity, S., Xiong, C., BradBurry J., and Socher, R. Pointer Sentinel Mixture Models. In International Conference on Learning Representations, 2017.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 2381-2391, 2018
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. I., Hanna, E. B., Bressand, F., et al. Mixtral of Experts. arXiv preprint ar arxiv:2401.0 N..02400. 2024.
Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., et al. OLMoE: Open Mixture-of- Ex-errt Language Models, arXiv preprint arXiv:2409.022 020600 2024.
Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. RACE: scrge-scale ReAding Comprehension Dataset From Ex- aminations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 785-794, Copenhagen, Denmark, September 2017. Asso- ciation for Computational Linguistics.
Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., and Houlsby, N. Multimodal Contrastive Learning with LIMoE: the Language- Image Mixture of Experts. In

---

Shortcut-comneeted Expert Parallelism for Accelerating Mixture of Experts
Annual Conference on Neural Information Processing Systems, 2022.
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. ZeRO-Infinit:: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. In Proceedings of the In- teratiioal Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14, 2021.
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, B., and Za- haria, M. PipeDream: Generalized Pipeline Parallelism for DNN Training. In Proceedings of the 27th ACM Symposium Operating Systems Principles (SOSP), pp. 1-15, 2019.
Rajbhandari, S., Li, C., Yao, Z.,, Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeed- MoF: Advancing Mixture-of-Experr
Inference and Train- ing to Power Next-Generation Al Scale. In International Conference on Machine Learning, pp. 19832-18346 PMLR, 2022.
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient Large-Scale Language Model Training GPU Clusters Using Megatron-LM. In International Con- ference for High Performance Computing, Networking, Storage and Analysis (SC), pp. 58, 2021.
Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Pinto, A. S., Keysers, D., and Houlsby, N. Scaling Vision with Sparse Mixture of Experts. In Annual Conference on Neural Information Processing Systems, pp. 8583 8595, 2021.
Nie, X., Zhao, P., Miao, X., Zhao, T., and Cui, B. HetuMoF: An Efficient Trillion-scale Mixture-of Expert Distributed Training System. arXiv preprint ar Xiv:22033.1468. 2022.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: An Adversarial Winograd Schema Chal- lenge at Scale. Communications of the ACM, 64(9):99- 106, 2021.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. Fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstra- tions), pp. 4 2019.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously Large Neural Networks: The Sparsely- Gatee Mixture-of- Experts Layer. In International Conference on Learning Representations, 2017.
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training Language Models to Follow Instructions with Human Feedback, Advances in Neural Information Processing Systems, 35:27730 27744, 2022.
Shen, L., Wu, Z., Gong, W., Hao, H., Bai, Y., Wu, H., Wu, X., Bian, J., Xiong, H., Yu, D., et SE-MoE: A Scalable and Efficient Mixture-of -f-errt Distributed Training and Inference System, arXiv preprint arXiv:2205. 100004 2022.
Patel, P.. Choukse, E.. Zhang, C., Shah, A., Goiri, í. Maleki, S., and Bianchini, R. Splitwise: Efficient Gener- ative LLM Inference Using Phase Splitting. In 2024 ACM/IEE 51st Annual International Symposium on Computer Architecture (ISCA), pp. 118-132. IEEE, 2024.
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., and He, Y. Scaling Vision-Language Models with Sparse Mixture of Experts. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11329-11344, 2023.
Qwen. Swenl.5-Moll Matching 7B Model Perfor- mance with 1/3 Activated Parameters", February 2024. URL httptt/p:// http///wlllmmmmmmmmm// warelll..g.t...... wtt.uu...//blo. oven-moe/.
Singh, S., Ruwase, O., Awan, A. A., Rajbhandari, S., He, Y., and Bhatele, A. Hybrid Tensor Tensor-Expert Data Parallelism Approach to Optimize Mfxture-of-Experts Training. In Proceedings of the 37th International Conference on Su- percomputing (ICS), pp. 203-214, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language Models Are Unsupervised Multitask Learners. OpenAl blog, 1(8)99 2019.
Smith, S., Patwary, M., Norick. B., LeGresley, P., Rajbhan- dari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zheng, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., He, Y., Houston, M., Tiwary, S., and Catanzaro, B. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. arXiv preprint arXiv:2201. 2022.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. ZeRO: Memory Optimizations toward Training Trillion Parame- ter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Anal- ysis, pp. 1-16. IEEF, 2020.
12

---

Shortcut-conneeted Expert Parallelism for Accelerating Mixture of Experts
Soboleva, D., Al-Khateeb, F,, Myers, R., Steeves,
Xue, F., Zheng, Z., Fu, Y., Ni, J., Zheng, Z., Zhou, W., and You, Y. OpenMoE: An Early Effort Open Mixture-of- Experts Language Models. In Forty-first International Conference on Machine Learning, 2024.
J. R., Hestness, J., and Dey, N. SlimPajama: A 627B Token Cleaned and Deduplicated Version of RedPajama. www.:lllwwccooollliiiiiaaooooona
Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., and Xu, M. EdgeMoF: Fast On-Device Inference of MoE-based Large Language Models. arXiv preprint arXiv:2308. 14352, 2023.
June 2023. URL https: https:://hugiinggfae.. /o/ datasets wwtp///eeebbeeimeeeeeeeee
627B.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open Foundation and Fine- Tuned Chat Models. arXiv preprint arXiv:2307.09286 009888, 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwap Can Machine Really Finish Your Sen- tence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is All you Need. In Annual Conference on Neural Information Processing Systems, pp. 5998-6008, 2017.
Zhang, P., Li, X., Hu, X., Yang. J., Zhang, L.,, Wang, L., Choi, Y., and Gao, J. VinVL: Revisiting Visual Repre- sentations in Vision-Language Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.
Wang, S., Wei, J., Sabne, A., Davis, A., Ilbeyi, B., Hecht- man, B., Chen, D., Murthy, K. S., Maggioni, M., Zhang, Q. Kumar, S., Guo, T., Xu, Y., and Zhou, Z. Over- lap Communication with Dependent Computation via Decomposition in Large Deep Learning Models. In Pro- ceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pp. 93-106 ACM, 2023.
Zhang, Z., Yang, D., Xia, Y., Ding. L.. Tao, D., Zhou, X., and Cheng, D. MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 167-177, 2023.
Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., Gonzalez, J. E., and Stoica, I. Alpa: Automating Inter- and Intra- Operator Parallelism for Distributed Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 559-578, 2022.
Wei. J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., et al. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research, 2022a.
Zhou, K. Yang, J., Loy, C. C., and Liu, Z. Learning to Prompt for Vision-I Vision-Language Language Models. International Jour- nal of Computer Vision, (30(9):2337-2345 2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-Thoughh Prompting Elicits Reasoning Large Language Models. Advances in Neural Information Processing Systems, 35: 24824 24837, 2022b.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In The Twelfth International Conference on Learning Representations, 2023.
Wu, B., Liu, S., Zhong, Y., Sun, P., Liu, X., and Jin, X. Loongserve: Efficiently Serving Long-Conters Large Language Models with Elastic Sequence Parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pp. 640-654, 2024.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. ST-MoE: Designing Stable and Transferable Sparse Expert Models. arXiv preprint arXiv:2202.08900 2022.
Wu, X., Huang, S., and Wei, F. MoLE: Mixture of LoRA Experts. In The Twelfth International Conference Learning Representations, 2023.
Xue, F. Shi, Z., Wei, F., Lou, Y.. Liu, Y., and You, Y. Go Wider Instead of Deeper. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pp. 8779- 8787, 2022.
13

---

Shortcut-comneeted Expert Parallelism for Accelerating Mixture of Experts
A. Appendix
Block-MoE
A.I. Theoretical Analysis
Block-MLP
E1
E2
E3
E4
In this section, we delve deeper into the understanding of our proposed shorteut-connected MoE (ScMoE) architec- ture, presenting theoretical foundation focused on the propagation of gradients guarantee consistent training and preserve model quality. Our analysis is confined to the ScMoE (Pos-2) architecture as depicted in Figure 4(b); however, the same principles and derivations can be eas- ily extended to other chortcut-connectedd MoE architectures. Building upon Equations 7 to 10, we can derive
Block-MoE
Gate
Block-MLP
Block-MoE
[N/2)X
Self-Attention
Block-MLP
Block-MoE
Block-MLP
MLP
Block-MoE
H1+1 H (MLP")(HH))
Self-Atrention
BLock-MLP
+ MultiHead (1)) "HH + MMP"
DoubleGating MoE (DGMoE)
+ (H,
"H
Figure 11. Illustration of the experimental DoubleGating MoE (DGMoE) architecture.
(18)
+ MultiHead (1 (H, + +LLP (M(M) (H)
+ Σ C(HMH) ) ).B.HHH
A.2. Analysis of the DoubleGating MoE (DGMoE)
To delve deeper into our architecture with shorteut con- nection, we introduce the DoubleGating MoE (DGMoE) architecture, which employs dual top-1 gating mechanisms to independently process the representations from the pre- ceding and current layers, as illustrated in Figure 11. Build- ing upon Equations 7 to 10, and contrasting with ScMoB, DGMoE can be formulated as
H H1-1 + MultiHead (1) (H4-1).
(19)
It is observable that Equations 18 and 19 share an identical structural expression. Consequently, we consider each pair of Block-MoE and Block-MLP layers as single entity, and every sub-layer, denoted as F, with its corresponding parameters W. conforms the equation
Z1+1 = xR + Fw: (xz)-
(20)
N Σ
HDOMME Hi41 (GHHH ) F(HAT)
Here, 24 represents the input, and 1 represents the output of the 1-th sub-layer. By applying this relationship recur- sively, the output of the uppermost I--th sub-layer, dz, can be deduced as follows
(23)
+ GCHAHHHH (.H
where R.DGMoE refers to the output from the MoE module.
L-1
5T. = I1 +
(m).
(21)
However, as delineated in Equation 23, a potential issue arises when a token at the current layer selects the same top-1 expert as the preceding layer, inadvertently collapsing the intended top-2 gating mechanism into a de facto top-1 gating mechanism. To mitigate this, introduce con- straint that ensures the activation of two distinct experts. In practice, this is achieved by first documenting the indices of experts triggered by the preceding- representations. Subsequently, if the preceding layer representation coinci- dentally targets the same expert as the current layer, that is, if T Topk' (H (H," 1) TopK (H (H ), 1), we activate the second-highest-ranl for the current layer, i.e., TopK (H (HH ),2)))
Let's consider the loss function as E. Using the chain rule, we can calculate the derivative of the loss with respect to 21, and we have
30 =
30 OEL =
30 (1 dzi
L-1 Σ Fms 1m1
dz:
dz dz dz
(22) It's clear that the additive component of the error gradient 88 der. ensures direct information propagation back to any sub-layer z₁. Additionally, its advantage is that the number of product elements on the right side is independent of the network's 's depth. Therefore, as L increases, it is less likely to encounter the gradient vanishing or exploding problem, ensuring stable training and sustained performance levels in our proposed MoE architectures.
expert from the top-2 selection
As illustrated in Table 6 and Table 7, our DGMoE achieves comparable accuracy to the standard top-2 MoE across both vision and language tasks. Mcanwhile, our ScMoE demon- strates performance more akin to the shared-expert MoF.
14

---

Shortcut-comneetel Expert Parallelism for Accelerating Mixture of Experts
A.3. Shortcut-connected MoE for Optimizing Memory-Limited Inference
CPU Memory
E2
E0
SE
While MoE effectively enhances LLMs in terms of model quality, it faces significant deployment challenges during on-device inference due to high memory demand. A com- mon approach is to offload expert parameters to CPU mem- ory (Shen et al., 2022) in scenarios where GPU memory insufficient to store the entire MoE model. Moreover, decoder-only models use an autoregressive process for nat- ural language generation (NIG) inference tasks, allowing for per-token processing of MoE. Specifically, only the two activated experts (top-2 gating) for each token need to be transferred from CPU to GPU memory for computation, thereby reducing peak GPU memory usage.
Non-MoE
E2
E3
Self-Attention
CPU
f Gate
MLP
EO
Gate
Memorr
E1
E2
Non-MoE
E3
Self-Attention
(a] Speculative
(b) ScMoE's Determinate Expert Migration
Expert Migration
Figure 12. Illustrations of various expert migration methods to improve the efficiency expert offloading: (a) speculative expert migration, exemplified by Pre-gated MoE (Hwang et al., 2024), and (h) our ScMoF's determinate expert migration. The red dashed line indicates expert selection and the transfer of expert parameters from CPU memory to GPU memory, while the black or red solid lines represent the data flow of representations processed by the Attention, MLP, and expert modules.
Since the migration of activated expert parameters from CPU GPU, which occurs after expert selection, blocks ex- pert computation until the transfer is complete, existing stud- ies (Hwang al., 2024; Yi et al., 2023; Du et al., 2024) have explored prefetching the experts. For instance, Pre-gated MoE (Hwang et al., 2024) uses information from preceding layers to predict expert selection, allowing for preloading of expert parameters into GPU memory, as shown in Fig- ure 12 (a). This method enables overlapping the expert migration duration with the computation of preceding mod- ules. Moreover, speculative expert migration methods adjust only the expert selection process, while expert computation continues along the same data flow of representations as in standard MoE.
1.0
3.5
GGUU-nlly Offioad
3.4
3.0
2.8
0.8
Offioad-Async
Normalized
2.5
0.6
0.5.0.
Norma
2.0
1.8
0.4
1.5
1.0
0.5
0.0
0.0
Medium
XL
Medium
XL
(a) Peak GPU Memory Usage
(b) MoE Block Latency
However, speculative expert migrations can suffer from estimation inaccuracies, they deviate from the original logic of pre-trained models, potentially reducing inference accuracy. In contrast, proposed ScMoE architecture uti- lizes the gate-routed expert to compute the preceding- laayer representations, inherently facilitating early expert migra- tion well before the expert computation in the current layer. This allows us to implement an expert offloading strategy with overlapping determinate migration, maintaining the pre-trained logic.
Figure 13. Peak GPU memory usage (a) and MoE block latency (b) for various memory-limite inference methods applied to the GPT2-MoF-Medium experts per MoE module) and GPT3-MoE- XL models using ScMoE. "GPU-onLy" indicates that the entire model is stored GPU memory. "Offload" refers to our strategy of offloading expert parameters to CPU with blocking expert mi- gration. "Oflodd- -Aynnce" denotes use of asynchronous expert migration to overlap its duration.
lection and issues asynchronous migration of the activated expert, as illustrated in Figure 12(b). This approach allows expert migration to overlap with the computation duration. Importantly, expert selection in our method adheres to the logic of the pre-trained ScMoE model, without speculation.
Additionally, existing expert migration methods cannot be adapted to overlap communication in expert parallelism. This is because they do not decouple dependencies in the data flow of expert processing representations, and there- fore cannot adjust the All-to-All communication of these representations
A.3.2. EVALUATION
We evaluate our proposed expert offloading strategy on models with our ScMoE (Pos-2) architecture, using plat- form with single A30-PCle GPU, As demonstrated in Figure 13(a), our expert offloading strategy reduces peak GPU memory usage by 50% for the GPT2-MoE-Medin model and by 60% for the GPT3-MoE-XL. model when deployed in the inference scenario using a single A30-PCle GPU. Furthermore, it anticipated that models with more
1. EXPERT OFFLOADING STRATEGY
We implement an expert offloading strategy that keeps non- expert and shared expert modules in GPU memory while offloading other gate-routed experts to CPU memory. Af- ter the Attention module in the preceding layer generates intermediate representations, the gate determines expert se-
15

---

Shortcut-comnected Expert Parallelism for Accelerating Mixture of Experts
Table Comparison of validation perplexity and end-to-end speedup analysis of train and inference (one iteration) for our pre-trained SPT33-MEE XL (Brown al., 2020) models with var- ious architectures in 8 A800-NVLink scenario, using standard MoE with lop-2 gating as the baseline. "ScMoE-2" refers to the activation of one shared expert and two gate- gate-routed experts.
Table 5. Comparison of top-1 accuracy on the ImageNet- IK set for SwinV2-MoEs models, using Direct Add and CG--
Model
CG-1
Direct Add
Shared-Expert MoE
79.53% %
79.02% %
Our ScMoE (Pos-1)
79.14%
78.78%
Our ScMoE (Pos-2)
79.38 %
78.98 %
Model
Validation (Perplexity))
Train Inference (Speedupt)
Our ScMoE (Pos-3)
79.20%
78.29%
(Specdupt)
Standard t
17.52
Table 6. Comparison of top-1 accuracy on the ImageNet- test set for V2 and SwinV2-MoB-a models with various architectures: top-2/top-1 gating standard MoE, shared-expert MoE, our DGMGE, and ScMoE, each pre-trained for 90 epochs on the ImageNet- IK classification dataset.
Our ScMoE
16.46
1.12x
1.18x
Standard top-3
17.26
0.94x
0.92x
Our ScMoF-2
16.27
1.05x
1.08x
gate-routed experts in each MoE module will experience larger percentage reduction in GPU memory usage.
Model
SwinV2-Moo MoE-S (Acc@ (1)
SwinV2-MoE- (Acc@) 11)
Since the offloaded expert parameters must be loaded into the GPU memory for expert computation, the blocking ex- ecution of this expert migration results in significant over- head. As shown Figure 13(b), the blocking expert mi- gration introduces an additional overhead of 80% in GPT2- substantially increasing the MoE block latency. To mitigate this issue, our strategy of asynchronously executing the determinate expert migration effectively reduces the additional costs by 75% in 122--MoE-Meddnn and 25% in GPT3-MoE-XX
Standard top-2 MoE
79.33%
80.48%
Standard top-1 MoE
78.95%
80.05%
Shared-Exper MoE
79.53%
30.62%
Our DGMgE (Pos-2)
79.35%
80.51%
Our ScMoF (Pos-2)
79.38%
80.56 %
MoE-Medium and 240% in GPT3-MoE.
Although activating more experts incurs higher time costs, the efficiency improvements of our overlapping strategy remain significant. For instance, our ScMoE-2 requires merely 95% and 93% of the time cost necessary for the standard top-2 MoE respectively in training and inference, despite processing increased computational loads.
Furthermore, it is evident that expanding the model size from Medium XI. significantly raises the cost proportion related to expert migration. This is because the per-token decoding process during inference is memory-bound (Patel et al., 2024; Wu et al.. 2024). The larger model size leads to a proportional increase in the duration of memory transfer, without a corresponding increase in computation time.
A.5. Coefficient Gating Network in Vision Task
As shown in Table 5, the incorporation of the coefficient gating network significantly enhances model performance in our experimental vision tasks. In the absence of the coefficient gating network, the quality of MoE architectures with shared experts declines from that of a standard top-2 MoE that of standard top-1 MoE, despite maintaining the same computational volume the standard top-2 MoE.
A.4. Analysis of More Activated Experts
As increasing the number of activated experts within stan- dard MoE is correlated with enhancements in model qual- ity. we implement this augmentation in our ScMoE by in- creasing the count of gate-routed experts that process the preceding-layer representations, while maintaining the pro- cess of current-layee representations. To investigate the ben- efits of more activated experts, we implement the ScMoE-2, which employs top-2 experts for the preceding layer and one shared expert for the current layer.
A.6. Evaluation Across Different Model Sizes
Table 6 and Table 7 illustrate that our experimental MoE architectures consistently achieve analogous model quality across different model sizes, expounded in the detailed analysis within the main body of this paper.
Comparative analyses with the standard top-3 MoE, which has the same computational volumes as our ScMoE-2, re- veal that our ScMoE architectures maintain superiority in both model quality and efficiency, as evidenced in Table 4. Furthermore, akin to the standard MoE, our ScMoE consis- tently improves with additional expert activation, shown by a decrease in validation perplexity from 16.46 with ScMoE 16.27 with ScMoE-2.
A.7. Share MoE Across Multiple Layers via Shorteut Connections
From certain point of view, our shortcut-connecter MoE architectures can be conceptualized as the sharing of one MoE module across multiple transformer layers. Parame- ter sharing across different layers has been validated as a method to enhance parameter efficiency and improve model
16

---

Shortcut-connected Expert Parallelism for Accelerating Mixture of Experts
Table 9 summarizes the hyperparameters for training the Swin-MoF. models including Swin and SwinV2- MoE-B. Specifically, the experiments related to overhead and acceleration analysis in 2-node [16xA80--NVinnk) scenario utilize 16 experts per MoF. module, while other cases use 8 experts. To maintain the comparability of our experiments, we limit our modifications solely to the MoE architectures and keep the hyperparameters and random seeds consistent. In addition, the experimental results re- lated to efficiency are the averages of multiple samples over di different periods.
Table Comparison of zero-shot perplexity on Wiki 103 for our pre-trained GPT2-MoE MoE-Small and GPT2-MoE MeeMMuu (8 experts per MoE module) models with various architectures.
Model
GPTl (Perplexity))
GPT2-MoF-Medium (Perplexity))
Standard 2 MoE
31.60
19.18
Shared-Expert MoE
29.15
17.94
Our DGMoE (Pos-2)
31.52
18.91
Our ScMoE (Pos-2)
29.10
17.62
quality, as evidenced in existing research (Lan et al., 2019;
Experiments on Language Model. For natural language generation (NLG) tasks, we utilize the standard implemen- tations of GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020) and LLaMA-2 (Touvron et al., 2023) from Fairseq (Ott et al., 2019), augmented with Tutel MoE con- struct GPT2-MoF, GPT3-MoE and LLaMA2-MoE models. Specifically, we implement GPT2-MoE and GPT3-MoE by substituting the MLP with MoE in the second Trans- former block of every consecutive pair, while implement I.LMA2-MMF. by by substituting the MLP with MoE in every Transformer block. For models undergoing zero-shot evaluation on downstream tasks such as HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sak- aguchi et al., 2021), BoolQ (Clark et al., 2019), ARC-Easy (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), RACE (Lai et al., 2017), and MathQA (Amini et al., 2019), we pre-train the models using various architectures on a 1B token subset of the SlimPajama-622 eva al., 2023). For models evaluated on WikiText-100 103 (Merity et al., 2017), we conduct pre-training with differ- ent architectures on the Open Webtext dataset (Gokaslan & Cohen, 2019). Table 8 summarizes the hyperparameters for training the GPT2-MoE-Small, GPT2-MoE-MMdiiug GPT3-MoE-XL and LLaMA-MoE models.
Dehghani et al., 2018; Xue et al., 2022; Huang et al., 2017). The empirical analysis of our novel MoE architectures sug- gests that the MoE modules shared across multiple layers via shortcuts could offer more parameter-effiient solu- tion. We conduct experiments on a preliminary architecture DGMoE- Mhaoree which shares a single MoE for two pairs of transformer blocks. It reduces the parameter count from 157M to 124M, while maintaining the same volume of ex- pert computation as the standard top-1 I MoE. The DGMoE- Share achieves a 78.45% accuracy the vision task, incur- ring minimal accuracy decrement of 0.5% relative to the standard top-1 MoE. We anticipate the discovery of more efficient architectures through future explorations. Addition- ally, the optimization of training hyperparameters for the shortcut-connected MoE requires more investigation.
dataset (Sobol-
A.8. Experimental Details
Hardware Configurations. To assess the effectiveness of our proposed overlapping strategy for enhancing ex- pert parallelism, conducted experiments on three hard- ware configurations: 8 A30-PCIe. x A800-NVLink and A6xA800-NLLink (across 2 nodes). These configurations cover scenarios with both high and low communication-too to- computation ratios. Additionally, we evaluate our proposed expert offloading strategy on a configuration with a single A30-PCle GPU.
Experiments Vision Model. To evaluate the efficacy of our MoE architectures on vision tasks, we conduct experi- ments on SwinV2-MoE model, which is a state-of- of-te-----heerr vision transformer model built upon the Tutel MoE frame- work (Hwang et al., 2023; Liu et al., 2021). Specifically, we pre-train the SwinV2-MoE models with various MoB architectures on ImageNet- IK image classification dataset, and subsequently evaluate their accuracy on the correspond- ing test set. It is noteworthy that the integration of the MoE module within S2 is confined to stages 3 and 4, with our architectural enhancements being selectively applied to the MoE modules in stage 3-the deepest submodel. Given our hardware constraints, we configure each MoE mod- ule with 8 experts, assigning one expert per GPU device.
17

---

Shortcut-comnettel Expert Parallelism for Accelerating Mixture of Experts
Table 8. Hyperparametens for GPT-MoE and LLaMA2-MoE models.
Parameter
GPT2-MoB-Smaal
GPT2-MoE-Medium
GPT3-MoE-XL
LLaMA2-MoE
Num. layers
24
Embedding dim
768
1024
2048
2048
Num. attention heads
12
16
32
16
Num. KV heads
12
16
32
4
Num. experts per layer
8
MoE frequency
1/2
1/2
1/2
Num. parameters
323M
1.7B
4.1B
6.7B
Context/sequence length
1K
2K
2K
2K
Capacity factor
2.00
2.00
2.00
2.00
MoE loss coefficient
0.01
0.01
0.01
0.01
Table 9. Hyperparameters for Swin V2-MoB models.
Parameter
SwinV2-MoE-B SwinV
0.8
Image size
x92 192
x92x 192
Window size
12x 12
12x 12
Embedding dim
128
Num. layers
[2, 2, 18,
[2. 2, 18, 2
0M
0.6
Num. attention heads
[3, 6, 12, 24
[4, 8, 16, 32
Num. experts per layer
8/16
8
Batch size
1024
1024
Epochs
Warmup epochs
Base LR
1.25e-4
1.25e-4
12
Warmup LR
1.25e-7
1.25e-7
0.2
Min LR
1.25e-6
1.25e-6
180
Capacity factor
1.25
1.25
MoE loss coefficient
0.01
0.01
A.9. Additoonal Examples of Intermediate
Figure 14. Intermediate representation similarities LLaMA2-
Representations Similarities
MoE.
1.0
06
0M
2M
0.8
34
3M
44.
4M
66
6M
7A
7M
56
9M
106
10M
0.2
134
1380
Figure 15. Intermediate representation similarities in OLMoE (Mucnnighoff et al., 2024).
18


Reviews:
[{"review_id": "eLO8ZCCxEn", "reviewer": "Reviewer_GE4M", "summary": "MoE model is an effective way to scale up model parameters while preserving the inference latency. When training MoE models, due to the extremely large parameters, expert parallelism is widely used to distribute the computational workload. However, this also introduces expensive all-to-all communication cost. This paper proposes a new MoE architecture to overlap communication and computation, which helps to greatly save the training time. Small-scale experiments show that the new architecture can speedup the training while maintaining the same model performance as the original MoE architecture.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. I think the accuracy on benchmark datasets are a bit low to provide meaningful comparisons. It'd be better to train bigger models or longer. It would be great if the idea can be validated at larger model scale and longer training runs. Currently the accuracy on benchmarks are a bit low.\n\nGiven our constraints on available hardware, we encounter difficulties in conducting empirical experiments on large-scale, industrial-level MoE models. Nevertheless, we perform additional experiments on the OLMoE model [1], an advanced MoE model that activates 1B of its total 7B parameters. We initiate training from scratch for both the original OLMoE and the OLMoE+ScMoE models using 40,000 samples from the TuluV3 dataset [2]. As illustrated in the following table, our proposed ScMoE architecture not only accelerates the training by 1.13x and inference by 1.19x, but also achieves a marginally higher average accuracy (43.77% vs 43.25%).\n\n**Experimental results on integrating ScMoE architecture into OLMoE model [1].**\n|             | Train Speedup | Inference Speedup | ARC-C  | BoolQ  | OBQA   | RTE    | WG     | PIQA   | AVG    |\n| ----------- | ------------- | ----------------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| OLMoE       | 1             | 1                 | 24.91% | 58.30% | 30.00% | 44.77% | 49.64% | 51.90% | 43.25% |\n| OLMoE+ScMoE | 1.13x         | 1.19x             | 25.51% | 58.10% | 30.80% | 45.49% | 49.25% | 53.48% | 43.77% |\n\nMoreover, we offer theoretical analysis upon our empirical findings to support the effectiveness of our methods. Specifically, we observe a correlation between the efficacy of ScMoE and the high similarity of intermediate representations. Given that such high similarity is commonly found in Transformer-based LLMs [3,4,5,6], we believe our ScMoE approach can be effective in larger-scale LLMs. Furthermore, our theoretical analysis of gradient propagation also demonstrates its ability to achieve consistent training while preserving model quality.\n\n\n\nAdditionally, we conduct simulations to validate the effectiveness of speedup in larger-scale scenarios (larger model and larger number of GPUs). Our simulations employ the ASTRA-SIM simulator [7], which models an A800 cluster in alignment with the real-world HPC cluster utilized for evaluations in Section 4. Specifically, the FP16 computational capability is configured to 312 TFLOPS, the intra-node NVLink bandwidth is set at 400 GB/s, and the inter-node network employs a 200Gb/s (25GB/s Bandwidth) InfiniBand connection.\n\nWe conduct simulations of the forward pass for LLaMA2-MoE, maintaining similar attention configurations to experiments in Table 2. Since the load balance of MoE significantly influences its efficiency and is determined by various inputs, we simulate a fully balanced distribution of input tokens to eliminate any random interference in this aspect.\n\nIn the simulation, we vary the configurations for the number of GPUs, the number of active and total experts, and expert intermediate size. As illustrated in the following table, while different configurations lead to varying communication and computation durations, our overlapping strategies consistently prove to be effective.\n\n**Simulation of larger-scale model training.**\n|No.|GPU Num|Act./Total Experts|Expert Intermediate Size|Communication Time (us)|Computation Time (us)|Overlap Speedup|\n|-|-|-|-|-|-|-|\n|1| 128| top-16 / 128| 1024| 7056| 2169| 1.31x |\n|2| 64| top-8 / 64| 2048| 6345| 2968| 1.47x|\n|3| 32| top-4 / 32| 4096| 5994 | 4578| 1.76x|\n|4| 16| top-2 / 16| 8192 | 5832| 7805| 1.75x|\n|5| 128| top-16 / 128| 8192| 7056| 7819| 1.90x|\n|6| 64| top-8 / 64|8192| 6345| 7811| 1.81x |\n|7| 32| top-4 / 32|8192| 5994| 7807 | 1.77x |\n|8| 16| top-2 / 16| 8192 | 5832| 7805| 1.75x|\n|9| 16| top-8 / 128| 8192| 23229| 20725| 1.89x|\n\n\n\n**Reference**\n\n[1] Muennighoff, Niklas, et al. \"Olmoe: Open mixture-of-experts language models.\" *arXiv preprint arXiv:2409.02060* (2024).\n\n[2] Lambert, Nathan, et al. \"Tulu 3: Pushing frontiers in open language model post-training.\" arXiv preprint arXiv:2411.15124 (2024).\n\n[3] Sun, Qi, et al. \"Transformer layers as painters.\" arXiv preprint arXiv:2407.09298 (2024).\n\n[4] He, Shwai, et al. \"What matters in transformers? not all attention is needed.\" arXiv preprint arXiv:2406.15786 (2024).\n\n[5] Men, Xin, et al. \"Shortgpt: Layers in large language models are more redundant than you expect.\" *arXiv preprint arXiv:2403.03853* (2024).\n\n[6] Gromov, Andrey, et al. \"The unreasonable ineffectiveness of the deeper layers.\" *arXiv preprint arXiv:2403.17887* (2024).\n\n[7] Rashidi, Saeed, et al. \"Astra-sim: Enabling sw/hw co-design exploration for distributed dl training platforms.\" ISPASS. IEEE (2020).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "I think the accuracy on benchmark datasets are a bit low to provide meaningful comparisons. It'd be better to train bigger models or longer.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The authors propose a new MoE architecture which may be insightful to readers.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n- The idea of overlapping communication and computation in MoE training is novel and interesting. This could be a useful technique for practitioners.\n\nWeakness:\n- It would be great if the idea can be validated at larger model scale and longer training runs. Currently the accuracy on benchmarks are a bit low.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhxQ2PsSne", "reviewer": "Reviewer_f6vs", "summary": "This paper proposes Shortcut-connected MoE (ScMoE) to reduce the All-to-All communication bottleneck in expert parallelism of MoE model training. Traditional MoE models suffer from high All-to-All communication costs due to dependencies between computation and communication. ScMoE solves this by using a shortcut connection that allows the previous layer’s representations to be processed in parallel with the current layer. It replaces top-2 gating with a shared expert for the current layer and a top-1 expert for the previous layer, enabling full overlap between computation and communication. The authors also propose an adaptive overlapping scheduling strategy, achieving 1.49x faster training and 1.82x faster inference while keeping model accuracy comparable or slightly better than standard top-2 MoE. Experiments demonstrate that ScMoE maintains strong performance while improving efficiency. The paper also provides theoretical analysis proving stable gradient propagation. Overall, ScMoE co-designs algorithm and infrastructure, and is a novel and effective method for accelerating MoE training.\n\n---\n\n### update after rebuttal\n\nAfter reading the rebuttal, my major concerns have been addressed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. Concerns regarding the model structure may be outdated, particularly in relation to the compatibility of ScMoE with the mainstream MoE architecture, where MoE layers are arranged sequentially rather than interspersed with MLP layers.\n\nWe would like to clarify that we have already conducted experiments on the current mainstream MoE structure of placing MoE module in every Transformer block, as presented by the LLaMA2-MoE experiments in Section 4.2.2, Table 2 and Table 7. Specifically, our LLaMA2-MoE experiments utilize ScMoE (Pos-1) and incorporates the MoE module in every Transformer block, which overlaps the communication and computation within the each Transformer block.\n\nAdditionally, we conduct experiments on OLMoE model, an advanced MoE model that integrates an MoE module at each layer. As shown in **the table \"Experimental results on integrating ScMoE architecture into the OLMoE Model\" in our response to Reviewer GE4M**, ScMoE architecture not only achieves acceleration, but also achieves a marginally higher average accuracy.\n\n> 2. Concerns regarding stronger baselines.\n\nWe select a recently proposed All-to-All optimization technique, \"MoE Shared Expert Overlap,\" which overlaps All-to-All communication with the computation of shared experts, as a stronger baseline for comparison.\n\nAs shown in the following table, ScMoE achieves acceleration compared to \"MoE Shared Expert Overlap,\" facilitated by breaking dependencies for improved overlap.\n\n|SwinV2-MoE|Train Speedup|Inference Speedup|\n|-|-|-|\n|Standard Top-2|0.76x|0.68x|\n|Shared-Expert|0.94x|0.92x|\n|MoE Shared Expert Overlap|1|1|\n|ScMoE|1.14x|1.25x|\n\n> 3. Concerns regarding larger multi-node cluster analysis. Does a larger number of experts negatively impact ScMoE’s effectiveness?\n\nTo demonstrate scalability in large-scale clusters, we conducted simulations across different number of GPUs, from 16 to 128, as shown in **the table \"Simulation of larger-scale model training\" in our response to Reviewer GE4M**. Despite differences in communication and computation durations across configurations, ScMoE consistently demonstrates effectiveness. Notably, in entries No.5 to No.8, an increased number of experts does not negatively affect the effectiveness of ScMoE.\n\n> 4. Incremental improvement: ScMoE mainly extends shared-expert MoE with shortcut connections.\n\nOur work focuses on integrating shortcut connections into the MoE architecture to enhance efficiency, which does not rely on the shared-expert MoE. As discussed in Appendix A.2, our DGMoE, which incorporates shortcut connection without shared expert, also offers improvements over the top-2 MoE. ScMoE is presented as the most efficient architecture to facilitate our proposed shortcut connections.\n\nFurthermore, ScMoE functions alongside both the standard MoE and shared-expert MoE architectures. When designing models, practitioners have the option to choose from these three architectures.\n\n> 5. Are GPT-2/LLaMA2 MoE models trained from scratch or fine-tuned from a dense model?\n\nThe models are trained from scratch. We introduce the ScMoE architecture as a viable option for pre-training a MoE model from scratch, enhancing the efficiency of its pre-training, fine-tuning, and inference.\n\n> 6. Are the experiments conducted on FP32 precision of BF16 precision?\n\nBF16 precision. \n\n> 7. Would it be possible to quantify and report exact percentages of communication hidden by computation in Table 1 or 2?\n\nThe exact percentages for Table 1 are directly reflected in Figure 7 (1), which hides 70% communication.\n\n> 8. What happens when communication is much slower than computation? Can ScMoE still provide significant speedup?\n\nAs shown by the simulation results of No. 1 in Question 3, ScMoE achieves a significant speedup of 1.31x, even when the communication time is three times longer than the computation duration.\n\n> 9. Do you observe any convergence differences between ScMoE and top-2 MoE during training? If so, does ScMoE require different hyperparameters (learning rate, warm-up schedule, etc.)?\n\nWe observed that the loss curve of ScMoE (3.2629 at the final point) demonstrates faster convergence and a lower final loss compared to the standard top-2 MoE (3.3157 at the final point). To ensure a fair comparison in our experiments, ScMoE demonstrates comparable accuracy under identical hyperparameters.\n\n> 10. Have you tested transfer learning performance? \n\nWe conduct additional fine-tuning experiments on the trained OLMoE models discussed in Question 1, using 5,000 samples from Alpaca dataset. The results shows that ScMoE still achieves a higher average accuracy (44.88% vs 44.07%).\n\n> 11. Is ScMoE scalable to larger MoE models (e.g., 13B, 30B, 70B, and even DeepSeek-V3 671B)? \n\nDue to our limited devices, we face challenges in conducting experiments on large-scale MoE models. Nevertheless, we offer theoretical analysis to support the effectiveness of our methods, as discussed in Section 5 and Appendix A.1.", "claims_and_evidence": "Most of the claims are well-supported by experiments and analysis.\n\n1. Decoupling and overlapping communication and computation via shortcut connection: The shortcut connection proposed by this paper does allow the overlapping between computation and communication, as demonstrated in Figure 5 and Figure 6. Experiments also confirm that ScMoE overlaps up to 100% of communication time, reducing the impact of All-to-All bottlenecks.\n2. Maintaining Model Quality: The authors claim that the shortcut connection maintains the model quality. This is supported by the experiments, where ScMoE achieves 79.3% Top-1 accuracy on ImageNet, similar to top-2 MoE, and performs well in NLP tasks.\n3. Generalization Capability: The experiments shows that ScMoE improves performance on different hardwares (A30-PCIe with high communication ratio, and A800-NVLink with low communication ratio), on different workloads (vision tasks, language tasks).\n\n---\n\nHowever, I still have the following concerns:\n1. The author states that the standard MoE model structure consists of Block-MLP and Block-MoE, as shown in Figure 2, which aligns with the design in DeepSpeed-MoE. However, I am afraid such a structure is out-dated, and in most mainstream MoE models (e.g., DeepSeekMoE, DeepSeek-V3), MoE layers are predominantly arranged sequentially rather than interspersed with MLP layers. I am curious whether ScMoE is compatible with this mainstream MoE architecture.", "methods_and_evaluation_criteria": "The proposed ScMoE architecture and adaptive scheduling strategy effectively target the MoE communication bottleneck. The evaluation criteria makes sence, covering:\n1. Benchmarks: Vision tasks (ImageNet) and NLP tasks (GPT-2/LLaMA2 MoE, with model size up to 7B).\n2. Metrics: Model quality and accuracy, training efficiency, inference efficiency.\n3. Baselines: Standard top-2 MoE (DeepSpeed MoE), Shared-expert MoE, and Standard top-1 MoE for comparison.\n4. Hardwards: A30-PCIe with high communication ratio, and A800-NVLink with low communication ratio. For A800, 8 GPUs within 1 node and 16 GPUs across 2 nodes are evaluated.", "theoretical_claims": "The appendix provides a theoretical analysis of gradient propagation, showing that ScMoE does not affect model convergence. The authors prove that:\n1. The shortcut connection preserves gradient flow between layers.\n2. The shared expert and expert gating maintain stable optimization dynamics.", "experimental_designs_or_analyses": "The experiments are well-controlled:\n1. Diverse hardware setups: 8×A30 PCIe (high communication overhead) and 8×A800 and 16xA800 NVLink (low communication overhead).\n2. Multiple Workloads: SwinV2-MoE-S (vision), GPT-2/LLaMA2-MoE (NLP).\n3. Ablation studies: Different shortcut placements (Pos-1/2/3) to find the best trade-off between speed and accuracy.\n\n---\n\nHowever, I suppose the following evaluation will strengthen the analysis:\n1. Evaluation on Mainstream MoE Structure: The workloads used in ScMoE is based on DeepSpeed-MoE, where MoE layers are interspersed with MLP layers. However, I think the current mainstream MoE structure (where MoE layers are predominantly arranged sequentially) should also be evaluated, e.g., DeepSeekMoE.\n2. Stronger Baselines: It would be valuable to compare ScMoE with other advanced All-to-All optimization techniques, such as hierarchical All-to-All and pipelining. Additionally, DeepSeek recently open-sourced DeepEP, a high-performance expert parallelism library. While I understand that ScMoE is either concurrent with or an earlier work relative to DeepEP, including an analysis of their differences and potential complementarities would be beneficial.\n3. Larger Models: Since ScMoE modifies the model structure, it may impact model convergence and overall quality. While the authors provide empirical results on models up to 7B and offer theoretical analysis, additional experiments on larger models (e.g., 13B, 30B) would strengthen the evaluation. \n4. Larger Multi-node Cluster Analysis: I am concerned of the performance on larger multi-mode clusters. I understand authors may not be able to obtain larger scale multi-mode cluster (e.g., 64 A800 GPUs). However, it would be beneficial if authors provide analysis or simulated evaluation of the theoretical performance of ScMoE on larger clusters.", "relation_to_broader_scientific_literature": "ScMoE builds upon existing MoE parallelism research, including:\n1. Expert Parallelism (Lepikhin et al., 2021), which introduces All-to-All communication for MoE.\n2. Shared Expert MoE (Rajbhandari et al., 2022), which uses a fixed expert for every layer, reducing All-to-All communication.\n3. Pre-gated MoE, which is optimized MoE inference by preselecting experts.\n\nScMoE combines shared experts with shortcut connections to further reduce communication overhead, making it a novel extension of these works.", "essential_references_not_discussed": "Here are several related works:\n- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity [JMLR’22], which is the first to use top-1 gating for efficiency.\n- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models [Arxiv’24], which is one of the mainstream MoE model structures.", "other_strengths_and_weaknesses": "Strengths:\n1. Significant speedup without accuracy loss.\n2. Well-structured experiments, with multiple baselines and ablation studies.\n3. Good clarity and organization, making complex ideas easy to understand.\n\nWeaknesses:\n1. Incremental improvement: ScMoE mainly extends shared-expert MoE with shortcut connections.\n2. Model structure may be out-dated: DeepSpeed-MoE, where MoE layers are interspersed with MLP layers, is not the mainstream MoE structure currently.\n3. Lack of evaluation on larger-scale models and larger-scale multi-node cluster.", "other_comments_or_suggestions": "See the questions for authors.", "questions_for_authors": "Experiment Related:\n1. Are GPT-2/LLaMA2 MoE models trained from scratch or fine-tuned from a dense model?\n2. The experiments test single-node (8 GPUs) and two-node (16 GPUs) training. How does ScMoE scale to hundreds or thousands of GPUs? More analysis of ScMoE on multi-mode clusters would be helpful.\n3. Are the experiments conducted on FP32 precision of BF16 precision?\n\n---\n\nOverlap Related:\n1. Figure 6 provides a visual timeline comparison of different MoE architectures. Would it be possible to quantify and report exact percentages of communication hidden by computation in Table 1 or 2?\n2. If we increase the number of experts per MoE layer, how does the overlap ratio change? Does a larger number of experts negatively impact ScMoE’s effectiveness?\n3. What happens when communication is much slower than computation? Can ScMoE still provide significant speedup? \n\n---\n\nModel Quality Related:\n1. Do you observe any convergence differences between ScMoE and top-2 MoE during training? If so, does ScMoE require different hyperparameters (learning rate, warm-up schedule, etc.)?\n2. Have you tested transfer learning performance? For example, if a vision model is pre-trained with ScMoE and fine-tuned on a different task, does the shortcut mechanism still provide benefits?\n3. Is ScMoE scalable to larger MoE models (e.g., 13B, 30B, 70B, and even DeepSeek-V3 671B)? I am concerned about the convergence and model quality.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyP2APboly", "reviewer": "Reviewer_rMhT", "summary": "This paper presents ScMoE to enhance the computational efficiency of Mixture-of-Experts (MoE) models. By incorporating a shortcut connection that integrates information from the preceding layer with the current layer's computations, ScMoE introduces a concurrent processing mechanism which allows for overlapping communication with computation through the parallel execution of two independent streams: one processing the current layer's input and another propagating residual from the previous layer. As a result, ScMoE significantly accelerates both the training and inference phases by minimizing communication latency while maintaining or even enhancing model quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The claim that model quality is always maintained or improved is only shown on small models. Sometimes the accuracy improvements are very small, and the results might not work for larger models (like OLMoE) or different tasks. The experiments are limited to only small models, which may limit the generalizability of the results to other models or tasks. The authors have only conducted experiments on small models for image classification and language tasks, so additional experiments on larger and more powerful models may be necessary.\n\n(1) Given our constraints on available hardware, we encounter difficulties in conducting empirical experiments on large-scale, industrial-level MoE models. Nevertheless, we perform additional experiments on the OLMoE model [1] and conduct some different evaluation tasks. The detailed experimental configurations are introduced in our response to Reviewer GE4M. As shown in the following table, our proposed ScMoE architecture not only accelerates the training by 1.13x and inference by 1.19x, but also achieves a marginally higher average accuracy (43.77% vs 43.25%).\n\n|             | Train Speedup | Inference Speedup | ARC-C  | BoolQ  | OBQA   | RTE    | WG     | PIQA   | AVG    |\n| ----------- | ------------- | ----------------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| OLMoE       | 1             | 1                 | 24.91% | 58.30% | 30.00% | 44.77% | 49.64% | 51.90% | 43.25% |\n| OLMoE+ScMoE | 1.13x         | 1.19x             | 25.51% | 58.10% | 30.80% | 45.49% | 49.25% | 53.48% | 43.77% |\n\n(2) Beyond the focus on accuracy, we also simulations to validate the effectiveness of speedup in larger-scale scenarios, as shown in **the table \"Simulation of larger-scale model training\" in our response to Reviewer GE4M**. Despite differences in communication and computation durations across configurations, our proposed ScMoE consistently proves to be effective. \n\n> 2. It relies heavily on the assumption that features in neighboring layers are similar, which is often true. Yet, the supplementary materials show that this similarity is not clear in the OLMoE model. In addition, the authors mention that ScMoE-Pos2 does not perform well, which makes me worry about how well the method can work in different cases. Does the proposed method fail if the features change dramatically between neighboring layers?\n\n(1) Recent studies [2,3,4,5] have commonly observed the high similarity between features in neighboring layers as a characteristic of Transformer-based LLMs, which can be utilized to improve the efficiency of LLM operations.\n\n(2) Our supplementary materials on the OLMoE model further corroborate that the observed feature similarity is within an expected range and aligns with prevalent findings in the field. As illustrated in Figure 15, most data points near the diagonal line are yellow, indicating nearly 100% similarity. Only the output features of 1A and 1M exhibit comparatively lower similarity, yet maintain a significant 50%.\n\n(3) We would like to clarify that the statement \"ScMoE-Pos2 does not perform well\" was made in a comparative context. In fact, the loss differences among ScMoE-Pos1 (3.2615), ScMoE-Pos2 (3.2818), and ScMoE-Pos2-L0Pos1 (3.2629) are relatively minor. Moreover, the loss values for the standard top-2 MoE (3.3157) and shared-expert MoE (3.2974) are higher than those of the three ScMoE configurations. This indicates that despite variations in feature similarity across certain layers, ScMoE remains effective. Additionally, we will include a detailed comparison of these loss curves in the appendix for further clarity.\n\n(4) Our strategy for selecting shortcut-connected positions, detailed in Section 5.1.3, is designed to achieve optimal accuracy. Additionally, based on these observations of similarity, we can adapt this strategy for subsequent MoE model design by opting not to use ScMoE in the first two Transformer layers. Similarly, DeepSeek-MoE [6] chooses not to apply the MoE module in the first Transformer layer.\n\n**Reference**\n\n[1] Muennighoff, Niklas, et al. \"Olmoe: Open mixture-of-experts language models.\" *arXiv preprint arXiv:2409.02060* (2024).\n\n[2] Sun, Qi, et al. \"Transformer layers as painters.\" arXiv preprint arXiv:2407.09298 (2024).\n\n[3] He, Shwai, et al. \"What matters in transformers? not all attention is needed.\" arXiv preprint arXiv:2406.15786 (2024).\n\n[4] Men, Xin, et al. \"Shortgpt: Layers in large language models are more redundant than you expect.\" *arXiv preprint arXiv:2403.03853* (2024).\n\n[5] Gromov, Andrey, et al. \"The unreasonable ineffectiveness of the deeper layers.\" *arXiv preprint arXiv:2403.17887* (2024).\n\n[6] Dai, Damai, et al. \"Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.\" *arXiv preprint arXiv:2401.06066* (2024).", "claims_and_evidence": "Communication and computation can overlap fully depending greatly on hardware and system conditions.", "methods_and_evaluation_criteria": "The claim that model quality is always maintained or improved is only shown on small models. Sometimes the accuracy improvements are very small, and the results might not work for larger models (like OLMoE) or different tasks.", "theoretical_claims": "The theoretical claims presented in the paper are solid.", "experimental_designs_or_analyses": "The experiments are limited to only small models, which may limit the generalizability of the results to other models or tasks.", "relation_to_broader_scientific_literature": "The authors demonstrate that neighboring Transformer layers share significant feature similarity, a pattern consistent with prior research.", "essential_references_not_discussed": "The references are comprehensive and well-organized.", "other_strengths_and_weaknesses": "The proposed method is simple, which is a clear advantage. However, it relies heavily on the assumption that features in neighboring layers are similar, which is often true. Yet, the supplementary materials show that this similarity is not clear in the OLMoE model. In addition, the authors mention that ScMoE-Pos2 does not perform well, which makes me worry about how well the method can work in different cases.", "other_comments_or_suggestions": "The authors have only conducted experiments on small models for image classification and language tasks, so additional experiments on larger and more powerful models may be necessary.", "questions_for_authors": "Does the proposed method fail if the features change dramatically between neighboring layers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2JEzVn4Lt", "reviewer": "Reviewer_hvmz", "summary": "The paper proposes an new method for expert parallelism, which is a paradigm for distributed training and inference of large scale MoE models by dividing experts across multiple devices. The authors address the bottleneck of all-to-all communication between experts and present a new strategy that can overlap communication with computation. Given a model with every other layer being an MoE and each MoE having a shared expert, the shortcut-connected approach overlaps MLP and shared expert computation with the all-to-all communications of routed experts. This results in tangible speedups for MoE training and inference for GPT2-size models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. I think the analysis can be stronger if all of the the results are consolidated across different model sizes.\n\nWe have summarized and supplemented evaluations of GPT models across various sizes to better demonstrate the impact of model size as a standalone variable. The following  table presents the speedup, zero-shot perplexity, peak GPU memory reduction, and MoE block latency reduction of ScMoE in comparison to the standard top-2 MoE. The results demonstrate that our proposed ScMoE consistently achieves improvements across various model sizes.\n\n| |Model Size|Train Speedup|Inference Speedup|Perplexity (ScMoE / Standard MoE) |Peak GPU Memory Usage Reduction|MoE Block Latency Reduction|\n|-|-|-|-|-|-|-|\n|GPT2-MoE-Small| 323M|1.15x|1.22x|29.10/31.60|-53%|-41%|\n|GPT2-MoE-Medium|1.7B|1.12x|1.17x|17.62/19.18|-50%|-33%|\n|GPT2-MoE-XL|4.1B|1.12x|1.18x|16.46/17.52|-60%|-18%|\n\n> 2. How do the validation loss results in Figure 10 compare with a regular MoE as the baseline?\n\nWe observed that the loss curve of ScMoE (3.2629 at the final point) demonstrates faster convergence and a lower final loss compared to the standard top-2 MoE (3.3157 at the final point) and shared-expert MoE (3.2974 at last).\n\n> 3. How does the overall speedup and benchmark performance scale with model size? Do you expect that the ScMoE approach becomes more viable for larger scale pretraining?\n\n(1) To illustrate the effectiveness of speedup in larger-scale scenarios, we conducted additional simulations, the results of which are detailed in **the table \"Simulation of larger-scale model training\" in our response to Reviewer GE4M**. The findings suggest that the relative ratio of communication to computation duration, influenced by various training and model configurations, is crucial for impacting speedup. Optimal speedup is achieved when these durations are balanced. Despite these variations, ScMoE consistently achieves acceleration.\n\n(2) Due to our limited devices, we face challenges in conducting experiments on large-scale MoE models. Nevertheless, we offer theoretical analysis to support the performance of our methods, as discussed in Section 5 and Appendix A.1.\nAdditionally, based on the findings presented in Question 1, we anticipate that the disparity in accuracy between ScMoE and existing MoE architectures will diminish as both model size and the volume of training data increase.\n\n> 4. Can this shortcut connection approach be applied to already pretrained models? What does the performance look like in such cases? Does your approach necessarily require pretraining a model from scratch to use the shortcut connections.\n\n(1) We propose the ScMoE architecture as a viable option for the backbone of MoE models. In line with the initial experiments conducted on previously proposed MoE architectures, such as standard MoE and shared-expert MoE, we train the ScMoE model from scratch to ensure a fair comparison.\n\n(2) The ScMoE models can be constructed using pre-trained shared-expert MoE models by modifying the inputs of the MoE module. Our experiments with this approach showed a decrease in accuracy, indicating a need for further refinements in the fine-tuning process to preserve accuracy.\n\n(3) Additionally, it is possible to construct and train a ScMoE model based on a pretrained dense model, employing techniques similar to sparse upcycling, which have been applied to standard MoE models.\n\n> 5. The method seems limited to architectures that use shared experts and alternating MoE layers with typical MLPs. How do you think your results could generalize to other MoE models that either do not use shared experts, or have an MoE for every transformer layer?\n\n(1) We would like to clarify that we have already conducted experiments on the current mainstream MoE structure of placing MoE module in every Transformer block, as presented by the LLaMA2-MoE experiments in Section 4.2.2, Table 2 and Table 7. Specifically, our LLaMA2-MoE experiments utilize ScMoE (Pos-1) and incorporates the MoE module in every Transformer block, which overlaps the communication and computation within the each Transformer block.\n\n(2) Our work focuses on integrating shortcut connections into the MoE architecture to enhance efficiency, which does not rely on the shared-expert MoE. As discussed in Appendix A.2, our DGMoE, which incorporates shortcut connection without shared expert, also offers improvements over the top-2 MoE.\n\nFurthermore, ScMoE functions alongside both the standard MoE and shared-expert MoE architectures. When designing models, practitioners have the option to choose from these three architectures.\n\n(3) We conduct experiments on the OLMoE model, which originally employs standard MoE without shared experts at each layer. As shown in **the table \"Experimental results on integrating ScMoE architecture into the OLMoE model\" in response to Reviewer GE4M**, our proposed ScMoE architecture not only achieves acceleration, but also achieves a higher average accuracy.", "claims_and_evidence": "The main premise of reducing expert parallelism overhead is well motivated. The proposed method is clear and the paper shows convincing evidence that the proposed overlapping strategy is beneficial for improving throughput. However, the MoE architecture is fundamentally changed in this method. This leads to slightly decreased performance on pretraining benchmarks, and it is not clear whether this error scales for larger models.", "methods_and_evaluation_criteria": "The evaluation section is very strong and uses sensible benchmarks. The most common pretraining benchmarks are reported along with validation loss. The paper also discusses the speedup of their method, which makes sense as the goal of their method was to reduce MoE distributed communication overhead.", "theoretical_claims": "I did not check for correctness of any proofs", "experimental_designs_or_analyses": "The experimental design makes sense. The authors apply their new MoE architecture for LM pretraining on multiple models, up to 7B parameters. I think the analysis can be stronger if all of the the results are consolidated across different model sizes. For example, what is the % speedup, peak GPU memory reduction, change in validation loss, change in pretraining benchmarks as a result of model size? Currently the evaluation is sparse as the benchmarks in Table 2 are reported for two models and the latency + peak memory analysis in Figure 13 uses two different models. Because the paper is fundamentally modifying the MoE architecture it would be more convincing to see all of the evaluations for each model.", "relation_to_broader_scientific_literature": "The authors address the bottlenecks expert parallelism, which is becoming an increasingly important paradigm for training and deploying large scale MoEs. The results in this paper have the potential to speed up MoE training in a broad range of settings, which makes this work widely applicable in the field of MoE research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The method seems limited to architectures that use shared experts and alternating MoE layers with typical MLPs. While this is a commonly used setup, I think the results would be more convincing if the approach could be extended to architectures that don't follow these exact constraints.\n\nThe approach also fundamentally modifies the MoE architecture by introducing these shortcut connections. I assume this means that ScMoE cannot be applied to speed up MoE inference for off-the-shelf pretrained models, which limits its wider adoption.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How do the validation loss results in Figure 10 compare with a regular MoE as the baseline?\n2. How does the overall speedup and benchmark performance scale with model size? Do you expect that the ScMoE approach becomes more viable for larger scale pretraining?\n3. Can this shortcut connection approach be applied to already pretrained models? What does the performance look like in such cases? Does your approach necessarily require pretraining a model from scratch to use the shortcut connections\n4. How do you think your results could generalize to other MoE models that either do not use shared experts, or have an MoE for every transformer layer?\n\nI think the results of this paper are very interesting with strong impact on the field of MoE research in general. However, they currently seem limited to a specific architecture setup, and I am not convinced if this is viable for MoE training at larger scale. The responses to these questions would increase my confidence in recommending this paper for acceptance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9S1VvKx76e", "reviewer": "Reviewer_qsHP", "summary": "The execution of mixture-of-experts model contains two all-to-all communication steps on the critical path of computation. The authors of this paper propose to use the activations before the attention layer of the current block as the input for the experts in the current block, which breaks the sequential dependency between the attention and expert layers, and opens the possibility to overlap the computation and communication. Experiments on the accuracy and inference time show the effectiveness of the new design.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. It's better to include more modern designs like the ones used in deepseek v2 (more smaller experts).\n\nGiven our constraints on available hardware, we cannot conduct experiments on large-scale MoE models like DeepSeek-V2 (236B). Nonetheless, we conduct experiments on the OLMoE model (7B), a recent MoE model that features more modern designs, incorporating the fine-grained expert architecture (more smaller experts) similar to DeepSeek-V2. We train both the original OLMoE and the OLMoE+ScMoE from scratch, using 40,000 samples from the TuluV3 dataset. As demonstrated in the following table, our proposed ScMoE architecture not only accelerates the training by 1.13x and inference by 1.19x, but also achieves a marginally higher average accuracy (43.77% vs 43.25%).\n\n| |Train Speedup|Inference Speedup|ARC-C|BoolQ|OBQA|RTE|WG|PIQA|AVG|\n|-|-|-|-|-|-|-|-|-|-|\n|OLMoE|1|1|24.91%|58.30%|30.00%|44.77%|49.64%|51.90%|43.25%|\n|OLMoE+ScMoE |1.13x|1.19x|25.51%|58.10%|30.80%|45.49%|49.25%|53.48%|43.77%|\n\n\n\n> 2. My primary concern is that the skip connection design lacks novelty. Additionally, the model quality requires further justification or experiments to substantiate that it's actually effective. Existing models such as GPT-J and GPT-NeoX already parallelize attention and feedforward networks, while architectures like Snowflake Arctic [1] employ a structure where the input to the self-attention layer is directly fed to the MoE layer - closely resembling this work’s proposal. \n\n(1) GPT-J and GPT-NeoX utilize parallel Attention and Feed-Forward Layers to address the All-Reduce communication overhead inherent in Tensor Parallelism, which is notably different from the All-to-All communication overhead encountered in Expert Parallelism. Distinct from existing methods, our proposed ScMoE architecture is specifically designed to minimize the All-to-All communication overhead in Expert Parallelism.\n\n(2) Regarding Snowflake Arctic, our work was conducted independently and concurrently with it, and it stands out in several distinct ways. Snowflake Arctic is primarily an industrial product and does not include any published papers or detailed analyses of their proposed Dense-MoE Hybrid Transformer architecture. In contrast, we thoroughly examine the effectiveness of Shortcut-connected MoE architectures and analyze the phenomena and reasons behind them, offering deeper insights for the research community. Furthermore, our proposed Shortcut-connected MoE demonstrates superior generalization across different MoE model designs (adapt to various MoE placement frequencies). It bears similarity to their Dense-MoE Hybrid Transformer architecture, specifically to our proposed ScMoE (Pos-1), which is more like a subset of our study. \n\n> 3. However, current SOTA models (e.g., DeepSeek V3/R1, Llama 3.2) do not adopt this parallel structure, raising questions about whether the proposed architecture can achieve competitive accuracy/quality at scale. The experiments in this paper inadequately validate model quality, as the metrics in Table 2 fall significantly below those of publicly available models, including smaller ones. In general, altering model architecture for performance optimization but do not validate the final accuracy on large scale carry risks, as such changes may compromise final accuracy. The accuracy number of Snowflake Arctic [1] also shows that this parallel architecture might harm the model quality (when it compares with other models with similar size of parameters). While the paper is well-argued on the motivation and the system-side optimizations appear sound, I remain cautious about the final model quality of the new architecture.\n\n(1) We propose the ScMoE architecture as a viable option for the backbone of MoE models. In line with the initial experiments conducted on previously proposed MoE architectures, such as standard MoE and shared-expert MoE, we train the ScMoE model from scratch to ensure a fair comparison. When trained under identical conditions, ScMoE demonstrates superior efficiency while achieving comparable accuracy, compared to both the standard MoE and shared-expert MoE methods.\n\n(2) Due to our limited devices, we are unable to train with pre-training datasets comprising 1 trillion tokens, as is common with existing industrial-level models. Consequently, our experimental models exhibit relatively lower accuracy, as we only train with datasets that are 1/1000th the size. Nevertheless, we offer theoretical analysis upon our empirical results to support the effectiveness of our methods, as discussed in Section 5 and Appendix A.1.\n\n(3) According to the provided document on Snowflake Arctic, this parallel architecture offers significant value for practical deployments, rather than solely impacting model quality negatively. The document states that \"Arctic is more capable than other open source models trained with a similar compute budget.\"", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes, the experimental design is good in general. But it's better to include more modern designs like the ones used in deepseek v2 (more smaller experts).", "relation_to_broader_scientific_literature": "A new design to MoE architecture.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**  \n1. The paper addresses an important problem: reducing communication overhead in the serving of MoE models.  \n2. By modifying the architecture to break the sequential dependency between attention and MoE layers, the authors enable overlapping computation and communication, effectively hiding communication overhead.  \n3. Experiments demonstrate that the proposed computation-communication overlap strategy in the new architecture is effective.  \n4. The paper is well-written and easy to follow.  \n\n**Weaknesses**  \nMy primary concern is that the skip connection design lacks novelty. Additionally, the model quality requires further justification or experiments to substantiate that it's actually effective. Existing models such as GPT-J and GPT-NeoX already parallelize attention and feedforward networks, while architectures like Snowflake Arctic [1] employ a structure where the input to the self-attention layer is directly fed to the MoE layer - closely resembling this work’s proposal. However, current SOTA models (e.g., DeepSeek V3/R1, Llama 3.2) do not adopt this parallel structure, raising questions about whether the proposed architecture can achieve competitive accuracy/quality at scale. The experiments in this paper inadequately validate model quality, as the metrics in Table 2 fall significantly below those of publicly available models, including smaller ones.  \n\nIn general, altering model architecture for performance optimization but do not validate the final accuracy on large scale carry risks, as such changes may compromise final accuracy. The accuracy number of Snowflake Arctic [1] also shows that this parallel architecture might harm the model quality (when it compares with other models with similar size of parameters). While the paper is well-argued on the motivation and the system-side optimizations appear sound, I remain cautious about the final model quality of the new architecture.\n\n[1] https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-models-snowflake/", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: None of the reviewers showed strong opinions either way, but all the reviews were at least borderline positive.

The method is intuitive, and the latency improvements make sense to me. The issue is in the empirical validation that the modification the authors are suggesting doesn't negatively impact the model. In Table 1, all of the proposed methods perform worse than the shared expert baseline. In Table 2, we see an Avg improvement across LM benchmarks of 0.01, which is well within the standard error, and the models are clearly undertrained (just 1B tokens of training data) because they are scoring near the noise floor for the benchmarks. This doesn't preclude acceptance because I have to imagine that the boost in training speed should translate into sufficiently more tokens that things are fine anyways.
Decision: Accept (poster)