NeurIPS 2025

FlashMoE: Fast Distributed MoE in a Single Kernel
Osayamen Jonathan Aimuyo
Cornell University
oja7@cornell.edu
Byungsoo Oh
Cornell University
bo239@cornell.edu
Rachee Singh
Cornell University
rs2293@cornell.edu
Abstract
The computational sparsity of Mixture-of-Experts (MoE) models enables sub-
linear growth in compute cost as model size increases, thus offering a scalable
path to training massive neural networks. However, existing implementations
suffer from low GPU utilization, significant latency overhead, and a fundamen-
tal inability to leverage task locality, primarily due to CPU-managed scheduling,
host-initiated communication, and frequent kernel launches. To overcome these
limitations, we develop FlashMoE, a fully GPU-resident MoE operator that fuses
expert computation and inter-GPU communication into a single persistent GPU ker-
nel. FlashMoE enables fine-grained pipelining of dispatch, compute, and combine
phases, eliminating launch overheads and reducing idle gaps. Unlike existing work,
FlashMoE obviates bulk-synchronous collectives for one-sided, device-initiated,
inter-GPU (R)DMA transfers, thus unlocking payload efficiency, where we elim-
inate bloated or redundant network payloads in sparsely activated layers. When
evaluated on an 8-H100 GPU node with MoE models having up to 128 experts
and 16K token sequences, FlashMoE achieves up to 9× higher GPU utilization,
6× lower latency, 5.7× higher throughput, and 4× better overlap efficiency com-
pared to state-of-the-art baselines—despite using FP32 while baselines use FP16.
FlashMoE shows that principled GPU kernel-hardware co-design is key to unlock-
ing the performance ceiling of large-scale distributed ML. We provide code at
https://github.com/osayamenja/FlashMoE.
FlashMoE
Comet
FasterMoE
Megatron
DeepEP
0
20
40
60
80
100
Average SM Util (%)
93.17%
42.31%
9.67%
59.11%
13.55%
Forward Pass | E = 64 | k = 2 | 2 A100s | 
 is better
(a) GPU SM Utilization
4K
8K
16K
Number of Tokens
0
10
20
30
40
Runtime (ms)...