HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient
  MoE Inference
Authors: name: Shuzhang Zhong, name: Yanfan Sun, name: Ling Liang, name: Runsheng Wang, name: Ru Huang, name: Meng Li

The Mixture of Experts (MoE) architecture has demonstrated significant
advantages as it enables to increase the model capacity without a proportional
increase in computation. However, the large MoE model size still introduces
substantial memory demands, which usually requires expert offloading on
resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU
inference has been proposed to leverage CPU computation to reduce expert
loading overhead but faces major challenges: on one hand, the expert activation
patterns of MoE models are highly unstable, rendering the fixed mapping
strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU
schedule for MoE is inherently complex due to the diverse expert sizes,
structures, uneven workload distribution, etc. To address these challenges, in
this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that
improves resource utilization through a novel CPU-GPU scheduling and cache
management system. HybriMoE introduces (i) a dynamic intra-layer scheduling
strategy to balance workloads across CPU and GPU, (ii) an impact-driven
inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to
mitigate expert activation instability. We implement HybriMoE on top of the
kTransformers framework and evaluate it on three widely used MoE-based LLMs.
Experimental results demonstrate that HybriMoE achieves an average speedup of
1.33times in the prefill stage and 1.70times in the decode stage compared
to state-of-the-art hybrid MoE inference framework. Our code is available at:
https://github.com/PKU-SEC-Lab/HybriMoE.