ICML 2025

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial
Runtime Reconfiguration
HamidReza Imani 1 Jiaxin Peng 1 Peiman Mohseni 2 Abdolah Amirany 1 Tarek El-Ghazawi 1
https://github.com/hamid-74/Multi-MoE
Abstract
The deployment of mixture-of-experts (MoE)
large language models (LLMs) presents signif-
icant challenges due to their high memory de-
mands.
These challenges become even more
pronounced in multi-tenant environments, where
shared resources must accommodate multiple
models, limiting the effectiveness of conventional
virtualization techniques. This paper addresses
the problem of efficiently serving multiple fine-
tuned MoE-LLMs on a single GPU. We propose a
serving system that employs similarity-based ex-
pert consolidation to reduce the overall memory
footprint by sharing similar experts across mod-
els. To ensure output quality, we introduce run-
time partial reconfiguration, dynamically replac-
ing non-expert layers when processing requests
from different models. As a result, our approach
achieves competitive output quality while main-
taining throughput comparable to serving a sin-
gle model, and incurs only a negligible increase
in time-to-first-token (TTFT). Experiments on a
server with a single NVIDIA A100 GPU (80GB)
using Mixtral-8x7B models demonstrate an 85%
average reduction in turnaround time compared to
NVIDIA’s multi-instance GPU (MIG). Further-
more, experiments on Google’s Switch Trans-
former Base-8 model with up to four variants
demonstrate the scalability and resilience of our
approach in maintaining output quality compared
to other model merging baselines, highlighting its
effectiveness.
1Department of Electrical and Computer Engineering, The
George Washington University 2Computer Science and Engineer-
ing Department, Texas A&M University. Correspondence to:
HamidReza Imani <hamidreza@gwu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the auth...