QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Authors: HamidReza Imani
Jiaxin Peng
Peiman Mohseni
Abdolah Amirany
Tarek El-Ghazawi
Venue: ICML 2025 poster (2025)
Keywords: Mixture-of-Experts
Large Language Models
Virtualization
Multi-Tenant Environments
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: The deployment of mixture-of-experts (MoE) large language models (LLMs) presents significant challenges due to their high memory demands. These challenges become even more pronounced in multi-tenant environments, where shared resources must accommodate multiple models, limiting the effectiveness of conventional virtualization techniques. This paper addresses the problem of efficiently serving multiple fine-tuned MoE-LLMs on a single GPU. We propose a serving system that employs \textit{similarity-based expert consolidation} to reduce the overall memory footprint by sharing similar experts across models. To ensure output quality, we introduce \textit{runtime partial reconfiguration}, dynamically replacing non-expert layers when processing requests from different models. As a result, our approach achieves competitive output quality while maintaining throughput comparable to serving a single model, and incurs only a negligible increase in time-to-first-token (TTFT). Experiments on a server with a single NVIDIA A100 GPU (80GB) using Mixtral-8x7B models demonstrate an 85\% average reduction in turnaround time compared to NVIDIA's multi-instance GPU (MIG). Furthermore, experiments on Google's Switch Transformer Base-8 model with up to four variants demonstrate the scalability and resilience of our approach in maintaining output quality compared to other model merging baselines, highlighting its effectiveness.

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial
Runtime Reconfiguration
RezidReza Imani
Jiaxin Peng
Peiman Mohseni
Abdolah Amirany'
Tarek El-Ghazawi
Abstract
1. Introduction
Incorporating the Mixture-of- (MoE) concept (Ja- cobs et al., 1991; Jordan & Jacobs, 1994) into transformer- based large language models (LLMs) (He et al., 2024) has enabled an expansion in model parameters, improving the quality of generated outputs across various language tasks (Shazeer al., 2017; 2018; Fedus et al., 2022; Du al., 2022; Kim et al., 2021; Zuo et al., 2021; Lin et al., 2021a; Rajbhandari et al., 2022). Rather than using single block (e.g., a feed-forward layer), MoE-based models employ a gating system and multiple parallel instances of the block (known as experts) with identical structures. During the for- ward pass calculation, based on the current input, the gating system assigns weights each expert, and one or more top experts will be selected to produce the block's 's output. This allows the model to scale in size while imposing negligible computation overhead (in the case of selecting only one expert).
The deployment of mixture-of-experts (MoE) large language models (LLMs) presents signif- icant challenges due to their high memory de- mands. These challenges become even pronounced in multi-tenan environments, where shared resources must accommodate multiple models, limiting the effectiveness of conventional virtualization techniques. This paper addresses the problem of efficiently serving multiple fine- tuned MoF-LLLSs on a single GPU. We propose a serving system that employs similarity- pert consolidation to reduce the overall memory footprint by sharing similar experts across mod- els. To ensure output quality, we introduce run- time partial reconfiguration, dynamically replac- ing non-expert layers when processing requests from different models. As result, our approach achieves competitive output quality while main- taining throughput comparable to serving sin- gle model, and incurs only negligible increase in time-to-first first-token (TTFT). Experiments on a server with a single NVIDIA A A100 GPU (80GB) using Mixtral-8x2 models demonstrate an 85% average reduction in turnaround time compared to NVIDIA's multi-instance GPU (MIG). Further- more, experiments on Google's Switch Trans- former Base-8 model with up to four variants demonstrate the scalability and resilience of our approach in maintaining output quality compared to other model merging baselines, highlighting its effecttiveness.
ex-
However, such scaling comes with its own drawbacks. MoE LLMs have significantly higher memory footprints and do not fit in most of the recently developed GPU memories. For example, Google's Switch Transformer (Fedus et al., 2022) model has a total of 1.6 trillion parameters which occupies 3.1 TB of memory (16 bits for each parameter). Therefore, for deployment in constrained environments, the model parameters should be partitioned between CPU and GPU memories. Accordingly, in the calculation of forward pass, repetitive host-to-device copy operations will be re- quired which stall the inference pipeline and limit the overall throughput of the system.
At the same time, an increasing number of private entities are adopting customized MoE- based LLMs to meet their specific needs and ensure the required output quality. These LLMs are typically fine-tuned versions of general-purpose model, trained on specific datasets (Jiang al., 2024) which should be deployed in local settings for efficiency and cus- tomization purposes. However, most of these entities lack access to sufficient GPU memory to accommodate large models, as they typically have only a limited number of GPUs. This issue becomes even more challenging when
'Department of Electrical and Computer Engineering, The George Washington University *Computer Science and Engineer- ing Department, Texas A&M University. Correspondence to: HamidReza Imani <hamidreza@e @
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
multiple entities attempt to perform inference with their cus- tomized models simultaneously in the described constrained
lower number of parameters compared experts, the contribution of non-expert layers is paramount in the forward pass calculation. Therefore, to serve requests for certain model, we instantly reconfigure the system by only swapping the non-expert layers loaded on the GPU with non-expert layers of the requested model.
environment and must share resources.
Current research efforts to improve MoE inference effi- ciency and performance focus on reducing the overall mem- ory footprint (Lin al.. 2023; Xiao et al., 2023; Huang al., 2024; Eliseev Mazur, 2023; Dettmers Zettle- moyer, 2023) and host-to-device communication overhead (Kamahori al., 2024; Eliseev & Mazur, 2023; Xue al., 2024; Suvizi et al., 2024) of a single MoE model. Quanti- zation and mixed precision approaches directly reduce the memory footprint, which allow the server to fit more pa- rameters in GPU memory. In another direction, previously proposed expert caching mechanisms identify the most fre- quently used experts and prioritize them for loading into GPU memory. As result, these approaches reduce the com- munication overhead by increasing the probability of future required experts being available on the GPU. However, these approaches face considerable performance degradation in the case of a system that supports the inference of multiple MoEs.
We implement and deploy our serving approach system equipped with a single NVIDIA A100 GPU, To evaluate the impact of our method on generated output quality, we benchmark two model families- 8x7B and Google Switch Transformer Base-8, on tasks such as language generation and instruction following. We use Poisson distribution with varying arrival rates assess the quality of service (QoS) provided by our approach, and compare it against NVIDIA MIG baseline.
2. System Overview and Problem Statement
Hard Disk 1782 GB
Conventional virtualization and sharing techniques, such as space and time sharing (El-Araby al., 2009; Li et al., 2011; Huang et al., 2010), are also inefficient for this class of problem. Space-sharing approaches, like NVIDIA Multi- Instance GPU (MIG) (Choquette et al., 2021), divide the available GPU resources among the requesting users, which amplifies the communication overhead. In contrast, in time sharing approaches, requests for different models cause the currently loaded model to be offloaded and another model to be uploaded onto the GPU, process that can take up to two minutes.
M0000
Host Main Memory 251 GB
CPU
PCIe Gen 4
NVIDIA A100 80 GB DRAM
GPU Instance
GPU Instance
GPU
GPU
GPU
GPU
GPU
In this paper, to optimize the performance and availability of a multi-model inference system, we propose a serving system which is expected to achieve a throughput compara- ble serving a single model. This feature will be supported at the cost of slight increase in time-to tone-to-first--ie first-token (TTFT) and decrease in generated output quality. Although this paper focuses specifically on fine-tuned LLMs with iden- tical architectures and text generation tasks, the proposed technique is applicable to MoEs with different sizes and number of parameters. summary, we make the following contributions:
DRAML 12 Slaoo
DRAM, L2
12 Slico
L22 Sliees
L2 Slace
Slaee
Slios
Sluee
Figure Diagram of inference system with a single NVIDIA A100 GPU.
The components of single-GPU inference system are de- picted in Figure 1. this single-GPU system, the models are originally stored the hard disk. For every ML serving application, the parameters must first be loaded into the host's 's main memory. If the inference is to be performed on the accelerator, the model parameters will be transferred from the host's main memory to the GPU's memory. After the load is completed, the model will be ready for inference, and for every new request, the inputs are copied to the GPU, processed, and the results are written back to the CPU's
Expert Loading on GPU via Similarity- -Based Consol- idation: We propose to find middle point between different fine-tuned models being served. Since experts are the major contributor to the model's 's large size, we propose to find the most similar experts and load them common experts in round-robin approach reduce the overall memory footprint.
Runtime Partial Reconfiguration: Despite having
2

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Expert Distance
2-
4.6
Export
4.4
4.2
6-
4.0
38
10
11
12
13
15
16
17 Layer
18
19
20
21
22
23
24
25
26
27
26
29
30
31
Figure 2. Expert-to- ex-eet distance between Mixtral base and Instruct model: Despite being tinettune for different tasks, experts in the same positions exhibit similarity.
main memory.
2 streaming multiprocessors (SMs). NVIDIA MIG (Cho- quette et al., 2021) allows us to make different partitions using different numbers of GPU slices. For example, in Fig- ure 1, the system partitioned into two equal GPU instances which can simultaneously and independently support infer- ence for models.
However, when handling an MoE LLM, the model does not completely fit in the accelerator's memory. Originally proposed in (Eliseev & Mazur, 2023), offloading techniques load the non-expert parameters of the model GPU initially and fill the remaining space with as many experts possible. When performing inference for a sparse MoE model, if the required expert is not available on GPU's memory. it must be transferred from the CPU's memory through the PCIe link, which stalls the inference process and slows the token generation speed.
3. Approach
3.1. Parameter Layout on GPU: A Consolidated Middle Ground
In the described environment, the parameters of the MoE model can be grouped into three categories: on-device non- expert weights, on-device expert weights, and off-device expert weights. The on-device expert weights constitute the majority of the parameters loaded into GPU memory. Accordingly, in this section, present the algorithm that generates unified layout for the on-device expert weights, which is used during the initial model load.
In this paper, consider single-GPU inference system that has support two more MoE LLMs. For such problem, accelerator resources must be shared and the conventional virtualization approaches are considered as follows:
Time Sharing. For this approach, there will queue that handles requests coming from the users. the next request's 's requested model matches with the currently loaded model, the inference will be performed. However, in the case of miss-match, the entire model must be offloaded to the CPU's main memory, and the requested model must be uploaded onto the GPU, Due to the large size the discussed models and based on our experiments, each model swap can take up to minutes, which introduces considerable delay in the inference process.
As shown in Figure 2, the expert-to-expert distances be- tween a fine-tuned Instruct model and a base model are not equal. this context, we flatten the weights of an expert block into 1D tensor and define the distance the L2 distance between experts from different models. As de- picted, the distance between experts generally increases in the deeper layers, which aligns with the experiments con- ducted (Shen et al., 2024). While the distances between same expert positions are the range of 3.6 to 4.9, expert- to-expert distances with different layer and expert numbers can vary from 150 250. This observation leads propose using the experts interchangeably.
Space Sharing. For space-sharing, the resources the GPU should be split among the models we are trying to serve. The A100 NVIDIA GPU consists seven GPU slices and starting with NVIDIA Ampere architecture each slice is capable of operating independently. Each slice comes with its own memory (DRAM and L2 slices), GPU Processing Cluster (GPC), and host-to-device link bandwidth. On an A100, each GPC consists of Texture Processing Clusters (TPCs), and each TPC is comprised of
Therefore, to find the most similar experts and generate unified layout, we associate a degree of similarity to each expert coordinate specified by layer and expert numbers. In the case of having only two models for serving, the degree of similarity for each location is the expert-to-expee dis-
3

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Algorithm Initial Expert Loader
Algorithm 2 Inference Orchestrator
Input:
List of models M, expert capacity C
Input:
prompt prompt, requested
model
Iterate over layers and experts
targetModel
for (i i.) (0,... ,I, {1, F3)
(3)d do
if ta PtaaggeModel is not loadedModel then
# Iterate over all model pairs
Load targetModel non-experts GPU end if
for (i,j) E M M do
distance [L] [iB] + distance
# Tokenize the input prompt and embed it into activations repeat
|
M [LL |F]
end for end for
act = tokenize and embed pr prompt.
Process the token through all transformer layers for iT. L. do
Initialize counter for expert assignments
Initialize count <-
Apply the multi-headed attention act
# Assign experts based sorted similarity
for (iL, iE') in sorted ( (dist ance) while count C do
layers [iL] www.tieeaaatttettmm.. ion (act)
Determine which expert to use
count + count + idr t- count mod [M]
(iL, iE) layers [iZ] . .gate (act)
if map [iL] [iE] is not on GPU then
mapliL]lib] + M[idr]
Load expert from CPU not on GPU
Load _M eijdrl/exxeerttilliaa to GPU end for
Load targetModel exp.rttteeeeeeettteeeoe expert [iL] [iE] to
GPU
# Load non-expert model weights
act = target expert [iL] [tE] (act)
Set loadedModel to the first model id in M
Load non-expert weights of loadedModel to GPU
else
Use the preloaded expert
id map [iL] [iE)
act M [id] expert [iL] [iE] (act)
tance for given (layer, expert) pair. However, for higher number of models, the distance for each expert location is computed as the sum of distances across all model-to-model combinations.
end if
Apply Add & Norm and compute act of next layer
end for
Subsequently, each expert location will be ranked based on the defined similarity measure, with the highest rank indicating the greatest similarity, corresponding to the low- generate the initial loader's expert map,
Predict nextToken using act Concatenate next Token the end of prompt
until nextToken is eos required number of output tokens generated
est distance. starting from the highest rank, each expert location will be assigned to specific fine-tuned model using round-robin approach. For instance, if we are serving two models, the expert locations with odd ranks will be assigned to the first model, while those with even ranks will be assigned to the second model. Although the time spent generating this ex- pert map can be substantial and depends on the number of models, this process only needs be performed once for each combination prior to load time and can be done offline.
As summarized in Algorithm 1, the loader's inputs are a list of models being served and the total expert capacity. At the beginning the loading process, all the models' weights are stored on CPU's main memory. By the end of the process, a fraction of each model's parameters is loaded onto the GPU, collectively forming unified model ready for inference. Moreover, the generated map is a table that keeps track of each expert and its corresponding model id which will route each input in performing forward pass calculations.
This approach sets the granularity of model merging at the expert level and ensures that the loaded parameters are equally representative all the models being served. After generating the expert map, the parameters should be loaded the GPU memory. Based the available GPU memory and the size of non-expert parameters, the total number of experts that can fit the GPU will be calculated. To start, non-expert parameters of randomly selected model from our model list will be loaded to GPU's memory. Follow- ing this, the loader begins with the highest-ranked expert locations, and transfers their corresponding parameters.
3.2. Inference Orchestration
The loader, explained in Algorithm 1, loads experts from different models. While this approach significantly reduces the memory footprint required to serve all the models, it may decrease the quality of output generation for tasks specific to each model variant.
The Mixtral model evaluated in this paper contains 1.605.64

---

QoS-Efficient Serving of Multiple Mixture-of-Exper LLMs Using Partial Runtime Reconfiguration
CPU Main
Mixtral-Instnal Model
Mixtral-base
Memory
Model
corgettaadei eppon/(4.01
GPU
Layer iL.
Espeiti
from previous layer
Multi- Hend Atteniion
Gate Network
Add
layer
Prompe: "Where is..." Arggttdfade--Miin
Ha
Nonu
Cach
Expert
BURTL Layen
Mistsl-oos
Mistral-hemaat
Expest
Figure Inference process for each layer of the consolidated model: the case of an expert hit, the inference is performed using the already loaded expert, which may not necessarily belong to the requested model. However, in the case of miss, the corresponding expert loaded from the host's memory to preserve the quality of the generated output.
million non-expert parameters. It has 32 layers, each with 8 experts, and uses top-2 sparse routing. During each for- ward pass, a total of 1,605.64 million non-expert parameters and 32 x 2 x 176. 16 million expert parameters (with each expert contributing 166. 16 million parameters) are actively involved in computations. Consequently, the non-expert pa- rameters contribute approximately 14.2% to the generated output.
cesses before arriving at the first layer of the model. Within each layer, the activations are first processed by a multi- headed attention layer and then passed to the gate network. The gate network generates weighted averaging coefficients for each expert, and in sparse gating scenario, the top experts are selected to produce the output of the layer.
As depicted in Figure 3, pairs of layer and expert numbers ((i,, iE)) are generated, and the status of the requested expert is retrieved from the expert map. If the expert is already loaded on the GPU, the corresponding expert's 's result will be calculated, although it is not guaranteed belong to the target odel. However, in the case of an expert miss (dashed lines in Figure 3), load request is sent the CPU's main memory. Since all models are stored in the CPU's main memory, the requested expert will belong to the target Model, preventing further quality degradation. Finally, after processing all layers, the activations generated by the last layer pass through the language model head to produce the output logits.
In contrast to experts, non-expert parameters are always utilized in each forward pass, regardless of activations. Fur- thermore, their total size is approximately GB (2 bytes per parameter). Based on the described system, it takes approxi- mately a second to transfer all non-expert parameters of given model via the host-to-device PCIe link. Therefore, for each new request to the serving system, we propose swap- ping the currently loaded non-expert parameters with those of the target. odel, which are transferred to the GPU, Al- though this process incurs a one-time cost per request and increases the TTFT, it ultimately enhances the quality of the generated output.
4. Evaluation Methodology
Accordingly, the inference orchestrator begins handling each request by comparing the targetModel with the oded. If they match, inference is performed imme- diately. However, if they do not match, the tar non-expert parameters are uploaded to the GPU, replacing the currently loaded non-expert parameters before proceed- ing with inference.
We evaluate our serving system using the Mixtral-xx7B-vl and Google Switch Transformer Base-8 family of models. The Mixtral model has 32 layers and 8 experts per layer and comes in two variants. The Switch Transformer model comes in four variants and has a total 96 experts, which are distributed equally across 12 layers (8 experts per layer). Since in our design, both QoS and the application output quality is affected, we evaluate the proposed system through experiments that independently assess each aspect of its
As summarized in Algorithm 2, during the inference phase, each sequence undergoes tokenization and embedding pro-
5

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
functionality.
measuring perplexity using 128 samples, with each sample consisting of 2048 tokens.
4.1. Quality of Service
To demonstrate the quality of output for instruct tasks, we evaluate each model using the MT-Bench (Zheng et al., 2023) benchmarks. MT--Bench a benchmark specifically designed for instruct models and includes 80 instructions across 8 different categories. Each request prompts the language model to generate an output, and requests may include a follow-up instruction to refine the output and fur- ther improve its quality. The quality of the responses is then judged by the GPT-4 (Achiam et al., 2023) model, which assigns each response grade from 0 to 10.
Since the Switch Transformer Base-8 model has consider- ably lower memory footprint (1.5 GB) compared to Mixtral and does not constrain our environment, we use only the Mixtral model family for this experiment. We evaluate our proposed system in a scenario where requests arrive inde- pendently for each model we support. We consider two separate Poisson processes generate requests for each supported model. To demonstrate the system's 's responsive- ness and performance under varying loads, we evaluate the system at different arrival rates (A). Each request to the system is a prompt consisting of 20 tokens, with the number of requested output tokens set to 25. To compare the QoS of each approach, we measure the average TTFT, average turnaround time, and the total number of processed requests (throughput).
Moreover, to demonstrate contextual understanding and general effectiveness of each model variant, we assess their performance using HellaSwag (Zellers al., 2019), MMLU (Hendryckss et al., 2020), and TruthfulQA (Lin et al., 2021b) benchmarks. For MMLU and TruthfulQA, we use 5-shot prompts, and for HellaSwag, we use O-shot prompts. For MMLU and HellaSwag, we evaluate each model with 1000 random samples, and for TruthfulQA, we evaluate them with 812 samples.
We implement our loader and orchestrator using the Py- Torch (Paszke et al., 2019) framework. Our experiments are conducted on a server with an AMD 16-Core MILAN CPU and a single 80GB NVIDIA A100 GPU, connected via a PCIe Gen4 host-to-device interconnect, as outlined in Section 2. QoS metries are measured from PyTorch's perspective using Python's 's time package. It is important to note that other inference optimizations, such as Flash Atten- tion (Dao et al., 2022), are complementary to the approach presented in this paper and fall outside the scope of this work. To evaluate the system's 's performance, we focus on single-batch requests, providing a controlled setting for our experiments.
Baselines. We compare the quality of the generated output from the proposed consolidated model against the follow- ing models: Mixtral-8x7B-vO). 1base, Mixtral-8x77 -xx-8-xx... 1- Instruct, and Mixtral-8x7B-vOO. 1-avg. The first two models are official relcases by Mistral-AL. However, proposed in (Izmailov et al., 2018), the Mixtral-8x7B-vO.-aay model is generated by averaging the weights of the first two mod- els. We include Mixtral-8x7B--v.. 1-avg in the comparison to evaluate the performance of the proposed model against conventional model-merging approaches. It is important to note that for benchmarks measuring perplexity, the non- expert parameters in our approach are from the base model, while for all other benchmarks, they are from the Instruct model.
QoS Baselines. We compare serving system against two baselines. The first baseline represents scenario in which single Mixtral model is served, but with an arrival rate that is twice that of each Poisson process used to evaluate the pro- posed approach. The second bascline involves partitioning the GPU resources into two using NVIDIA MIG, with each partition assigned to an independent process that handles requests for the base and Instruct models, separately.
4.3. Scalability
To demonstrate the scalability and applicability of our ap- proach to other model architectures with higher num- ber of fine-tuned variants, we also evaluate our approach against the Averaging baseline (Izmailov et al., 2018) using Google's Switch Transformer Base-8 family of models. For this experiment, we use the base model provided by Google along with three other coanmunity-providee fine-tuned vari- ants:
4.2. Application Quality
The designed serving system must provide access to multi- ple models. Therefore, for the Mixtral model, we measure the quality of the generated output using two specific fami- lies of benchmarks corresponding to each model variant: the base model and the Instruct model. During the evaluation of each benchmark family, the non-expert layers of the model set to the respective model variant.
Model A: smre/switch - finetuned-sams
For the base model, we evaluate the quality of text genera- tion using the WikiText2 (Merity et al., 2016), PTB (Marcus et al., 1994), and C4 (Raffel et al., 2020) datasets. For each dataset, we assess the models' generated output by
Model B: google/switchh-aeeeee
Model C: g gglanpoo/swwtcch-bee s-sst2
Model D: gl glamproo/switchhheee
6

---

QoS-Efficient Serving of Multiple Mixture-of-Exper LLMs Using Partial Runtime Reconfiguration
To evaluate how output quality is affected by the number of merged models, we consider the following serving scenar- ios:
Table 1. Latency (milliseconds) of a layer for different hit rates (out of 2 experts).
Serving models A &&
Hit Rate
Attention
0 expert
expert
experts
Serving models A & B & C
Single/
Serving models A & B&CCD
0.72
56.8
29.2
1.2
Proposed
NVIDIA MIG
0.78
104.3
54.1
1.7
5. Results
5.1. Performance and Quality of Service
Table 2. Average TTFT and tumarruund time (scconds) for the pro- posed and compared serving approaches.
Figure 4 compares the throughput of each serving approach across varying arrival rates. For NVIDIA MIG, the reported throughput represents the combined throughput of two in- dependent GPU instances. At lower arrival rates, all three approaches exhibit low throughpur, which is attributed to system under-utilization meaning they can handle all in- coming requests within the experiment's 's time window. As the arrival rate increases, throughput improves until reach- ing distinct ridge points for each approach. For both the proposed system and the single-model serving approach, the ridge point occurs at À = 0.060, while for NVIDIA = 0.040. This indicates that the
TTFT
Turnaround Time
Single
0.89
8,34
Proposed
1.41
8.78
NVIDIA MIG
5.86
49.67
be copied from the GPU's GMEM to SMEM. Consequently, each GPU instance in NVIDIA MIG has sufficient compute resources, and we do not observe significant increase in the latency of the attention layer (0.72 - 0.78).
MIG, observed proposed system and the single-mode approach have higher capacities and can continue handling more requests at higher arrival rates, whereas NVIDIA MIG's performance plateaus earlier.
However, the expert block is compute-bound (Austin et al., 2025). Therefore, when both required experts are avail- able in GMEM, the latency increase is more noticeable (1.2 - 1.7). On the other hand, when the hit rate decreases for the expert block, an overhead (27.8 88 ms for each expert) is introduced due to copying experts from the CPU's DRAM to the GPU's GMEM via the PCIc link. In our approach, when serving a single model, the full bandwidth of the PCIe link is available to the process. However, using NVIDIA MIG with two GPU instances splits the available PCIe band- width between them, nearly doubling the imposed overhead (51.3 ms for each expert).
Table 2 illustrates the average TTFT and turnaround times. Single-model serving and the proposed system achieve sig- nificantly lower TTFT (a maximum of 5.86 seconds) and turnaround times (a maximum of 49.67 seconds) compared to NVIDIA MIG, which supports the results depicted in Figure 4. As demonstrated, each NVIDIA MIG instance takes an average of 49 seconds to complete request.
Moreover, comparing the proposed system against the single-serving approach highlights the overhead of non- expert runtime reconfiguration. As shown in the TTFT sec- tion of Table 2, our approach introduces a delay of approxi- mately half second average. However, the actual mea- sured latency for swapping non-experts is approximately 1.2 seconds, which does not occur for consecutive requests tar- geting the same target odel. It is also important to note that the numbers reported in this subsection are specific to the requests defined earlier and may vary for prompts with different lengths and requested numbers of output tokens.
5.2. Quality of Generated Output
Table 3 presents comparative analysis of the generated out- put quality across various models, evaluating perplexity on the WikiText, C4, and PTB datasets, as well performance metrics on MT-Bench, MMLU, HellaSwag, and TruthfulQA benchmarks. Lower perplexity scores indicate better lan- guage modeling capabilities, while higher scores indicate improved performance on the remaining tasks. While the base model serves as the optimal bascline for text generation, the proposed approach outperforms the averaged model and achieves lower perplexity across all datasets, except for PTB, which has been observed to show irregular performance im- provements even at higher levels of quantization (Eliseev & Mazur, 2023).
To provide deeper insights into the performance of each approach, the inference latency for each model layer is presented in Table 1. As mentioned in (Austin et al., 2025), during the generation phase of a transformer-basee LLM, the attention kernel is memory-bound. This is because the results of prior computations, stored in the KV cache, must
On MT-Bench, the proposed model outperforms others with
7

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Throughpul (landled jequesis/ninuii
Proposed
Single Model
NVIDIA MIG
4-
0.005
0.010
0.015
0.020
0.025
0.030
0.035
0.040
0.045
0,050
0.085
0.080
0.085
0.070
0.075
0 0,080
Arrival Rate (roquest/sec)
Figure 4. The throughput of each serving approach is measured in completed requests per minute. Since NVIDIA MIG utilizes two independent processes to handle requests for different models, the arrival rate for each instance is half of the value shown on the x-axis, and the reported throughput the sum of the throughput for both instances. The reported values represent the average five independent runs.
Table The quality of the generated output, evaluated across different task families, including language generation (Perplexity), instruction following (MT-Bench), and general-purpose purpose multiple-choice questions.
WikiText
C4 |
PTB
MT-Bench (out of 10)
MMLU 1
HellaSwag t
TruthfulQA
Model
(Perplexity)
1st Tum
2nd Turn
Avg
(5-shot)
(0-shot)
(5-shot)
7.24
13.59
3.01
2.60
70.0%
81.1%
66.6%
Instruct
4.12
7.62
14.60
8.28
7.98
8.13
71.2%
83.0%
73.6%
Avg
3.89
7.38
13.44
8.38
7.61
7.99
72.2%
82.0%
73.0%
Proposed
3.85
7.33
13.90
8.33
7.98
8.16
71.7%
82.2%
70.6%
average of 8.16, closely aligning with the high- quality outputs of the Instruct model. Notably, although the averaged model demonstrates superior performance in the first turn, its quality drops significantly when responding to the follow-up refinement prompt and exhibits high variance.
0.49 0.42 -> 0.33 0.25. However, approach demonstrates greater resilience, maintaining higher out- put quality even with more variants, with scores of 0.49 + 0.49 + 0.46 + 0.46. For more detailed results, in- cluding ROUGE-2, ROUGE-L, and ROUGE-Lsume which follow a similar trend refer to Table Appendix
Additionally, the proposed system achieves competitive re- sults on MMLU (71.7%). HellaSwag (82.2%), and Truth- fulQA (70.6%) which demonstrates balanced performance. These results highlight the ability of the proposed system to maintain high output quality at other gureral- benchmarks.
6. Related Work
Reducing the overhead of the host-to-device link is crucial for efficiently serving MoE models in constrained environ- ments and efficient expert scheduling. Therefore, model compression techniques such as pruning, and quantization (Lin et al., 2023; Huang al., 2024; Eliseev & Mazur, 2023; Imani al., 2024; Dettmers & Zettlemoyer, 2023; Xiao et al., 2023) have been utilized extensively (Liu et al., 2024) to speed up the inference process.
5.3. Scalability
Presented in Figure 5, we report the ROUGE-1 scores (an N-gram based summarization evaluation metric) for a summarization task on the SAMSum dataset (Gliwa al., 2019), which Model A specifically finetuned for. As shown, increasing the number of merged mod- els significantly degrades output quality for the Aver- aging baseline, with ROUGE-1 scores dropping from
Model Pruning Expert pruning in MoE models aims to reduce the number of parameters while maintaining model accuracy. This process is typically categorized into struc-
8

---

QoS-Efficient Serving of Multiple Mixture-of-Exper LLMs Using Partial Runtime Reconfiguration
0.5
0.434
0.401
0.490
0,463
0.422
0.4
ROU/EE- Soore
0.256
0.2
0.149
0.1
0.000
Ave (A,
Proposed
Proposed D.
Proposed (,,
Model
Model
Model
Model D
D.
(A. C.
0.0
Figure 5. ROUGE-1 scores (higher is better) on SAMSum dataset for individual models and their combinations, comparing the proposed approach against the averaging haseline. Model layouts illustrated beneath each bar. In our approach, a portion of the expert parameters (squares) and all non non-expert parameters (rectangles) are retained from the model specifically trained for the benchmark (Model A, shown in bluc). In contrast, the averaging approach modifies all parameters by averaging across the merged models.
tured and unstructured pruning. Structured pruning methods focus on reducing the number of experts, with some ap- proaches directly removing unimportant experts and others merging them. Importance of expert can be defined us- ing different metrics. For example, approaches proposed in (Chen et al., 2022; Muzio et al., 2024; Park, 2024) identify the unimportant experts based on frequency and scareity and prune non- non-essential experts while fine-tuning important ones for target tasks. Moreover, similar to our approach, authors in (Chowdhury et al., 2024; Yang et al., 2024a) pro- pose to prune the experts of a fine-tuned model that have lower L2 distances from the base model. However, these approaches focus on performance of single model and do not consider a multi-model multi-task system.
our approach enhances both efficiency and effectiveness by integrating dynamic runtime reconfigurability with static merging.
7. Conclusion
In this paper, we presented an efficient serving system designed address the challenges of deploying multi- ple fine-tuned MoE-based LLMs in single-GPU resource- constrained settings. By leveraging similarity- -bsed consol- idation, our approach reduced the overall memory footprint by sharing similar experts across models. Additionally, runtime partial reconfiguration allowed non-expert layers to be dynamically swapped, maintaining competitive out- put quality. Our evaluation, conducted on server with single NVIDIA A100 GPU, demonstrated the system's s's ability to significantly reduce turnaround time compared to conventional virtualization approaches like NVIDIA MIG, resulting in comparable throughput to single-model serving with only a minimal increase in TTFT. The results also high- lighted that our approach maintained high output quality, as evidenced by strong performance across various bench- marks.
Multi-Task Model Merging Model merging integrates multiple models into a single model by performing weight interpolation at the parameter level, serving as an efficient alternative (Singh & Jaggi, 2020; Yang et al., 2024b; 2023). Existing model merging methods include weighted merging (Matena & Raffel, 2022; Wortsman et al., 2022; Jin et al., 2022), which assigns varying importance to different mod- els, and subspace merging (Du et al., 2024; Suvizi et al., 2025; Yadav al., 2023), which eliminates unimportant neurons reduce task interference. However, these ap- proaches use static merging strategies, limiting adaptability. Moreover, techniques such as activation matching, and per- mutation invariance help minimize discrepancies. Although these approaches demonstrate improved performance on considerably smaller models, their applicability for larger MoE architectures remains unexplored. Additionally, un- like previous methods that rely solely on static merging,
A Acknowledgements
This work was supported by the National Science Founda- tion under Grant No. 2038682. We thank Pedram Akbarian and Rasool Sharifi for the helpful discussions.
9

---

QoS-Efficient Serving of Multiple Mixture-of-Exper LLMs Using Partial Runtime Reconfiguration
Impact Statement
reconfigurable computing. ACM Transactions Recon- figurable Technology and Systems (TRETS), 1(4):1-23, 2009.
This paper presents work whose goal to advance the field of Machine Learning. There are potential societal conse- quences of our work; however, given the experimental na- ture of the paper, consequences here.
Eliseev, A. and Mazur, D. Fast inference of mixture-of- experts language models with offloading. arXiv preprint arXiv:23/2. /2.2332....... 2023.
unable to anticipate any negative
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Sam- sum corpus: human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Asso- ciation for Computational Linguistics, 2019. doi: 10. 18653/v1/41--400. URL http://dx.co www.ooo..oooo 10. wt.16633/vl/D1se
Austin, J., Douglas, S., Frostig, R., Levskaya, A., Chen, C., Vikram, S., Lebron, F., Choy, P., Ramasesh, V., Web- son, A., and Pope. R. How to scale your model. 2025. Retrieved from
Chen, T., Huang, S., Xie, Y., Jiao, B., Jiang, D., Zhou, H., Li, J., and Wei, F. Task-specific expert pruning for sparse mixture-of-eppers. arXiv preprint arXiv:2206.00277, 2022.
He, Vero, M., Krasnopolska, G., and Vechev, M. Instruc- tion tuning for secure code generation. arXiv preprint arXiv: 24fi::2002.999999 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 2xXiv:2009003000 2020.
Choquette, J., Gandhi, W., Giroux, O., Stam, N., and Krashinsky, R. Nvidia a100 tensor core gpu: Perfor- mance and innovation. IEEE Micro, 41(2):29-33 35, 2021.
Chowdhury, M. N. R., Wang, M., Maghraoui, K. E., Wang, N., Chen, P.-Y,, Y., and Carothers, C. provably effective method for pruning experts in fine-tuned sparse mixture- of-experts. arXiv preprint arXiv:2405.1664 2024.
Huang, M., Narayana, K., Simmler, H., Serres, O., and EI- Ghazawi, T. Reconfiguration and communication aware task scheduling for perffommoe reconfigurable com- puting. ACM Transactions on Reconfigurable Technology and Systems (TRETS), 3(4):1-25. 2010.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io- awareness, 2022. URL
Huang, W., Liu, Y., Qin, H., Li, Y., Zhang, S., Liu, X., Magno, M., and Qi, X. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:
org/ abs/
14135.
2024.
Dettmers, T. and Zettlemoyer, L. The case for 4-bit preci- sion: k-bit inference scaling laws. In International Con- ference on Machine Learning, pp. 7750-7774. PMLR, 2023.
Imani, H., Amirany, A., and El-Ghazawi, T. Mixture of experts with mixture of precisions for tuning quality of service, arXiv preprint vrXiv:447..14417 2024.
Du, G., Lee, J., Li, J., Jiang, R., Guo, Y., Yu, S., Liu, H., Goh, S. K., Tang, H.-K., He, D., et al. Parameter competition balancing for model merging. arXiv preprint arXiv:2410.02396 2024.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: /803. 0803005477, 2018
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of- experts. In International Conference on Machine Learn- ing, pp. 5547-5569. PMLR, 2022.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
El-Araby, E., Gonzalez, I., and El-Ghazawi, T. Exploit- ing partial runtime reconfiguration for perforanfe
10

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language 2022.
Muzio, A., Sun, A., and He, C. Seer-moe: Sparse expert efficiency through regularization for mixture-of-experss arXiv preprint arXiv:24044 05089, 2024.
models. arXiv preprint
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2): 181-214, 1994, doi: 10.
Park, S.
Learning more generalized experts by merg- ing experts in mixture-of-experts arXiv preprint arXiv:2405.1 S.Ki::205..11300 11530, 2024.
Kamahori, K., Gu, Y., Zhu, K., and Kasikci, B. Fiddler: Cpu-gpu orchestration for fast inference of mixture-of: experts models. arXiv preprint arX1v:2402.07033 2024.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Kim, Y. J.. Awan, A. A., Muzio, A., Salinas, A. F. C., Lu, L., Hendy, A., Rajbhandari, S., He, Y., and Awadalla, H. H. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109, 10465, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Li, T., Narayana, V. K., El-Araby, E., and El-Ghazawi, T. Gpu resource sharing and virtualization on high perfor- mance computing systems. In 2011 International Confer- ence on Parallel Processing, pp. 733-742. IEEE, 2011.
Rajbhandari, S., Li, C.. Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., RaRley. J., and He, Y. Deepspeed-moe: Advancing mixture-of-expert inference and training to power next-generation scale. In International con- ference on machine learning, pp. 2022.
Lin, J., Yang, A., Bai, J., Zhou, C., Jiang. L., Jia, X., Wang, A., Zhang, J., Li, Y., Lin, W., et al. M6-10t: A sharing- delinking paradigm for efficient multi-trillion parameter pretraining, arXiv preprint arXiv:2110.0388 2021a.
PMLR,
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. 2017.
Lin, J., Tang, J., Tang. H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awe weight quantization for Ilm compres- sion and acceleration. arXiv preprint urXiv:2306.0000 00978, 2023.
arXiv preprint arXiv:
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young. C., al. tess-tenso Deep learning for super- computers. Advances in neural information processing systems, 31, 2018.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprinl arXiv:2109.07958, 2021b.
Shen, L.,, Tang, A., Yang, E., Guo, G., Luo, Y., Zhang, L., Cao, X., Du, B., and Tao, D. Efficient and effec- tive weight-ensembling mixture of experts for multi-task model merging. arXiv preprint arXiv:2410.21804, 2024.
Liu, J., Tang, P., Wang, W., Ren, Y., Hou, X., Heng. P.-.., Guo, M., and Li, C. survey on inference optimization techniques for mixture of experts models. arXiv preprint arXiv:2412, 142 2024.
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 0122005-220555 2020.
Marcus, M. Kim, G.. Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
Suvizi, A., Subramaniam, S., Lan, T., and Venkatara- mani, G. Exploring in-memory accelerators and fp- gas for latency-sensitiv dnn inference on edge servers. In 2024 IEEE Cloud Summit, pp. 1-6, 2024. doi: 10.1 Suvizi, A., Ahmadi, R., Zamani, M. S., and Meybodi, M. R. A parallel computational approach for energy-efficient hydraulic analysis of water distribution networks using learning automata. Sustainable Computing: Informatics and Systems, pp. 101 135, 2025. ISSN 2210-5379.
Matena, M. S. and Raffel, C. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 15:17703-17716, 2022.
Merity, S., Xiong, C. Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 1o09v 1609.07843 2016.

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
doi: htts://dd..oeo//ooooo//.sssssoss
URL httpps//www.acieeneedeeeeee sutance/Arttceee/////22222222200
com/
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontij-LLpess, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998, PMLR, 2022.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023,
Xue, L., Fu, Y., Lu, Z., Mai, I.,, and Marina, M. Moe- infinity: Activation-aware expert offloading for efficient moe serving. arXiv preprint arXiv:2401. 14361, 2024.
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. Resolving interference when merging models. arXiv preprint 21X.:22306001008,, 1, 2023.
Yang. C., Sui, Y., Xiao, J., Huang. L., Gong, Y., Duan, Y., Jia, W., Yin, M., Cheng, Y., and Yuan, B. Moe-i²: Compressing mixture of experts models through inter- expert pruning and intra-expert low-rank decomposition. arXiv preprint Xr 1.01016, 2024a.
Yang, E., Wang, 7.,, Shen, L.,, Liu, S., Guo, G., Wang. X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv22310.02555, 2023.
Yang, E., Shen, L.,, Wang, Z.,, Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Surgeryv2: Bridging the gap be- tween model merging and multi-task learning with deep representation surgery. arXiv preprint arXiv:2410. 14389, 2024b.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:19055 1905.07830 2019.
Zheng, L., Chiang, W.-., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging -lm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zuo, S., Liu, X., Jiao, J., Kim, Y. J., Hassan, H., Zhang, R., Zhao, T., and Gao, J. Taming sparsely activated transformer with stochastic experts. arXiv preprint 10.04:2110.04266, 2021.
12

---

QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
A. Complete Results of SAMSum Dataset
Table presents the detailed results for summarization task on SAMSum dataset. As demonstrated, all metrics ROUGE-2, ROUGE-L, and ROUGE-Lsunn follow similar trend, highlighting the greater resilience of our approach as the number of served models increases.
Table 4, ROUGE scores (higher is better) on SAMSum dataset for individual models and their combinations.
Model
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-Lsum
Model A
0.493
0.250
0,409
0,409
Model B
0.148
0.026
0.128
0.128
Model C
0.052
0.004
0.046
0.046
Model D
0.040
0.009
0.036
0.036
Avg (A, B)
0.422
0.201
0.344
0.344
Avg (A, B, C)
0.333
0.132
0.277
0,277
Avg (A, B, C, D)
0.256
0.079
0.209
0,209
Proposed (A, B)
0.491
0.245
0.407
0.406
Proposed (A, B, C)
0.460
0.229
0.377
0.377
Proposed (A, B, C, D)
0.463
0.229
0.377
0.377
13


Reviews:
[{"review_id": "KSSCyMb1FF", "reviewer": "Reviewer_R9A4", "summary": "Updated score 2->3 after rebuttal.\n\n-----------------------\n\nThe paper proposes a novel serving system to address the problem of efficiently serving multiple finetuned mixture-of-experts (MoE) large language models (LLMs). The idea is to run a similarity based expert consolidation to share similar experts across different models, which can reduce the memory overhead on a single GPU. The authors also propose a runtime partial reconfiguration scheme to dynamically unload non-expert layers to process requests from different models. The authors run experiments on single NVIDIA A100 GPU using Mixtral-8x7B models. The authors compare their approach with NVIDIA's multi-instance GPU (MIG), and show that the proposed method can achieve an 85% reduction in turnaround time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable comments. The experiments and additional explanations provided here have been incorporated into the revised manuscript.\n\nWeaknesses:\n\nW1: I mentioned above about using only ...\n\nResponse 1:\n*For a detailed response with numerical results, please refer to “Response 1” given to Reviewer 2 (xadW). The response given below is a summarized version:\nTo demonstrate the applicability of our approach to other model families with a higher number of fine-tuned variants, we evaluated our approach against the Averaging baseline (Avg: Izmailov et al., 2018) using Google’s Switch Transformer version 8 MoE-based model (Hugging Face (HF) ID: google/switch-base-8). This model has a total of 96 experts, which are distributed equally across 12 layers (8 experts per layer).\nMoreover, to assess the scalability of our approach we used the base model provided by Google along with three other community-provided fine-tuned variants.\nAs shown, by increasing the number of merged models, the averaging approach suffers considerably from quality reduction (0.49 -> 0.42 -> 0.33 -> 0.25). However, our approach demonstrates greater resilience and can preserve the quality of generated output even for higher number of variants (0.49 -> 0.49 -> 0.46 -> 0.46).\n\nW2: Additionally, the use case of the proposed ...\n\nResponse 2: We agree with the reviewer. Our approach is specifically designed for MoE models with identical network architectures, which is a key assumption in our problem statement. Despite this limitation, the network architectures discussed in the paper are among the leading performers in language tasks, demonstrating significant capabilities when fine-tuned for various functionalities and datasets. That said, we believe the model-merging problem addressed in this paper remains relevant and important to the research community.\n\nW3: The experiments use the Base version and the Instructed version ...\n\nResponse 3: In the new experiments included in the revised manuscript, a new family of models with 4 different variants has been added for evaluation and these models exhibit more differences. For more details, please refer to Response 1.\nThe cost of swapping models is directly proportional to the number of non-expert parameters which results in a higher volume of data transferred through the CPU-GPU link (PCIe). Therefore, since the PCIe bandwidth is fixed, a higher number of non-expert parameters equals higher swap time (cost). This is the overhead that is added to TTFT which increases it by almost 58%. Although this increase may seem significant for TTFT, it allows us to decrease the total turnaround time by 85% and to preserve the quality of generated output.\n\nW4: Line 162 left: \"The on-device expert weights ...\n\nResponse 4: The Mixtral model has a total of 256 experts (32 layers and 8 experts per layer). Each expert has 176.16 million parameters. All non-expert parameters from 32 layers total 1,605 million parameters. The A100 GPU can fit only 217 experts out of 256. The memory footprint of each part of the model is summarized below:\n\nOn-GPU non-expert params: 1605 million * 2B (3.5%)\n\nOn-GPU expert params: 217 * 176.16 million * 2B (81.8%)\n\nOff-GPU expert params: 39 * 176.16 million * 2B (14.7)\n\nW5: Figure 2: I have difficulty understanding how L2 distances ...\n\nResponse 5: Assuming identical network structures, the experts from different variants have the same structure. By flattening the weights, we mean that all the weights for a given expert are stored in a 1D list while preserving the order. \nIn a hypothetical case, each expert has parameters in the shape of 3*6 (a 2D tensor). This 2D tensor can be converted into a 1D tensor of size 18. Therefore, assuming we have two model variants, the expert-to-expert distance is the L2 distance between the two converted 1D tensors. \nTransformer-based LLMs typically have multiple layers. The Mixtral model has 32 layers. Figure 2 shows the expert-to-expert distance from Mixtral-base to Mixtral-Instruct. \nHere, expert-to-expert refers to experts in the same positions. For example, it represents the distance from expert 3 of layer 21 in Mixtral-base to expert 3 of layer 21 in Mixtral-Instruct. \n\nW6: Line 140 right: I can infer it but (iL, iE) is ...\n\nResponse 6: These are variables that show the expert position in the model structure. Will be clarified in the revised manuscript. \n\niL: layer number \n\niE: expert number  \n\nW7: Same place: How is this range 150 to 250 ... \n\nResponse 7: By \"the same expert positions,\" we mean two different experts from different model variants with identical iL and iE. \n\nW8: Line 149 right: It would be helpful if the authors ... \n\nResponse 8: A formal mathematical definition for a higher number of model variants is provided in the revised manuscript. \n\nW9: Algorithm 1: what is m on Line 183? Seems not defined. \n\nResponse 9: This is a mistake from our end. The correct variable inside the bracket should be “idx” instead of “m”.", "claims_and_evidence": "The authors mainly compare the proposed approach against the NVIDIA's multi-instance GPU (MIG) and the single model approach (where one single Mixtral model is served with twice the number of requests on average). While I agree that from the experimental results, the proposed approach is better than the MIG, I am not sure what's the gain of the proposed approach over the single model in terms of the output quality (Table 1), latency (Table 3) and throughput (Figure 4). The difference looks not significant to me.\n\nThe authors also claim the proposed approach \"incurring only a negligible increase in time-to-first-token\", which might be an understatement. Table 2 shows an increase of TTFT from 0.89s to 1.41s which is a 58% increase. It would be helpful if the authors could clarify this.\n\nThe experiments only take one set of settings (prompt tokens, arrival rates, GPU, etc.) and one base model (Mixtral-8x7B). It's not clear how the proposed approach would perform in more general settings, or if the TTFT gap would be larger in more demanding settings or with larger models.", "methods_and_evaluation_criteria": "The datasets and evaluation metrics (latency, throughput, quality) make sense and are relevant for the claims made in the paper. But my same concerns about the experimental settings above apply here as well: the authors only run experiments on one set of settings and one base model, which makes it hard to generalize the results. It would be helpful if the authors could include more experiments on different settings and models to support their claims.", "theoretical_claims": "N/A. This is an experimental paper and does not contain theoretical claims or proofs.", "experimental_designs_or_analyses": "Yes. See above.", "relation_to_broader_scientific_literature": "I am not an expert in this area and I cannot comment on the literature discussion.", "essential_references_not_discussed": "I am not an expert in this area and I am not familiar with the literature.", "other_strengths_and_weaknesses": "My biggest concern about the proposed approach is its generalizability:\n\n- I mentioned above about using only one setting and one base model. \n\n- Additionally, the use case of the proposed approach feels a bit limited. If my understanding about the similarity calculation is correct, the proposed approach can only be used when the MoE models have exactly the same network architecture. This is a strong limitation, as it means that the proposed approach cannot be used to serve different models with different architectures (unless the authors can provide a consistant way to define block and layer similarity between different architectures), which arguably would be way more useful in practice than serving two versions of the same model. Could the authors elaborate on this?\n\n- The experiments use the Base version and the Instructed version of the Mixtral-8x7B model, and naturally the similarity between the two models is high. If the two models are not similar enough, will the proposed approach still work? Or the cost of swapping layers will be too high?\n\nBeyond that, I feel the writing of the paper in Section 3 could be improved as some definitions are not clear enough:\n\n- Line 162 left: \"The on-device expert weights constitute the majority of the parameters loaded into GPU memory.\" It would help if the authors could include some numbers about the usage of the three categories.\n\n- Figure 2: I have difficulty understanding how L2 distances are calculated between the experts. On Line 134 right, the authors mention that the weights of an expert block are flattened into a vector, and the L2 distances are calculated between the vectors. However, the authors do not mention how the weights are flattened. Additionally what does layer mean in Figure 2? Are the authors running the L2 distance between the weights of the same layer in different experts? If so, it would be helpful to clarify this or even include a diagram to illustrate the flattening and calculation process.\n\n- Line 140 right: I can infer it but (iL, iE) is not defined.\n\n- Same place: How is this range 150 to 250 related to the range of 3.6 to 4.9 mentioned above? What do the authors mean by \"same expert positions\"?\n\n- Line 149 right: It would be helpful if the authors could clarify by giving a math definition for the higher number of models case. \n\n- Algorithm 1: what is $m$ on Line 183? Seems not defined.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B0f9ACepsq", "reviewer": "Reviewer_xadW", "summary": "This paper presents a novel serving system for multiple fine-tuned Mixture-of-Expert (MoE) Large Language Models (LLMs) on a single GPU. The approach uses similarity-based expert consolidation to share similar experts across models, coupled with runtime partial reconfiguration to dynamically replace non-expert layers when processing requests from different models. Experiments with Mixtral-8x7B models demonstrate significant reduction in turnaround time compared to MIG.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable comments. The experiments and additional explanations provided here have been incorporated into the revised manuscript. \n\nWeaknesses: \n\nW1: Limited analysis of scaling behavior with >2 models: \n\nResponse 1: To demonstrate the scalability and applicability of our approach to other model architectures with a higher number of fine-tuned variants, we evaluated our approach against the Averaging baseline (Avg: Izmailov et al., 2018) using Google’s Switch Transformer version 8 MoE-based model (Hugging Face (HF) ID: google/switch-base-8). This model has a total of 96 experts, which are distributed equally across 12 layers (8 experts per layer). \n\nFor this experiment, we used the base model provided by Google along with three other community-provided fine-tuned variants: \n\nModel A: emre/switch-base-8-finetuned-samsum \n\nModel B: google/switch-base-8 \n\nModel C: glamprou/switch-base-8-sst2 \n\nModel D: glamprou/switch-base-8-mnli \n\nTo compare the reduction in quality of output based on the number of merged models, we consider three different serving scenarios:  \n\nServing models A & B \n\nServing models A & B & C \n\nServing models A & B & C & D  \n\nIn the table below, we report the ROUGE scores (an N-gram based summarization evaluation metric) for a summarization task on the SAMSum dataset (HF id: samsung/samsum), which Model A is specifically finetuned for: \n\nTable 1: \n\n---- \n\n** R1: ROUGE-1, R2: ROUGE-2, L:  ROUGE-L (For all three metrics higher is better) \n\nModel A: R1: 0.4939, R2: 0.2504, L: 0.4096 \n\nModel B: R1: 0.1487, R2: 0.0268, L: 0.1287 \n\nModel C: R1: 0.0523, R2: 0.0048, L: 0.0460 \n\nModel D: R1: 0.0404, R2: 0.0093, L: 0.0361 \n\nAvg(A, B): R1: 0.4217, R2: 0.2018, L: 0.3445 \n\nProposed(A, B): R1: 0.4911, R2: 0.2453, L: 0.4070 \n\nAvg(A, B, C): R1: 0.3335, R2: 0.1325, L: 0.2776 \n\nProposed(A, B, C): R1: 0.4600, R2: 0.2292, L: 0.3774 \n\nAvg(A, B, C, D): R1: 0.2561, R2: 0.0799, L: 0.2091 \n\nProposed(A, B, C, D): R1: 0.4633, R2: 0.2290, L: 0.3772 \n\n---- \n\nAs shown, by increasing the number of merged models, the averaging approach suffers considerably from quality reduction (0.49 -> 0.42 -> 0.33 -> 0.25). However, our approach demonstrates greater resilience and is able to preserve the quality of generated output even for higher number of variants (0.49 -> 0.49 -> 0.46 -> 0.46).  \n\n \n\nW2: No discussion of privacy implications when sharing experts: \n\nResponse 2: We agree with the reviewer that a central merging system should be established to implement our approach and to generate a consolidated model, which may raise privacy concerns. However, in our approach, only model weights are shared with the central system (similar to Federated Learning), while training data and validation samples are not shared with other users. \n\nW3: Lack of ablation studies isolating the impact of different components: \n\nResponse 3: To address this comment, we added further experiments to the evaluation section that demonstrate the output quality without the non-expert reconfiguration component. The following results will be included in Table 1 of the revised manuscript. \n\nProposed-No-Reconfiguration: WikiText: 4.05, C4: 7.55, PTB: 13.12, MT-Bench(1st): 8.06, MT-Bench(2nd): 7.35, MT-Bench(avg): 7.70, MMLU: 71.80%, HellaSwag: 81.2%, TruthfulQA: 69.7%  \n\nAs shown, without the non-expert reconfiguration component, the quality of the generated output decreases. Although this reduction in quality is not as significant as that of the original opposing models, it causes our proposed approach to perform worse than the Averaging baseline (Avg: Izmailov et al., 2018). \n\nW4: Energy efficiency analysis is absent: \n\nResponse 4: While it is not feasible to assess power consumption within the given time frame, we will include an in-depth analysis of power consumption and its behavior for our approach and the compared baselines in the future extension of this work.", "claims_and_evidence": "The authors maintain that their technique:\n1. Reduces memory overhead through expert consolidation (supported by implementation data)\n2. Achieves competitive output quality (backed by comprehensive benchmarks)\n3. Delivers throughput comparable to single-model serving with minimal increase in TTFT (validated in experiments)\n4. Reduces turnaround time by 85% compared to NVIDIA MIG (well-documented)\n\nThe evidence presented generally validates these assertions, particularly the performance improvements over NVIDIA MIG.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not provide sufficient theoretical analysis. While the similarity-based consolidation is intuitively sensible, the authors offer minimal theoretical justification for why this approach preserves model performance.", "experimental_designs_or_analyses": "Experimental design on realistic hardware (A100 GPU), is thorough: 1) multiple arrival rates examined using Poisson distribution; 2) comprehensive benchmarking across diverse tasks; 3) appropriate baselines (single model, NVIDIA MIG, weight-averaged model)", "relation_to_broader_scientific_literature": "The paper satisfactorily situates the work within MoE serving literature and model merging research. However, connections to multi-tenant serving systems outside the MoE context could be enhanced.", "essential_references_not_discussed": "NaN", "other_strengths_and_weaknesses": "Strengths:\n1. Practical solution to an important real-world problem\n2. Clear exposition of the approach\n3. Strong empirical results\n\nWeaknesses:\n1. Limited analysis of scaling behavior with >2 models\n2. No discussion of privacy implications when sharing experts\n3. Lack of ablation studies isolating the impact of different components\n4. Energy efficiency analysis is absent", "other_comments_or_suggestions": "The paper would benefit from:\n1. Ablation studies separating expert consolidation and runtime reconfiguration effects\n2. Analysis of how performance varies with different expert similarity thresholds", "questions_for_authors": "How does the approach scale to >2 models? Does performance degrade?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7ED4bI1YC", "reviewer": "Reviewer_6X1s", "summary": "This paper addresses the challenge of efficiently serving multiple fine-tuned MoE on a single GPU. The authors propose a novel serving system with two key components: Similarity-based expert consolidation and Runtime partial reconfiguration.\nThe authors evaluate their approach using Mixtral-8x7B models on a server with a single NVIDIA A100 GPU. They demonstrate that their method achieves output quality comparable to individual models while maintaining throughput similar to serving a single model, with only a small increase in TTFT. Compared to NVIDIA's MIG approach, their system shows an 85% average reduction in turnaround time.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable comments. The experiments and additional explanations provided here have been incorporated into the revised manuscript.  \n\nWeaknesses: \n\nW1: The approach is demonstrated only for two models from the same model family (Mixtral base and instruct variants). It's unclear how well it would scale to more diverse models with potentially greater differences. \n\nResponse 1:  \n\n*For a complete, detailed response with numerical results, please refer to “Response 1” given to Reviewer 2 (xadW). Due to the character limit, the response given below is a summarized version: \n\nTo demonstrate the applicability of our approach to other model families with a higher number of fine-tuned variants, we evaluated our approach against the Averaging baseline (Avg: Izmailov et al., 2018) using Google’s Switch Transformer version 8 MoE-based model (Hugging Face (HF) ID: google/switch-base-8). This model has a total of 96 experts, which are distributed equally across 12 layers (8 experts per layer). \n\nMoreover, to assess the scalability of our approach we used the base model provided by Google along with three other community-provided fine-tuned variants. \n\nAs shown, by increasing the number of merged models, the averaging approach suffers considerably from quality reduction (0.49 -> 0.42 -> 0.33 -> 0.25). However, our approach demonstrates greater resilience and is able to preserve the quality of generated output even for higher number of variants (0.49 -> 0.49 -> 0.46 -> 0.46). \n\n \n\nW2: While the authors mention the approach is applicable to MoEs with different sizes and parameters, they don't provide experimental evidence for this claim.  \n\nResponse 2: As discussed in \"Response 1,\" we expanded our evaluation to assess the applicability of our approach to Google's Switch Transformer version 8, an MoE-based model. Compared to Mixtral, the switch-base-8 model has a smaller memory footprint. Each expert in the Mixtral model consists of three layers, resulting in a total of 176,162,304 parameters. In contrast, each expert in switch-base-8 has two layers, with a total of 4,718,592 parameters. The expert sparsity degree for both models is the same, as they each select the top two experts per layer for computation. \n\nW3: The latency analysis in Table 3 could benefit from more explanation about why certain patterns occur and their implications. \n\nResponse 3: As mentioned in Austin et al., \"How to Scale Your Model\" [1], during the generation phase of a transformer-based LLM, the attention kernel is memory-bound. This is because the results of prior computations, stored in the KV cache, must be copied from the GPU’s GMEM to SMEM. Consequently, each GPU instance in NVIDIA MIG has sufficient compute resources, and we do not observe a significant increase in the latency of the attention layer (0.72 -> 0.78). \n\nHowever, the expert block is compute-bound [1]. Therefore, when both required experts are available in GMEM, the latency increase is more noticeable (1.2 -> 1.7). On the other hand, when the hit rate decreases for the expert block, an overhead is introduced due to copying experts from the CPU’s DRAM to the GPU’s GMEM via the PCIe link. In our approach, when serving a single model, the full bandwidth of the PCIe link is available to the process. However, using NVIDIA MIG with two GPU instances splits the available PCIe bandwidth between them, nearly doubling the imposed overhead. \n\n \n\nSingle/Proposed: 1.2 -> 29.2 -> 56.8 (27.8 ms for each expert not being available in GMEM) \n\nNVIDIA MIG with two instances: 1.7 -> 54.1 -> 104.3 (51.3 ms for each expert not being available in GMEM) \n\n[1] Austin et al., \"How to Scale Your Model\", Google DeepMind, online, 2025. \n\nQuestions:  \n\nQ1: The author mentioned that \"Although this paper focuses specifically on fine-tuned LLMs with ... \n\nResponse 4: Please check our response to the first weakness (“Response 1”) \n\nQ2: Would your approach be applicable to other MoE architectures beyond the Mixtral family, particularly those with different gating mechanisms or expert structures? \n\nResponse 5: Please check our response to the second weakness (“Response 2”) \n\nQ3: The experiment does not have a detailed analysis of the overhead of each part of this method. \n\nResponse 6: In our approach, a central system must be established to generate a consolidated map by analyzing each model variant. Although calculating expert-to-expert distances can be time-consuming, it is a one-time process for a given list of models and can be performed offline. The time required for this process depends on the structure of the experts within each model family. A higher number of parameters per expert or a greater number of model variants will increase the processing time. \n\nRegarding the overhead of the inference process (Algorithm 2) compared to baseline approaches, please refer to the response provided for Weakness 3 (\"Response 3\").", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "It helps improve efficiency.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- The paper addresses a practical and significant problem in deploying multiple fine-tuned MoE LLMs in resource-constrained environments.\n- The proposed system achieves a good balance between quality and performance, outperforming conventional virtualization approaches.\n\nWeaknesses:\n- The approach is demonstrated only for two models from the same model family (Mixtral base and instruct variants). It's unclear how well it would scale to more diverse models with potentially greater differences.\n- While the authors mention the approach is applicable to MoEs with different sizes and parameters, they don't provide experimental evidence for this claim.\n-The latency analysis in Table 3 could benefit from more explanation about why certain patterns occur and their implications.", "other_comments_or_suggestions": "- It would be interesting to see how this approach performs with more than two models and with models that have been fine-tuned for more diverse tasks.", "questions_for_authors": "I have the following concerns,\n1. The author mentioned that \"Although this paper focuses specifically on fine-tuned LLMs with identical architectures and text generation tasks, the proposed technique is applicable to MoEs with different sizes and number of parameters.\" The approach is demonstrated only for two models from the same model family (Mixtral base and instruct variants). It's unclear how well it would scale to more diverse models with potentially greater differences.\n2. Would your approach be applicable to other MoE architectures beyond the Mixtral family, particularly those with different gating mechanisms or expert structures?\n3. The experiment does not have a detailed analysis of the overhead of each part of this method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission initially got Weak Reject, Accept, and a Weak Reject ratings. After discussion from the rebuttal phase, one reviewer updated his rating from Weak Reject to Weak Accept. Across the board, all Reviewers were concerned that the submission only included a single Model Family to prove their technique. The Authors included experiments in the discussion w/ Reviewers showing that their technique is also useful when using “Google’s Switch Transformer version 8 MoE-based model”. Given that this was the main concern and it was addressed at least with another model, I am recommending acceptance. I would encourage the authors to include the new results and any useful feedback that reviewers raised during the discussion phase.
Decision: Accept (poster)