ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models

## Abstract

Abstract The typical process for LLM’s development involves pre-training a general foundation model on massive data, followed by fine-tuning on task-specific data to obtain a series of specialized experts. Serving these experts can pose significant memory challenges, as loading all experts onto devices is impractical, and frequent switching between experts in response to user requests can incur substantial I/O costs.
Previous approaches decompose the expert weights as the pre-trained weights plus delta weights, followed by quantizing the delta weights using output channel-wise step sizes to reduce the model size. However, these methods overlook the fact that certain input channels of delta weights can cause significant quantization errors at extremely low bitwidths. Additionally, existing methods assume that the appropriate model for a user request is known in advance, which is not the case in practice. To this end, we introduce ME-Switch, a memory-efficient expert switching framework tailored for serving multiple LLMs . To condense the number of bits required for describing the delta weights, we propose a salient-aware delta compression method that first identifies which input channels of delta weights are salient based on reconstruction error and then employs mixed-precision quantization that selectively quantizes non-salient input channels of delta weights to extremely low bits while keeping the salient ones intact, significantly reducing storage demand while maintaining performance.
Moreover, we develop a model-level routing method that efficiently directs user queries to the most suitable expert by performing domain classification . Extensive experiments show the promising memory efficiency and routing performance of ME-Switch. For example, when serving three models from the Mistral-7B family, ME-Switch reduces the model size by 1.74 × 1.74\times 1.74 × and maintains nearly lossless performance on instruction, mathematical reasoning, and code generation tasks. Furthermore, our method can efficiently serve 16 Mistral-7B models on an NVIDIA A100 GPU.

## 1 Introduction

Large language models (LLMs) such as GPT-4 (Achiam et al., ) and Gemini (Team et al., ), have achieved significant advancements in natural language processing (NLP). Trained on large-scale text datasets, these models develop a broad foundation of language understanding, enabling them to excel at tasks requiring common-sense knowledge. To acquire task-specific knowledge, LLMs can be further fine-tuned on specialized tasks, thereby adapting them for diverse applications such as interactive agents (Touvron et al., ; Jiang et al., ), code generation (Luo et al., ; Lozhkov et al., ), and mathematical problem solving (Luo et al., ; Lozhkov et al., ), showing the remarkable versatility of LLMs. Nevertheless, given that no single model can master all tasks simultaneously, serving multiple LLMs, each precisely tailored for specific tasks, becomes essential for good performance.

Figure: Figure 1: An illustration of the input channel-wise maximum and minimum values for the delta weights of Speechless-Code-Mistral-7B. The variability across input channels highlights that certain salient channels, irrespective of their magnitude, can cause significant quantization errors when quantized with ultra low-bitwidth, which underscores their critical role in preserving performance.
Refer to caption: extracted/5956088/figures/mistral_in_channel_min_max.png

However, serving multiple models poses several major challenges. First, even with a relatively small number of models, the storage demands are significant due to the extensive number of parameters each model contains. For example, three LLaMA-2-70B models would collectively require over 384GB of storage, calculated as 128GB per model times three. Second, the substantial memory requirements of these models may make it impractical to load all of them into GPU memory simultaneously. While dynamically swapping model weights in and out of GPU memory as needed is feasible, the large size of the models makes this process slow and inefficient, significantly delaying response times and adversely affecting user experience.

To address the above challenges, existing methods (Liu et al., ; Yao & Klimovic, ) decompose the weights of fine-tuned models into pre-trained weights and delta weights introduced during fine-tuning. While low-rank approximation (Hu et al., ) can compress these delta weights, it falls short for full fine-tuned models whose delta weights lack low-rank properties (Liu et al., ; Lialin et al., ; Hao et al., ). As a result, prior work has adopted per-tensor quantization (Liu et al., ) as an alternative, significantly reducing storage needs and facilitating efficient sharing of the base model’s storage across multiple models.
Nevertheless, this method ignores the
distinction
in delta weight values across different input and output channels, resulting in substantial information loss. To mitigate this issue,
output channel-wise quantization (Xiao et al., ; Wei et al., ; Liu et al., )
where each output channel is allocated its own learnable step size, still
neglects input channel variations, as shown in Figure 1.
To further mitigate information loss, rescaling input channels before quantization can be employed, but this provides only limited alleviation at extremely low bitwidths.
In addition, existing methods assume that the appropriate model to utilize in response to a user request is known in advance. In reality, dynamically selecting the optimal model is essential for effectively handling diverse queries.
This gap highlights the need for an intelligent model selection mechanism that can identify the most suitable model based on specific user needs.

In this paper, we propose ME-Switch, a memory-efficient expert switching framework tailored for LLMs. To reduce the quantization error while simultaneously reducing storage needs,
we develop a mixed-precision quantization method that quantizes non-salient input channels of delta weights to extremely low bits while preserving those salient ones, which could substantially increase quantization errors when quantized at very low bitwidths, in full precision.
Since the number of salient input channels is relatively small, incorporating a limited amount of high-precision delta weights incurs negligible memory overhead and inference cost. To identify the important input channels of delta weights, one may select based on their magnitudes (Dettmers et al., ), as shown in Figure 2(a), which however fails to capture those leading to high quantization error. In contrast, our approach identifies salient input channels of delta weights based on their impact on reconstruction errors in the output activations, as illustrated in Figure 2(b).
Additionally, we develop a routing method that selects the most suitable LLM in response to a user request, enabling efficient and accurate model switching. To simplify the routing problem, we assume that each LLM specializes in a distinct domain, such as code generation or mathematics reasoning. This allows us to treat model selection as a multiple-choice question-answering task where each option represents a target domain, effectively transforming the problem into a domain classification task, as shown in Figure 3. Notably, dialogue LLMs exhibit strong instruction-following capabilities. Leveraging this, we propose using a small pre-trained dialogue LLM as a model-level router to solve the multiple-choice question-answering task. We then construct a multiple-choice question-answering dataset and fine-tune the router to improve its performance. Given its smaller size compared to the specialized LLMs, the router operates efficiently without adding significant overhead.

Our contributions can be summarized as follows. 1) We introduce ME-Switch, a memory-efficient framework designed for serving multiple LLMs. It only stores a single full-precision pre-trained model and dynamically loads the appropriate compressed delta model weights in response to user queries.
2) We develop a mixed-precision quantization method that significantly reduces the storage demands of serving multiple LLMs while maintaining performance, which is achieved by selectively quantizing the non-salient input channels of delta weights and leaving the salient ones unchanged. We further introduce a routing method that dynamically selects the most appropriate LLM for a given query, which is accomplished by transforming the model selection problem into a domain classification task and then solving it with a small LLM.
3) We conduct extensive experiments demonstrating the promising memory efficiency and routing performance of ME-Switch. Remarkably, when serving three models from the Mistral-7B family, ME-Switch not only delivers near-lossless performance on instruction, mathematical reasoning, and code generation tasks but also reduces the model size by $1.74\times$. More impressively, our method is able to serves up to 16 Mistral-7B models on a single NVIDIA A100 GPU without running out of memory.

## 2 Related Work

Efficient LLM serving.
LLMs can be efficiently deployed on GPUs for high-throughput serving using several inference frameworks, such as vLLM (Kwon et al., ) and Orca (Yu et al., ). Given that a single LLM cannot excel across all domains, it is crucial to serve multiple LLMs simultaneously to handle diverse user queries effectively.
To determine which model to use during inference, Zooter (Lu et al., ) utilizes a task-level router to load different LLMs based on the incoming task requirements.
This approach, however, introduces significant memory challenges, as hosting all LLMs on a GPU simultaneously can excessively strain the available VRAM. One solution is to serve multiple Low-Rank Adaptation (LoRA) modules within a multi-tenant serving system, such as Punica (Chen et al., ) and S-LoRA (Sheng et al., ). However, these methods fail to accommodate full fine-tuned models because such models typically do not display the low-rank characteristics necessary for LoRA approximation.
Another viable strategy to address the above issue involves dynamically loading different LLMs from CPU memory to GPU memory as needed, thereby reducing peak GPU memory utilization. Nevertheless, this dynamic loading competes for GPU memory bandwidth with model-swapping operations, presenting a significant bottleneck. To mitigate these challenges, techniques like Deltazip (Yao & Klimovic, ) and BitDelta (Liu et al., ) have been developed to compress the delta parameters. These methods allow task-specific delta parameters to be loaded into GPU memory on-demand, ensuring that only a single pre-trained model’s parameters reside permanently in GPU memory. This approach aims to achieve low-latency inference while minimizing the costs associated with maintaining numerous fine-tuned models in GPU memory. Our approach stands out by serving multiple LLMs with a minimal memory footprint with nearly lossless performance, which is achieved through an advanced delta weights quantization strategy and an adaptive model-level router.

Delta compression.
In recent years, numerous approaches have focused on reducing the storage overhead for maintaining different task-specific models through delta parameter compression. Model merging strategies often incorporate multiple tasks’ delta parameters (Ding et al., ) into the pretrained parameters to minimize the number of parameters needed for multi-task operations.
To address the parameter conflicts that often arise during model merging, Yu et al. (Yu et al., ) proposed a method involving massive unstructured random pruning (achieving 90% sparsity) of delta parameters and subsequent rescaling. This approach ensures that the accuracy of downstream tasks is not compromised. In a similar vein, Ties-merging (Yadav et al., ) developed a pruning strategy based on the magnitude and sign of delta parameters to further reduce conflicts and storage needs.
Additionally, the LoRA-based PEFT methods (Hu et al., ; Valipour et al., ; Ping et al., ) introduce a novel approach by learning one or several low-rank matrices to represent the delta parameters. ZipLoRA (Shah et al., ) explores the sparsity within these low-rank matrices, allowing the fusion of low-rank matrices from different tasks to further reduce the spatial overhead of multi-task models. However, these methods struggle to accurately approximate delta parameters in full fine-tuned models, as these models usually lack low-rank properties (Liu et al., ; Lialin et al., ; Hao et al., ).
Conversely, some studies focus solely on compressing delta parameters for each task without merging, thereby mitigating performance degradation across multiple tasks. For example, Ryu et al. (Ryu et al., ) combine quantization and low-rank estimation techniques to reduce the storage size of delta parameters. Liu et al. (Liu et al., ) push this further by quantizing each task’s delta parameters to 1 bit, achieving more than a tenfold compression. However, these quantization methods often lack robust handling of outliers, resulting in performance declines compared to uncompressed delta parameters.
Moreover, models that do not integrate multi-task delta parameters require manual activation of specific delta parameters for each task, which reduces the model’s applicability in multi-task environments.

## 3 Preliminaries

For simplicity, we employ uniform quantization (Jacob et al., ) to compress the models. Given a matrix ${\bf X}$ with floating-point values (*e.g.*, FP16), the quantization process can be expressed as:

$$ $\begin{array}[]{ll}\hat{{\bf X}}=\mathrm{quant}({\bf X})=\mathrm{clamp}\left( \lfloor\frac{{\bf X}}{s}\rceil,-Q_{N},Q_{P}\right)\times s,\end{array}$ (1) $$

where the function $\mathrm{clamp}({\bf V},{\bf V}_{\mathrm{min}},{\bf V}_{\mathrm{max}})$ clamps all elements in ${\bf V}$ within the range $[{\bf V}_{\mathrm{min}},{\bf V}_{\mathrm{max}}]$, the operator $\lfloor\cdot\rceil$ rounds a given value to the nearest integer, and $s$ is a learnable quantization step size initialized by $\max(|{\bf X}|)/(2^{b-1}-1)$. Here, $Q_{N}$ and $Q_{P}$ denote the number of negative and positive quantization levels, respectively. For $b$-bit quantized weights, $Q_{N}$ and $Q_{P}$ are set to $2^{b-1}$ and $2^{b-1}-1$, respectively.
Since the rounding function is not differentiable, we use the straight-through estimator (STE) (Esser et al., ) for gradient approximation following (Esser et al., ). For binary quantization, we use the quantization method following (Rastegari et al., ).

Recent studies (Liu et al., ; Yao & Klimovic, ) have shown that the weights of a fine-tuned model can be decomposed into the weights of the pre-trained model and the delta weights introduced during fine-tuning. Let ${\bf W}\in\mathbb{R}^{m\times n}$ and ${\bf W}_{\mathrm{FT}}\in\mathbb{R}^{m\times n}$ be the weight matrices of the pre-trained model and the fine-tuned model, respectively, where $m$ represents the number of input channels and $n$ denotes the output number of channels. The delta weight is defined as $\Delta={\bf W}_{\mathrm{FT}}-{\bf W}$. To reduce storage requirements, one can perform quantization using Eq. (1) to compress $\Delta$.
However, this method often results in significant performance degradation because it assumes all delta weight channels are equally sensitive to quantization noise.
Additionally, existing methods assume the appropriate model for a user request is predetermined. In practice, dynamically choosing the most suitable model based on the specific nature of each user’s query is more effective.

## 4 Proposed Method

In this section, we propose ME-Switch, a memory-efficient expert switching framework designed to optimize the deployment of multiple LLMs. ME-Switch reduces memory demands while maintaining performance by using mixed-precision quantization, which quantizes non-salient weights to extremely low bitwidths while keeping salient weights intact. Additionally, ME-Switch introduces model-level routing to efficiently handle task-specific queries. We will detail ME-Switch in the following sections.

### 4.1 Salient-Aware Delta Compression

In this section, we introduce our salient-aware delta compression approach, which specifically targets the preservation of salient input channels during quantization.
While quantization methods with learnable step sizes for each output channel can handle variations in output channels effectively (Xiao et al., ; Liu et al., ), variations in input channels pose a significant challenge to maintaining model performance, as shown in Figure 1.
Some salient input channels of delta weights, regardless of their magnitude, can cause substantial information loss and degrade model performance when quantized to a very low bitwidth. To address this, rescaling the input channels of delta weights before quantization (Lin et al., ) offers a solution to mitigate some of the quantization error, which however provides only limited alleviation under the context of extremely low-bit quantization.

Figure: Figure 2: An illustration comparison between the magnitude-based selection of salient delta weights and our reconstruction-error-based selection method. Given a delta weight matrix $\Delta\in\mathbb{R}^{m\times n}$, its quantized version $\hat{\Delta}$, and input ${\bf x}\in\mathbb{R}^{m}$, where $m$ and $n$ denote the number of input and output channels, respectively, our method measures the importance of each input delta channel by $\sum_{j=1}^{n}\|{\bf x}_{i}\Delta_{ij}-{\bf x}_{i}\hat{\Delta}_{ij}\|_{2}^{2}$.
Refer to caption: x1.png

Salient-aware delta selection. To protect salient input channels of delta weights, we develop a mixed-precision quantization method, where the majority of input channels of delta weights are quantized to low-bit precision, while only a small number of critical input channels are represented in their original precision.
This approach achieves significant reductions in storage requirements with minimal performance loss. The remaining challenge lies in identifying these salient input channels. A naive approach might involve selecting the input channels with large magnitudes as the salient delta weights (Dettmers et al., ), as shown in Figure 2(a). However, this method neglects the influence of input activations on the outputs, failing to identify channels that lead to high quantization error. To address this, inspired by the pruning metric (Sun et al., ), we introduce a salient delta weights selection metric based on reconstruction errors in the outputs, considering both weights and input activations, as shown in Figure 2(b). Specifically, given an input ${\bf x}\in\mathbb{R}^{m}$, the reconstruction error for the input channel $i$ can be calculated by

$$ $\begin{array}[]{ll}\sum_{j=1}^{n}\|{\bf x}_{i}\Delta_{ij}-{\bf x}_{i}\hat{ \Delta}_{ij}\|_{2}^{2},\end{array}$ (2) $$

where $\hat{\Delta}$ is the quantized delta weights and $n$ denotes the output channel number. With the reconstruction error defined, we choose those input channels with the top-$k$ largest reconstruction errors as the salient ones and retain them in 16-bit representation to maintain performance.

Efficient distillation. The quantization step size in Eq. (1) plays an important role in the final performance. To learn the quantization step size, we employ knowledge distillation to guide the alignment of the output logits of the quantized model with those of the full-precision fine-tuned model following (Liu et al., ). To reduce training overhead, we freeze the delta weights and focus solely on optimizing the quantization step size ${\bf s}$ using a small calibration dataset $\mathcal{X}$ by solving

$$ $\begin{array}[]{ll}\arg\min_{{\bf s}}\|f({\mathcal{X}})-\hat{f}(\mathcal{X};{ \bf s})\|_{2}^{2},\end{array}$ (3) $$

where $f(\cdot)$ and $\hat{f}(\cdot)$ denote the output logits of the fine-tuned and quantized models, respectively. Thanks to the reduced number of trainable parameters, this training process is highly efficient.

On-demand swapping. The extremely low-bit compression for delta weights significantly reduces the model size, alleviating storage demands. This approach enables us to maintain a single pre-trained model while storing multiple sets of compressed delta weights, facilitating efficient on-demand swapping. In this scenario, the pre-trained model remains in GPU memory, and the corresponding compressed delta weights are loaded dynamically based on the user query. Compared with directly serving multiple models, our method is more GPU memory-efficient. Since LLM decoding is memory-bound (Liu et al., ; Sheng et al., ) due to the auto-regressive nature, reducing model size effectively decreases the parameter loading time, thereby improving decoding latency. To achieve fast inference, we decouple the matrix multiplication during inference into two components:

$$ $\begin{array}[]{ll}{\bf y}={\bf x}{\bf W}_{\mathrm{FT}}={\bf x}({\bf W}+\Delta )\approx{\bf x}{\bf W}+{\bf x}\tilde{\Delta},\end{array}$ (4) $$

where $\tilde{\Delta}$ represents the compressed delta weights, including both the quantized unsalient and the full-precision salient delta weights. For ${\bf x}{\bf W}$, the computation is performed using a FP16 batched GEMM kernel. For ${\bf x}\tilde{\Delta}$, we implement an efficient Triton kernel (Tillet et al., ) that fuses dequantization and matrix multiplication for efficient computation. The reduced memory footprint of the compressed delta weights enables our method to perform batched forward passes across multiple models simultaneously. This batching at the model level significantly enhances efficiency compared to the traditional approach, which processes each model individually—especially beneficial when serving multiple models (See Figure 8).

### 4.2 Model-level Routing

Given a user query,
we present a method to determine the appropriate model. Consider a set of LLMs represented as $\mathcal{F}=\{f_{1},f_{2},\cdots,f_{M}\}$, where $M$ denotes the number of models. Given a user query $q$, we aim to find the most suitable LLM by solving the following problem $\arg\max_{f\in\mathcal{F}}P(q,f(q))$, where $P$ is a function that measures the quality or performance of the LLM response. To simplify the routing process, we assume that each LLM in $\mathcal{F}$ specializes in distinct domains such as code generation or mathematical problem solving.
This setup allows us to treat the routing challenge as a multiple-choice question-answering task, where each option corresponds to a specific domain, thereby transforming the problem into a domain classification problem. Note that dialogue LLMs like Qwen1.5-1.8B-Chat (Bai et al., ) exhibit capabilities in following instructions, which inspires us to utilize a small pre-trained LLM as a model-level router. As illustrated in Figure 3, we first prompt the router with the user’s question using a template designed to elicit domain classification. For the prompt template, please refer to Section A of the appendix. Based on the router’s response, we then dynamically load the corresponding compressed delta weights for the selected domain-specific model, such as a mathematical model, to generate outputs.

Figure: Figure 3: An illustration of the model-level routing. We first prompt the model-level router with the user query using a template (See Section A for more details) that presents a list of potential domains. The router then assesses these options and selects the most relevant domain by answering a multiple-choice question, effectively classifying the query into the corresponding category.
Refer to caption: x2.png

Since the router is not explicitly trained for query domain classification, its initial routing performance may be suboptimal. To improve the routing performance, we construct a multiple-choice question-answering dataset tailored for our routing problem. We collect instruction-following data from various domains and insert the query into the prompt template as shown in Figure 3. The responses are constructed by considering the correct domain-specific model choice. We then fine-tune the router with our constructed dataset using supervised fine-tuning to further improve the routing accuracy.

### 4.3 Discussions

Comparisons with routing in Mixture of Experts (MoE). Besides model-level routing, another approach to handle diverse user queries efficiently is to construct a MoE using a set of pre-trained expert models. This can be achieved by integrating the feedforward layers from all pre-trained LLMs into a single MoE module at each attention-FFN block, and merging other layers, such as self-attention layers, by simply averaging their weights (Sukhbaatar et al., ). An additional gate network is introduced for each MoE module to perform token-level routing. However, this approach requires extensive fine-tuning of the entire network parameters and the gating network, as there is a significant gap between MoE experts and pre-trained LLMs. Notably, each expert in an MoE tends to become a generalist across all domains due to the load balancing loss, which encourages an even distribution of the workload among experts (Fedus et al., ; Jiang et al., ). This contrasts with pre-trained expert models, which are typically specialized for specific domains. Compared with MoE, our model-level routing has a significantly lower training cost, as we only need to train a router while keeping the expert models frozen.

Model size reduction analysis.
Let $\Psi$ be the model size of an FP16 pre-trained model. When storing $M$ FP16 models, the total model size is $M\Psi$. In contrast, using our method, we store a single base model, a model-level router and $M$ compressed delta models. The total model size is $\Psi+M\tilde{\Psi}+\Phi$, where $\tilde{\Psi}$ represents the size of the compressed delta model and $\Phi$ denotes the storage requirement for the router. Therefore, the compression ratio can be computed by $M\Psi/(\Psi+M\tilde{\Psi}+\Phi)$. For example, when serving 9 LLaMA-13B models, our method achieves a compression ratio of 3.63$\times$. Empirical studies on compression ratios for varying model numbers are shown in Figures 6 and 8.

## 5 Experiments

Candidate LLMs. We apply our ME-Switch to two model families, Mistral-7B (Jiang et al., ) and LLaMA-2-13B (Touvron et al., ). For the Mistral family, we include Dolphin-2.2.1-Mistral-7B (^1^11https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b) as the instruction expert, Speechless-Code-Mistral-7B (^2^22https://huggingface.co/uukuguy/speechless-code-mistral-7b-v1.0) as the code expert, and MetaMath-Mistral-7B (^3^33https://huggingface.co/meta-math/MetaMath-Mistral-7B) as the math expert. For the LLaMA-2-13B family, LLaMA-2-13B-Chat (Touvron et al., ) serves as the instruction expert, MetaMath-13B (^4^44https://huggingface.co/meta-math/MetaMath-13B-V1.0) as the math expert, and LLaMA2-Chinese-13B-Chat (^5^55https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat) as the Chinese expert. The above models are fine-tuned based on pre-trained backbones. We use Qwen1.5-1.8B-Chat (Bai et al., ) as the model-level router.

Training and testing datasets. We collect a diverse set of instruction samples from various open-source datasets, including Alpaca (Taori et al., ) for the instruction domain, MetaMathQA (Yu et al., ) for the mathematics domain, Code-74k-ShareGPT (^6^66https://huggingface.co/datasets/ajibawa-2023/Code-74k-ShareGPT) for the code domain, and Chinese Alpaca (^7^77https://huggingface.co/datasets/hfl/alpaca_zh_51k) for the Chinese domain. To measure the performance of the resulting LLMs, we report accuracy on several benchmarks across different domains: MMLU (Hendrycks et al., ) for the instruction, GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ) for the mathematics, HumanEval (Chen et al., ) and MBPP (Austin et al., ) for the code, and C-Eval (Huang et al., ) and C-MMLU (Li et al., ) for the Chinese. We use the WizardCoder toolbox to evaluate on HumanEval and MBPP, and the OpenCompass toolbox (Contributors, ) to evaluate on other datasets. For MMLU, we report accuracy based on 5-shot in-context learning. To determine the answer for each question, we assess the perplexity of various response options and select the one with the lowest perplexity. For GSM8K and MATH, we adopt a 4-shot Chain of Thought (CoT) methodology to obtain the final answer following (Wei et al., ). For the HumanEval and MBPP datasets, we employ a 0-shot configuration and generate answers using greedy decoding. We assess the functional correctness using the pass@1 metric following (Liu et al., ).

Implementation details. For salient-aware delta compression, we construct a calibration set from each domain-specific dataset and use these sets to compress the delta weights for each respective domain. Each calibration set consists of 1600 randomly sampled sequences, each with a length of 128 tokens.
The bitwidth $b$ and the number of FP16 input channels $k$ are set to 2 and 8, respectively. We use the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $10^{-5}$ and a mini-batch size of 4 for training over 1 epoch. Delta weights compression experiments for the Mistral family are conducted on two NVIDIA A100 80G GPUs, while for the LLaMA-2-13B model, we use four NVIDIA A100 80G GPUs.

For the model-level router training, we construct the training data using samples from various domains, as mentioned in Section 4.2. To balance the dataset, we extract an equal number of samples from each domain-specific dataset. This constructed dataset is used to fine-tune model-level router through supervised fine-tuning for 4 epochs on a machine with 8 $\times$ A100 GPUs. We use the AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.95$, setting the learning rate to $3\times 10^{-4}$ and applying a linear learning rate warmup. The weight decay is set to 0.01. We set the per-device mini-batch size to 8 and use gradient accumulation steps of 2.

**Table 1: Main results for Mistral-7B and LLaMA-13B families.**
| Model | MMLU (%) $\uparrow$ | Mathematical Reasoning (%) $\uparrow$ | Code Generation (%) $\uparrow$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STEM | Hums. | Social | Other | Avg. | GSM8K | Math | Avg. | HumanEval | MBPP | Avg. |  |
| Dolphin-2.2.1-Mistral-7B | 52.05 | 68.83 | 73.42 | 65.43 | 63.43 | 63.68 | 12.80 | 38.24 | 42.70 | 54.90 | 48.80 |
| MetaMath-Mistral-7B | 50.45 | 66.82 | 71.63 | 64.60 | 61.87 | 73.92 | 20.62 | 47.27 | 0.00 | 21.60 | 10.80 |
| Speechless-Code-Mistral-7B | 51.82 | 68.35 | 73.74 | 65.69 | 63.36 | 61.18 | 13.52 | 37.35 | 51.20 | 60.40 | 55.80 |
| ME-Switch w/o Routing | 53.17 | 69.09 | 73.88 | 65.40 | 63.95 | 73.62 | 20.48 | 47.05 | 51.80 | 60.70 | 56.25 |
| ME-Switch | 51.49 | 68.37 | 73.60 | 66.08 | 63.32 | 73.39 | 20.30 | 46.85 | 51.80 | 60.70 | 56.25 |
| Model | MMLU (%) $\uparrow$ | Mathematical Reasoning (%) $\uparrow$ | Chinese (%) $\uparrow$ |  |  |  |  |  |  |  |  |
| STEM | Hums. | Social | Other | Avg. | GSM8K | Math | Avg. | C-Eval | C-MMLU | Avg. |  |
| LLaMA-2-13B-Chat | 44.26 | 59.79 | 63.20 | 56.57 | 54.60 | 43.75 | 5.20 | 24.48 | 36.13 | 38.71 | 37.42 |
| MetaMath-13B | 37.81 | 52.77 | 56.00 | 50.05 | 47.84 | 69.14 | 8.48 | 38.81 | 33.62 | 32.70 | 33.16 |
| LLaMA2-Chinese-13B-Chat | 45.24 | 60.01 | 62.47 | 55.92 | 54.67 | 38.89 | 4.54 | 21.72 | 40.28 | 39.16 | 39.72 |
| ME-Switch w/o Routing | 44.57 | 60.87 | 64.00 | 58.04 | 55.45 | 70.05 | 13.20 | 41.63 | 40.13 | 39.91 | 40.02 |
| ME-Switch | 44.51 | 60.87 | 64.00 | 58.04 | 55.43 | 69.90 | 13.14 | 41.52 | 40.13 | 39.84 | 39.99 |

### 5.1 Main Results

To evaluate the efficacy of our proposed model, we apply ME-Switch to the Mistral-7B and LLaMA-13B model families. For results in terms of LLaMA-3-8B model family, please see Section B of the appendix. We include ME-Switch without model-level routing for comparisons.
The experimental results, detailed in Table 1, demonstrate that ME-Switch without routing, even with extremely compressed delta weights, achieves performance comparable to that of the respective unquantized expert models across various downstream tasks. For the Mistral-7B family, on MMLU, ME-Switch without routing
lags behind the math expert by just 0.22% in mathematical reasoning tasks.
Notably, ME-Switch without routing consistently outperforms the code expert in code generation tasks.
The performance improvements over uncompressed expert models are primarily attributed to additional training through efficient distillation, which improves the models’ task-specific performance by optimizing the quantization step size.
With model-level routing, ME-Switch nearly matches the performance of the ones without it, demonstrating its precise routing for user queries.

**Table 2: Performance comparisons between fixed-precision quantization and mixed-precision quantization for MetaMath-Mistral-7B and Speechless-Code-Mistral-7B.**
| Method | Model Size (GB) | Mathematical Reasoning (%) $\uparrow$ | Code Generation (%) $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GSM8K | Math | Avg. | HumanEval | MBPP | Avg. |  |  |
| Full-precision | 13.48 | 73.92 | 20.62 | 47.27 | 51.20 | 60.40 | 55.80 |
| \cdashline1-9<br>Fixed-precision | 2.11 | 73.31 | 20.44 | 46.88 | 47.00 | 59.10 | 53.05 |
| Mixed-precision | 2.13 | 73.62 | 20.48 | 47.05 | 51.80 | 60.70 | 56.25 |

### 5.2 Ablation Studies

Fixed-precision quantization *vs.* Mixed-precision quantization.
To validate the effect of mixed-precision quantization, we compress the delta weights of MetaMath-Mistral-7B and Speechless-Code-Mistral-7B using both fixed-precision quantization and our salient-aware mixed-precision quantization. We
evaluate their performance on mathematical reasoning and code generation tasks, respectively. The bitwidth $b$ and the number of FP16 input channels $k$ are set to default values as specified in the implementation details. The results in Table 3 indicate that despite retaining a minimal number of FP16 channels, the model size of the mixed-precision model (2.13 GB) is nearly identical compared to fixed-precision model (2.11 GB).
However, introducing a small number of FP channels significantly improves performance. For example, our compressed Speechless-Code-Mistral-7B, with a model size reduced by 6.33$\times$, even outperforms the full-precision counterpart by 0.45% in the average accuracy on code generation tasks. This underscores the capability of salient-aware mixed-precision quantization to minimize model size while preserving model performance.

Low-rank adaptation *vs.* our salient-aware delta compression. In addition to mixed-precision quantization, we can also employ low-rank adaptation (LoRA) to compress delta weights. Specifically, we decompose the delta weights as $\Delta=\mathbf{U}\mathbf{\Sigma}\mathbf{V}$ and approximate delta weights using low-rank approximation $\tilde{\Delta}=\mathbf{A}\mathbf{B}$ where $\mathbf{A}=\tilde{\mathbf{U}}\sqrt{\tilde{\mathbf{\Sigma}}}$ and $\mathbf{B}=\sqrt{\tilde{\mathbf{\Sigma}}}\tilde{\mathbf{V}}$. Subsequently, $\mathbf{A}$ and $\mathbf{B}$ are refined using our efficient distillation mentioned in Section 4.1. To compare the effectiveness of LoRA against mixed-precision quantization, we apply both methods to the delta weights of Dolphin-2.2.1-Mistral-7B and MetaMath-Mistral-7B and evaluate performance on instructional and mathematical reasoning tasks. For LoRA, we set the rank to 512.
The results are detailed in Table 5. Our approach with a much smaller model size outperforms LoRA, especially on Math. These results reveal that LoRA cannot accurately approximate delta weights for full fine-tuned models like Dolphin-2.2.1-Mistral-7B and MetaMath-Mistral-7B, given that their delta weights lack low-rank properties, as also shown in Figure A in the appendix.

Effect of different number of FP16 channels. To assess the impact of varying FP16 input channel counts $k$, we compress the delta weights of Speechless-Code-Mistral-7B and evaluate performance on code generation tasks. The bitwidth $b$ is set at 2. From Table 3, our method already achieves lossless performance with $k=8$. Increasing $k$ further yields no significant performance improvements, indicating that performance has plateaued. Therefore, we set $k$ to 8 by default.

**Table 4: Performance comparisons between LoRA and our salient-aware delta compression for Dolphin-2.2.1-Mistral-7B and MetaMath-Mistral-7B.**
| Model | Model Size (GB) | MMLU (%) $\uparrow$ | Mathematical Reasoning (%) $\uparrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STEM | Hums. | Social | Other | Avg. | GSM8K | Math | Avg. |  |  |
| LoRA | 2.99 | 52.16 | 68.38 | 73.15 | 65.52 | 63.33 | 73.62 | 17.30 | 45.46 |
| Ours | 2.11 | 53.17 | 69.09 | 73.88 | 65.40 | 63.95 | 73.62 | 20.48 | 47.05 |

Effect of different quantization bitwidths. To investigate the impact of varying bitwidths $b$, we compress the delta weights of Speechless-Code-Mistral-7B and evaluate the performance on a code generation task. Table 5 shows that increasing $b$ from 1 to 2 significantly improve performance, achieving lossless results. Given that further increasing $b$ lead to negligible performance differences due to saturation, we set $b$ to 2 as the default.

Figure: Figure 4: Average accuracy *vs.* delta weights size across different domains. “Baseline” refers to the fixed-precision quantization baseline. The dashed line indicates the full-precision counterpart.
Refer to caption: x3.png

Performance comparisons with other weight-only quantization methods.
To demonstrate the promising performance of our salient-aware delta compression, we include the following weight-only quantization methods: AWQ: we use AWQ (Lin et al., ) to rescale input channels of delta weights before quantization to mitigate quantization errors. Random: using our salient-aware delta compression, we randomly select some input channels from the delta weights as important channels. Wanda: leveraging our salient-aware delta compression, we select important input channels of delta weights using the pruning metric from Wanda (Sun et al., ). Magnitude: within our salient-aware delta compression framework, we select sensitive input channels of delta weights based on their weight magnitude, following the method proposed by (Dettmers et al., ). We also include fixed-precision quantization for comparisons. We applied all methods to compress the delta weights of Dolphin-2.2.1-Mistral-7B, MetaMath-Mistral-7B, and Speechless-Code-Mistral-7B, using bitwidths $b=1$ and $b=2$. The results are shown in Figure 4. The detailed number of different methods can be found at Section C of the appendix.
From the results, we observe that AWQ achieves comparable performance to the 1-bit baseline on code domain, highlighting the limitations of rescaling in extremely low-bitwidth quantization. In contrast, keeping the salient delta input channels performs favourably against the rescaling input channel counterpart. Moreover, our salient channel selection demonstrates superior performance than Random, Wanda, and Magnitude metrics across various bitwidths and tasks. For example, for $b=2$, our salient-aware delta compression outperforms Wanda by 1.0% on the average accuracy on code domain, underscoring the effectiveness and superiority of our approach in selecting the salient delta weights.

Figure: Figure 5: Effect of supervised fine-tuning (SFT) in model-level routing. We assess the performance of routing by measuring the accuracy on a 4-domain classification task (instruction, mathematics, code, and Chinese).
Refer to caption: x4.png

Effect of supervised fine-tuning in model-level routing.
To investigate the effect of supervised fine-tuning (SFT) on model-level routing, we evaluate the domain classification performance of the router (*i.e.*, Qwen1.5-1.8B-Chat) across four domains: instruction, mathematics, code, and Chinese. As shown in Figure 6, the pre-trained router performs poorly in domain classification without fine-tuning, achieving a Top-1 accuracy of only $5.80\%$ on C-Eval. However, with SFT, the router’s performance improves significantly, reaching nearly 100% accuracy across all domains. This demonstrates that supervised fine-tuning greatly enhances the instruction-following capabilities of the router, thereby improving its routing performance.

Figure: Figure 7: Model size reduction results in terms of LLaMA-13B family. The model sizes for a single FP16 model, a compressed model, and the router are 24.23 GB, 3.60 GB, and 3.42 GB, respectively.
Refer to caption: x6.png

Model size reduction analysis. To investigate the model size reduction as discussed in Section 4.2, we compare the total storage requirements of full-precision models with those of our compressed models for the Mistral-7B and LLaMA-13B families across varying model counts, as shown in Figures 6 and 8. As the number of models increases, the compression ratios improve substantially. For instance, with nine models, our method achieves a 3.63$\times$ reduction compared to full-precision models for Mistral-7B family. These savings become even more pronounced with larger model sizes, reaching up to a 3.63$\times$ reduction for LLaMA-13B family.

Latency analysis. To assess the latency improvements from delta weights compression, we measured the end-to-end decoding latency of the Mistral-7B model with an input sequence length of 128 on a single NVIDIA A100. Due to page limitations, additional latency decomposition results are provided in Section F of the appendix.
Decoding latency is critical, as it typically dominates processing time in LLM operations (Lin et al., ; Liu et al., ). Our efficient Triton kernel, which enables batched matrix multiplication between multiple compressed weight matrices and high-precision input activations, is compared against the conventional approach of individually processing multiple models.
Results depicted in Figure 8 illustrate that while our method may perform slightly slower than the naive approach for a small number of models due to additional dequantization overhead, it provides lower latency as the number of models is greater than 4. More importantly, unlike the naive approach, our method is able to simultaneously serve 16 models on GPUs without running into out-of-memory (OOM) issues, demonstrating better scalability and efficiency in high-load scenarios.

## 6 Conclusion and Future Work

In this paper, we have introduced ME-Switch, a memory-efficient expert switching framework designed for LLMs. Our method has addressed the critical challenge of balancing model performance with storage efficiency. The core of our ME-Switch lies in a novel mixed-precision quantization method that selectively compresses non-salient delta weights to extremely low-bit precision while preserving salient delta weights. Additionally, we have developed a model-level router that dynamically selects the most suitable LLM for a given query by transforming the model selection problem into a domain classification task. Extensive experiments on Mistral-7B and LLaMA-13B families have demonstrated that ME-Switch achieves performance comparable to unquantized expert models across various tasks while significantly reducing model size. In terms of limitations, our ME-Switch relies on accurate domain classification by the model-level router, which can sometimes misclassify and impact performance. Future work could explore more advanced or hybrid routing strategies, such as Mixture of Experts (MoE), to enhance accuracy. Additionally, quantizing the base model itself could further reduce the overall model size. This approach would require careful consideration of the combined effects of quantizing both the base model and the delta weights to ensure performance is maintained. Furthermore, reducing the bitwidth of the KV Cache could accelerate the decoding speed, offering additional efficiency improvements.

## Appendix A More Details about Prompt Template for Model-level Routing

In this section, we presents the prompt template for our model-level routing, as illustrated in Table A. When a user query is received, it is embedded into the prompt template to form a structured question. This structured question is then processed by the router to perform domain classification.

**Table A: Prompt template for model-level routing.**
| Prompt template for model-level routing |
| --- |
| Classify the query based on the required expertise. Route the query to the appropriate model for a precise response. Only output the letter corresponding to the best category (A, B, C, …, F).<br>Query: {Insert the user’s query here. }<br>Options: A) Instruct - For general guidance, explanations, or broad advice. B) Code - For programming-related queries, like debugging or coding. C) Math - For mathematical inquiries, such as problems or theories. D) Chinese Language Expert - For inquiries related to the Chinese language, including translation, grammar, and usage. … F) {Specify additional categories and their descriptions here.}<br>Response should be only ’A’, ’B’, ‘C’, … or ‘F”, with no additional text. |

## Appendix B More results on LLaMA-3-8B model family

We also applied our method to the LLaMA-3-8B family using the same experimental setup. We employ LLaMA-3-8B-Instruct as the instruction expert and LLaMA-8B-Chinese-Chat as the Chinese expert. The results, presented in Table B, demonstrate that without model-level routing, our ME-Switch achieves nearly lossless performance compared to the unquantized expert models across various downstream tasks, underscoring the superior performance of our salient-aware delta compression. When model-level routing is integrated, performance remains nearly unchanged, highlighting its precise handling of user queries.

**Table B: Main results for LLaMA-3-8B family. “FP Baseline” refers to the performance metrics of experts without compression.**
| Method | MMLU (%) $\uparrow$ | Chinese (%) $\uparrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STEM | Hums. | Social | Other | Avg. | C-Eval | C-MMLU | Avg. |  |
| FP Baseline | 57.30 | 71.64 | 77.83 | 71.13 | 68.05 | 51.99 | 52.25 | 52.12 |
| \cdashline1-9<br>ME-Switch w/o Routing | 57.12 | 70.89 | 78.60 | 70.92 | 67.93 | 52.67 | 52.70 | 52.69 |
| ME-Switch | 57.16 | 70.44 | 78.53 | 71.25 | 67.90 | 52.67 | 52.70 | 52.69 |

## Appendix C More performance comparisons with different weight-only quantization methods

We present the detailed results of Figure 4 in Table C. A comprehensive analysis is available in Section 5.2.

Figure: Figure A: An illustration showing the cumulative energy of delta weights for MetaMath-Mistral-7B model derived through Singular Value Decomposition (SVD).
Refer to caption: x8.png

**Table C: Performance comparisons with different weight-only quantization methods.**
| Domain | Dataset | FP | 1-bit | AWQ | Random | Wanda | Magnitude | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Instruct (%) $\uparrow$ | MMLU | 63.43 | 63.14 | 63.32 | 63.19 | 63.19 | 63.04 | 63.43 |
| Math (%) $\uparrow$ | GSM8K | 73.92 | 53.45 | 53.75 | 54.66 | 58.07 | 54.89 | 59.14 |
| Math | 20.62 | 1.50 | 1.72 | 1.82 | 2.14 | 1.64 | 1.74 |  |
| Avg. | 47.27 | 27.48 | 27.74 | 28.24 | 30.11 | 28.27 | 30.44 |  |
| Code (%) $\uparrow$ | HumanEval | 51.20 | 47.00 | 47.00 | 46.30 | 48.20 | 48.20 | 47.60 |
| MBPP | 60.40 | 58.40 | 58.10 | 58.60 | 58.60 | 58.90 | 60.70 |  |
| Avg. | 55.80 | 52.70 | 52.55 | 52.45 | 53.40 | 53.55 | 54.15 |  |
| Domain | Dataset | FP | 2-bit | AWQ | Random | Wanda | Magnitude | Ours |
| Instruct (%) $\uparrow$ | MMLU | 63.43 | 63.72 | 63.65 | 63.71 | 63.90 | 63.94 | 63.95 |
| Math (%) $\uparrow$ | GSM8K | 73.92 | 73.31 | 73.24 | 73.01 | 72.71 | 73.16 | 73.62 |
| Math | 20.62 | 20.44 | 19.98 | 20.52 | 20.64 | 20.08 | 20.48 |  |
| Avg. | 47.27 | 46.88 | 46.61 | 46.77 | 46.68 | 46.62 | 47.05 |  |
| Code (%) $\uparrow$ | HumanEval | 51.20 | 47.00 | 47.00 | 48.80 | 50.60 | 50.00 | 51.80 |
| MBPP | 60.40 | 59.10 | 60.20 | 59.60 | 59.90 | 60.20 | 60.70 |  |
| Avg. | 55.80 | 53.05 | 53.60 | 54.20 | 55.25 | 55.10 | 56.25 |  |

## Appendix D More analysis on the rank of delta weights

We show the cumulative energy of delta weights for MetaMath-Mistral-7B in Figure A, using squared singular values to measure the “energy” of the projection matrix. The results show that all projection layers consistently exhibit a similar trend and possess a relatively high rank. Therefore, due to the absence of low-rank properties, LoRA cannot accurately approximate the delta weights of full fine-tuned models.

## Appendix E BERT vs. small LLM for model-level routing

In addition to smaller LLMs, we can also use BERT for domain classification given user queries. Specifically, we employ DistillBERT (Sanh, ) as the backbone and fine-tune it on our collected dataset (described in Section 4.2) with just the queries and corresponding domain labels. We show the router accuracy in Table D and the latency comparisons in Table E. From the results, DistillBERT with a faster response time performs well across most datasets, except for MMLU where its limited capacity struggles with complex data. In contrast, our method consistently achieves good performance across all datasets, demonstrating its effectiveness even in complex scenarios. Moreover, our router’s inference latency is just 17.60ms for sequence lengths of 128, which is less than 7% of the inference time for expert models. Therefore, we continue to leverage Qwen1.5-1.8B-Chat for its proven effectiveness in challenging scenarios.

**Table D: Router performance comparisons.**
| Model | MMLU | GSM8K | Math | HumanEval | MBPP | C-Eval | C-MMLU |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DistillBERT | 73.29 | 100.00 | 99.80 | 96.34 | 100.00 | 99.97 | 100.00 |
| Qwen1.5-1.8B-Chat | 99.73 | 99.92 | 99.70 | 100.00 | 100.00 | 100.00 | 99.91 |

**Table E: Router latency (ms) comparisons.**
| Sequence Length | 128 | 256 | 512 |
| --- | --- | --- | --- |
| DistillBERT | 3.70 | 3.80 | 4.90 |
| Qwen1.5-1.8B-Chat | 17.60 | 18.30 | 19.10 |

## Appendix F More results on latency decomposition

We provide a detailed breakdown of decoding times for Mistral-7B model in Figure B, using an input sequence length of 128 on a single NVIDIA A100 GPU. Since loading delta weights, dequantization, and matrix multiplication are integrated into our efficient Triton kernel, it is challenging to isolate and measure the latency of each individual component separately. Therefore, we measure the inference time for the operations $\mathbf{x}\mathbf{W}$ and $\mathbf{x}\tilde{\Delta}$. Notably, as the number of models increases, the increased latency attributed to $\mathbf{x}\tilde{\Delta}$ exceeds that of $\mathbf{x}\mathbf{W}$, primarily due to the augmented loading time of the more compressed delta weights.

Figure: Figure B: Latency decomposition of our method for Mistral-7B on a single NVIDIA A100 80G GPU.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Introduction — the authors might consider reframing motivations around MoE as well and listing memory pressure as a primary motivator for LLM task specialization.
- Line 78 the "information loss" that occurs with subsequent methods can be further specified. Is this something that affects downstream accuracy, or does this refer to something else?
- Why doesn’t reconstruction loss merely end up choosing values which are closest to values which are fp16 quantized, i.e. representable with low approximation error? More analysis of reconstruction loss would be helpful to prove efficacy.
- Why is fp16 used throughout the experiments setups rather than bf16, which is supported on A100s (used in the paper) and is far more common in inference settings?
- What is the effect of using different backbones for the same task with the proposed approach?
- What additional details can be shared about the Triton kernel? What is the performance benefit of its inclusion? How is the implementation structured?
soundness: 2
strengths: - The application of reconstruction-based quantization to input model weights is novel insofar as other methods (low rank approximation, uniform quantization) are not as effective.
- Results show minor to negligible losses in accuracy across many tasks, including some gains, though the degree to which these are due to noise is not immediately clear.
- The reduction in memory usage as compared to other baselines is somewhat compelling, although compared to baselines without any delta-based approaches, which the authors did not contribute.
summary: The authors introduce ME-Switch, a memory-efficient framework for serving MoE-based LLMs. Given a set of MoE models across specified domains, they present a quantization technique for deltas from the base model which they argue is better than other approaches per its retention of accuracy and effectiveness in reducing model size. The approach attempts to estimate the saliency of particular weight deltas for quantization based on a reconstruction loss-based scheme As part of the assumption that models are domain-specific, the authors train a routing model to delegate user inputs to a domain-specific model.

The authors showcase retention of performance task performance across various domains in conjunction with a reduction in GPU memory usage attributed to efficient quantization.
weaknesses: - The assumption that each LLM specializes in a distinct domain is a large one, and limits the general applicability of the authors' approach. Indeed, in many MoE-based settings, model diversity is sufficient to improve performance significantly, and task specialization is not even considered. The given assumption also requires training a domain-based router, which itself requires ablations. The authors need to argue that this setting is representative and useful.
- There are missing baselines. The sensitivity and quality of the domain-based routing setup as compared to a baseline without adaptive quantization makes it difficult to disambiguate the effect of improved quantization alone.
  - Similarly, there is no baseline that foregoes SFT or one that compares SFT without quantization or task routing. These components should be independently ablated.
- Results are defined only on a very narrow set of task domains. This makes it quite unclear how the proposed approach generalizes. Domain-specific expert setups in a general setting might involve dozens of models in a large-scale system.
- In general, the paper’s writing is redundant, i.e. the core premises, contributions, and prior work are presented multiple times each, with similar levels of depth. The authors could make room to do more experiments by removing redundant components.
- The approach with which the router is trained generalizes very poorly. A router must be trained based on a collection domain-specific models for every single setup. A router trained on a specific domain-specific set of multiple choice questions is quite specific to the input tasks.
- There is no presentation of an accuracy-memory tradeoff across approaches. This, combined with weak baselines reduces the strength of the contribution.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. I am a little bit confused by how the router is related to the first contribution -- 
it would be great if the authors could elaborate more? these two parts look quite
orthogonal to me.

2. There are a lot of previous work (especially for the router) tackling similar problems,
it seems that the authors only compared with the most natural baseline, but not
more advanced methods. it would be great if the authors can justify it or provide more
results.
soundness: 3
strengths: - The proposed idea makes sense, and seems to work better than the baseline.
summary: In this paper, the authors focus on the problem of serving multiple models (fine-tuned over
the same base) together. The challenge is that each model occupies a lot of memory and 
the authors build on the idea of delta compression between the FT'ed models and the base.

Specifically, the authors proposed a mixed precision compression technique for the delta.

Moreover, the authors proposed a routing-based method to pick the right model to use
given user input.
weaknesses: - It would be great if both contributions can be evaluated more thoroughly against previous methods
- The second contribution could be put better into context

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Will the code for this be open source?
2. Are there any insights on why mixed precision quantization outperforms the full precision models in certain settings?
3. Is there an estimate of how often in practice is the appropriate model not know in advance ? Is this more common than the case where it is known?
soundness: 3
strengths: 1. This paper is generally well written and easy to follow, and the problem is well-motivated.
2. Experiments are comprehensive with convincing results and there are sufficient ablations to justify different design choices.
3. The problem of serving multiple finetuned models is a very poignant problem in this era of LLMs.
summary: This paper addresses the problem of serving multiple expert models i.e multiple models finetuned for different tasks from the same base model. Serving all fully finetuned models is expensive since they take up memory and switching these in and out of memory can be slow. Existing methods use low-rank finetuning to reduce the storage required for finetuned models, however low-rank finetuning does not match full finetuning in quality hence recent methods instead quantize the delta between the finetuned model and the base model. However, these quantization methods can lead to significant quantization errors at low bitwidths. This paper proposes a saliency aware method for quantizing delta between the finetuned and base model.  In addition, this paper suggests a dynamical routing method to determine which expert model to route user requests to when the appropriate model for a user request is not known in advance.
weaknesses: Major
1.  How does the saliency based delta quantization technique in this paper differ from other non-magnitude based saliency quantization techniques like https://arxiv.org/pdf/2405.14917 ?
2. Model routing is a well studied problem! Are there other existing approaches that study model routing in the same scenario where the appropriate model is not known? How is your solution different or better than them ?

Minor
1. Related work like GPT-Zip https://openreview.net/pdf?id=hO0c2tG2xL is not cited
2. On line 46, the paper claims that no single model can master all tasks simultaneously. While this is not currently the state of affairs is there a formal proof or evidence that this is impossible?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - The author highlights the advantages of ME-Switch over MoE, such as reduced fine-tuning work and higher efficiency. However, it would be beneficial to include a more in-depth discussion of its limitations compared to MoE models. This would provide readers with a deeper understanding of the nature of model-level routing, particularly in addressing challenges like handling overlapping expert domains and managing a larger number of models when adapting to new experts, areas where MoE models may offer distinct advantages. Notably, the latency advantage of ME-Switch is fully realized when the number of models increases.

- Latency analysis: The discussion of latency mentions that the naive implementation has a slight advantage when the number of models is fewer than 4, as shown in Figure 8, and the author attributes this to the quantization overhead. However, why does the quantization overhead stop being the dominant factor as the number of models increases? What is the trend in the profiling of quantization overhead as the number of models continues to increase? A deeper analysis of this would be valuable.
soundness: 3
strengths: - The author proposes an innovative and effective framework for improving the storage efficiency of serving multiple LLM experts.

    - This framework leverages mixed-precision quantization for delta weights, intelligently selecting non-salient channels based on quantization errors. The strategy of maintaining performance by selecting the top-k salient channels as non-quantized has been proven effective, as the accuracy surpasses baseline models across multiple domains.

    - The framework also introduces model-level routing by fine-tuning a small LLM router on domain-specific datasets, transforming the routing problem into a simpler domain classification task. This simplifies the problem and making the finetuning of small LLM router more lightweight compared to finetuning the MoE models

    - The framework implements on-demand swapping, enhancing GPU memory efficiency by loading only the quantized delta weights onto the GPU. This optimization is particularly effective for switching between large LLM experts. Combined with model-level routing and a specialized routing kernel, this approach significantly improves decoding efficiency. As demonstrated in the paper, it enables hosting up to 16 models on a single GPU, whereas the naive approach encounters out-of-memory (OOM) issues after just 4 models.


- A detailed ablation study is presented to demonstrate the effectiveness of mixed-precision quantization compared to baseline approaches across multiple domains. These experiments significantly strengthen the framework's methods. For example, the approach preserves accuracy competitively when compared to fixed-precision and low-rank adaptation methods. Furthermore, the ablation study clearly illustrates the selection of parameters, such as the number of non-quantized channels (k) and quantization bits (b). Lastly, the performance comparison with other weight quantization methods like AWQ, Random, Wanda, and Magnitude supports the intuition that selecting non-salient input channels yields the best performance.


- The clarity of writing, including the structure, figures and tables, is good. The author explains the background context and methodology in a good manner.

- The paper also discusses other important aspects, such as latency, with a comparison to MoE.
    - Model-level routing offers the advantage of requiring less intensive training compared to MoE.
    - The author employs a Triton kernel to further reduce latency.
summary: The author contributed a memory-efficient expert-switching framework for LLMs, called ME-Switch. The framework is designed to address the challenge of LLMs' storage efficiency when serving multiple large expert models, while maintaining performance through model-level router fine-tuning and efficient distillation fine-tuning.

The proposed method applies mixed-precision quantization for delta weights to preserve model performance and enhance efficiency. This is achieved by selecting non-salient input channels based on quantization errors. Additionally, the approach incorporates a model-level router, implemented through supervised fine-tuning of a relatively small LLM, to switch between expert domain-quantized LLMs according to the user's inquiry. A detailed ablation study is included to further validate the approach.
weaknesses: - The model-level router relies heavily on domain classification, which requires fine-tuning the router model on domain-specific datasets. This approach may have several disadvantages:
    - The author conducted research on at most four distinct domains for the router: mathematics, code, Chinese, and instruction. Although the router demonstrates strong classification ability for these domains, it would be worthwhile to explore its performance across more domains, particularly those with potential overlap. For example, Text Classification Experts and Sentiment Analysis Experts, or Mathematics and Physics Experts, Computer Science Experts and Data Science Experts. In such scenarios, MoE could leverage the combined knowledge of multiple experts, as it is fine-tuned based on a token-level router. A potential consequence is that, when adding a large number of experts to the MoE, fine-tuning may converge faster due to shared knowledge and parameters across domain experts. In contrast, a model-level router may struggle to distinguish between overlapping domain experts and always rely on a single expert, potentially causing overall performance degradation.
    - This approach requires the construction of a domain classification dataset, which can be inconvenient. For instance, when adding a new domain expert, the router would need to be re-trained on an updated domain dataset.
    - While the approach demonstrates effectiveness compared to other quantization methods in ablation studies, the experiment results do not include a comparison with an end-to-end token-level router approach, such as MoE baselines.

- Latency analysis: The comparison only covers up to 4 models, after which the naive implementation encounters out-of-memory (OOM) issues. It is a little subtle to conclude that latency is lower than naive model as the number of models is greater than 4.

Meta-review: This paper tackles the growing challenge of serving multiple fine-tuned LLMs while keeping memory usage in check. ME-Switch’s approach—using salient-aware delta compression to carefully quantize non-salient channels and preserve the important ones—feels genuinely promising. The experiments are generally well-conducted, and the authors have made a solid effort to broaden their evaluations into medical and legal domains, which is some what encouraging.

Still, certain issues remain. The scope of evaluation could be expanded, and a deeper comparison with Mixture-of-Experts (MoE) methods would better contextualize the work’s impact. While the authors’ response did offer improvements, including moving the router details to the appendix and clarfying latency considerations, some core questions about generalization and scalability remain unanswered.

Given the borderline average score, the paper’s contribution seems strong but not quite ready for acceptance. I would urge the authors to strengthen the breadth of their evaluations, incorporate MoE comparisons, and provide a clearer narrative around how these techniques can scale. With these changes, I believe this work could become a valuable contribution to the community.

Decision: Reject