Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters
Authors: Yifei Xia
Fangcheng Fu
Wentao Zhang
Jiawei Jiang
Bin CUI
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-LoRA serving system; LLM serving; LoRA; Post Training Quantization; Multi-task Scheduling
Primary area: infrastructure
TL;DR: We proposed LoRA-Inlaid, a resource-efficient and high-performance system for multi-task LLM quantization and serving.

Abstract: With the remarkable achievements of large language models (LLMs), the demand for fine-tuning and deploying LLMs in various downstream tasks has garnered widespread interest. Parameter-efficient fine-tuning techniques represented by LoRA and model quantization techniques represented by GPTQ and AWQ are of paramount significance. However, although these techniques have been widely adopted in single-task scenarios, research is scarce in multi-task scenarios. To be specific, we find that mainstream quantization methods would prevent the base LLM from being shared among tasks, so current LLM serving systems are infeasible to integrate LLM quantization with multiple LoRA adapters to achieve memory-efficient multi-task serving. Moreover, existing LLM serving systems lack support for dynamic task addition and overlook the workload differences among tasks, leading to inefficiencies in multi-task scenarios.

This work proposes LoRA-Inlaid, an efficient multi-task LLM serving system. On the one hand, LoRA-Inlaid designs a flexible and efficient multi-task quantization algorithm (MLGPTQ) that facilitates the sharing of a single quantized model for multiple LoRA adapters, which significantly reduces the memory consumption for model deployment. Meanwhile, it supports adding LoRA adapters for new tasks on the fly, without sacrificing the stability of online services. On the other hand, LoRA-Inlaid develops a novel multi-task scheduling algorithm guided by output length prediction and grouping among different tasks, which effectively shrinks the memory consumption and avoids frequent switching of LoRA adapters. Empirical results verify that LoRA-Inlaid outperforms existing state-of-the-art LLM serving systems by up to 1.58 times in terms of throughput, 1.76 times in terms of average latency, 2 times in terms of job completion time, and 10 times in terms of SLO Attainment, while maintaining the same level of model quality.

Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters
Yifei Xia
Fangcheng Fu Peking University ccchengff@pku edu. en
Wentao Zhang
Peking University yifeixia@stu, pku. edu.
Peking University
wentao. www.ooaaaannu.ooooooooo edu. cn
Jiawei Jiang Wuhan University
Bin Cui
Peking University
j jiawei. jiang@whu. edu. cn
bin. cui@pku edu. cn
Abstract
With the remarkable achievements of large language models (LLMs), the demand for fine-tuning and deploying LLMs in various downstream tasks has gamerred widespread interest. Parameter-efficient fine-tuning techniques represented by LoRA and model quantization techniques represented by GPTQ and AWQ of paramount significance. However, although these techniques have been widely adopted in single-task scenarios, research scarce in multi-task scenarios. To be specific, we find that mainstream quantization methods would prevent the base LLM from being shared among tasks, so current LLM serving systems are infeasible to integrate LLM quantization with multiple LoRA adapters to achieve memory-efficiem. multi-tast serving. Moreover, existing LLM serving systems lack support for dynamic task addition and overlook the workload differences among tasks, leading inefficiencies in multi-task scenarios.
This work proposes LoRA-Inlaid, an efficient multi-task LLM serving system. On the one hand. LoRA-Inlaid designs flexible and efficient multi-task quantization algorithm (MLGPTQ) that facilitates the sharing of single quantized model for multiple LoRA adapters, which significantly reduces the memory consumption for model deployment. Meanwhile, supports adding LoRA adapters for new tasks the fly, without sacrificing the stability of online services. On the other hand, LoRA-Inlaid develops a novel multi-task scheduling algorithm guided by output length prediction and grouping among different tasks, which effectively shrinks the memory consumption and avoids frequent switching of LoRA adapters. Empirical results verify that LoRA-Inlaid outperforms existing state-of-the-art LLM serving systems by up to 1.58x in terms of throughput, 1.76x in terms of average latency, 2x in terms of job completion time, and 10x in terms of SLO Attainment, while maintaining the same level of model quality.
1
Introduction
L Large language models (LLMs) have demonstrated impressive effectiveness in various domains 26
30 31 39].
and the demand of deploying LLMs in downstream tasks continues to grow 10
Yifei Xia, Fangcheng Fu, and Bin Cui are with the School of Computer Science and Key Lab High Confidence Software Technologies (MOE), Peking University. Bin Cui is also with the Institute of Computational Social Science, Peking University (Qingdao). Wentao Zhang is with the Center for Machine Learning Research, Peking University. Jiawei Jiang is with the School of Computer Science, Wuhan University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table Comparison supported features of different LLM serving systems
Mask Serving
Multi-task Quantization
Dynamic Task Addition
Multi-task Scheduling
System
vLLM [9] & TensorRT-LLLM 29
X
X
S-LoRR [55 & Punica S
X
LoRA-Inlaid (this work)
20
Given the explosive increase in model size and the limitations of hardware resources, "parameter-efficiet fine-tuning" (PEFT) and "quantiztionn-tee--dlllyyy
47
have become the most common pathways for deploying LLMs in downstream tasks [50]. On the one hand, PEFT techniques, represented by LoRA (Low-Rank Adaptation) [16]. only train small-scale adapters to adapt the base model to specific task, significantly reducing the cost of model fine-tuning,
On the other hand, low-bit quantization techniques like GPTQ and AWQ [12 22 can substantially
reduce the memory requirements of model deployment and alleviate memory access overhead during inference, while maintaining model quality.
Although mainstream LLM serving systems like vLLM and TensorRT-LLM 19 29 have integrated support for the quantized deployment of fine-tuned models, these systems focus on single-task serving scenarios. With the rising demand for various downstream tasks, efficiently supporting multi-task servicing scenarios has become increasingly crucial. This has led to the emergence of multi-task serving systems supporting multiple LoRA adapters concurrently. such S-LoRA and Punica 35). These systems share a unified base model across different tasks and activate different LoRA adapters based on the incoming requests, enabling the simultaneous processing of multiple tasks in a single batch. However, in multi-task scenarios, existing systems still face three major challenges.
First, existing multi task serving systems cannot effectively incorporate mainstream model quantiza- tion methods such as GPTQ and AWQ. Specifically, these quantization methods require calibration of numerical distributions using task-specific datasets, and the quantization process for each task neces- sitates activating the corresponding LoRA adapter. Consequently, the base models after quantization are divergent across different tasks, and thus it is infeasible to share unified quantized model. This limitation leads to performance deficiencies or even unavailability in resource-cons conurrcc-onstrrieed scenarios.
Second, in practical multi-task serving scenarios, it would be necessary to add new tasks in real time. However, existing systems only support a static number tasks and are incapable of dynamically adding LoRA adapters. More importantly, after a quantized model is deployed, current solutions do not support any subsequent quantization and deployment for new tasks without affecting the existing tasks. In contrast, adding new tasks typically requires suspending and restarting the serving process, which severely harms the stability and robustness of online services.
Third, incoming requests for different tasks inevitably have workload variations (such request length, processing time, etc.) and require loading different LoRA adapters for processing. Existing systems overlook these issues during the scheduling for multi-task requests, and thus necessitate loading large number of adapters in single scheduling step as well as frequently switching adapters between adjacent scheduling steps, leading to significant efficiency degradation.
To address these challenges, we develop LoRA-Inlaid, system for multi-task LLM serving. The main contributions of this paper are as follows.
essource-efficien and gigh-performance
To begin with, propose innovative multi-task quantization algorithm termed MLGPTQ (Multi- LoRA GPTQ), which utilizes multi-task data to perform joint quantization on the base model. This allows the quantized base model be shared across multiple tasks. addition, supports incremental quantization for newly added tasks without impacting the performance of online services.
Subsequently, we introduce novel multi-task scheduling strategy based output length prediction and grouping. This effectively reduces memory consumption and memory swapping overhead in multi-task scenarios, significantly enhancing overall system performance.
Based on these two techniques, we develop a brand new multi-task LLM serving system, namely LoRA-Inlaid. As shown in Table LoRA-Inlaid integrates multi-task quantization, enables dynamic task addition, and employs the multi-task scheduling strategy, achieving high-performance and flexible multi-task LLM serving in resource-constrained environments.

---

Finally, experimental results demonstrate that, compared to existing systems. LoRA-Inlaid can increase throughput by up to 1.58x, reduce average latency and job completion time by up to 1.76x and 2x, improve SL. Attainment by up to 10x, and support larger-scale language models under the same resource constraints, all while maintaining nearly the same level of model quality.
2 Background and Related Works
Low-Rank Adaptation. LoRA [16.. short for Low-Rank Adaptation, is one of the most widely used parameter-efficient fine-tuning (PEFT) techniques. Unlike full fall-parameter fine-tuning, LoRA fine-tunes only small adapter, which consists of much fewer parameters than the base model, significantly reducing the training cost. The key idea behind LoRA is that the fine-tuning process should only introduce small changes to the weight matrix of the base model (denoted by W E IR"""), so we can learn two small, low-rank matrices (denoted by A E R B E RRxr where r <m,, n), and approximate such changes with the product of two matrices (i.e., AW BA).
shrinks the model size effectively and thus reduces the memory requirement when deploying the model. In addition, it usually helps to improve efficiency by decreasing the memory access overhead of the model weights. Conse- quently, it has been widely adopted in LLM serving. There various quantization paradigms, with post-training quantization (PTQ) being among the most popular ones. Typically, PTQ computes XINT Round (a Clip(XR//C Qmin- Qmax)), where XR represents the valled parameters be- fore quantization, XINT represents the parameters after quantization to integers, Qini and Qmaz denote the minimum and maximum values of the quantization range, and represents the scaling factor. Various PTQ methods calculate the quantization knobs like a with diverse approaches or implement different approximation methods. While mainstream PTQ methods (e.g., GPTQ [[2], AWQ (22)) have common ground that they need to calibrate the numerical distribution based on small task-specific dataset (a. the calibration sef), since numerous studies have revealed the accuracy after quantization with dataset calibration is usually significantly higher than that without dataset calibration [17.. Therefore, this paper focuses on quantization with dataset calibration.
Low-bit Quantization. Low-bit quantization 7 8 12
22 23
Scheduling in LLM Serving, With the explosive applications of LLMs, more and more studies try to evolve the scheduling strategies in LLM serving for better performance. Early systems like PasterTrrnsformer (28) rely on request-level scheduling. Notably, Yu et al. [45) introduced Orca, the first iteration-level scheduling with firttcoomefirst-seevee (FCFS) order for better batching. Building on this, mainstream LLM serving systems leverage various batching approaches, such as continuous
batching in vLLM [19 and in-flight batching in TensorRT-LLM [29]. FastServe (40) takes the semi-information of requests (e.g., input length, processed time, etc.) into account and tries to minimize average job completion time. However, none of these scheduling strategies consider the
characteristics of multi-task scenarios, as we will discuss in $3.3
Multi-task Serving Systems. Since the LoRA fine-tuning technique keeps the base model unaltered, it is fcasible to share the same base model across multiple LoRA adapters, so that we can serve requests from multiple tasks within single batch. Punica [5] and S-LoRA (35) are notable multi-task serving systems, putting forward the initial efforts to support multi-task LLM serving with multiple LoRA adapters. Specific optimization techniques are proposed. For instance, the Segmented Gather Matrix- (SGMV) kernel is developed to enhance memory and computation efficiency when processing requests from different tasks together. In addition, to allocate more GPU memory to intermediate results (typically, KV cache), existing systems maintain the LoRA adapters in CPU memory and only preserve a relatively small number of LoRA adapters in GPU memory. When LoRA adapter outside GPU memory is needed, it is necessary to perform memory swapping between the CPU and GPU memory.
3 LoRA-Inlaid
The overview of LoRA-Inlaid is depicted in Figure I Given an LLM with multiple LoRA adapters for various downstream tasks, LoRA-Inlaid initiates a joint quantization process ($3.T). which produces a unified quantized base model that can be shared across the adapters. During the online serving, new tasks are to be included on the fly, LoRA facilitates a dynamic task addition process 3.2b that efficiently conducts incremental re-quantization and scamlessly deploys the added tasks.
3

---

LORA-INLAID Server
Inferee:
Clients
Joint quantization ($3.1)
Dynamic Task addition Module (532)
Quantac and Repke
Quantized Model
Request Scheduler (83.3)
Scheckii
Buse
Model
New
Tude!
WWTThb
(Add Tack
Lichedale
Design overview of LoRA-Inlai. The workflow is labeled with numbers in the diagram. ! Quantize and Deploy indicates the initiation of the server performing the multi-task quantization and de- ploying the quantized model and LoRA online. @ Schedule involves utilizing a multi-task scheduling strategy for (3) Inference. a new task is detected, it invokes (4) Add Task to dynamically add the new task without interrupting the ongoing services.
Figure 1:
Furthermore,
employs a multi- task scheduling strategy ($..3 that takes the workload differences into account for better efficiency.
3.1 Multi-task Joint Quantization
As introduced in 2 mainstream quantization methods require task-specific datasets for calibration. In addition, they mostly follow the Forward-Aggregatt Info-Modify Weight- Quaatt paradigm in Figure This paradigm first simulates the activation distribution for given task through Forward propagation and Aggregates Information of this specific task. Subsequenty, uses the aggregated information to Modify model Weights to adapt to the task. Finally, the quantization knobs like scales are calculated based on the modified weights to Quantize the base model.
However, in multi-task scenarios, since different tasks should provide diverse calibration sets and necessitate unique LoRA adapters for computation, the quantized models of different tasks inevitably divergent. Intuitively, if we wish to tweak existing quantization methods to make the quantized model shareable across tasks, we should quantize the model without any LoRA adapters. In addition, we should either (i) quantize the model without calibration or (ii) quantize the model with mixed calibration set consisting of the datasets from all tasks.
However, these approaches fail to accurately capture the unique numerical distribution of each task, and suffer from severe accuracy loss (as evaluated in $4.2). Below we first elaborate the reason why these approaches fail with the widely used GPTQ 12 and then propose our solution !
Drawbacks of GPTQ in multi-task scenarios. Directly applying GPTQ in multi-task scenarios has the following drawbacks. First, as aforementioned, GPTQ can only quantize the model without any LoRA adapters, which is infeasible to accurately capture the correct activation information for multiple tasks during Forward. Second, in Aggregate Info, since the calibration sets from all tasks mixed, GPTQ simply accumulates the information from different tasks into one Hessian matrix, making each task's s specific information diluted and losing the emphasis on critical information from different tasks. Third, in Modify Weight, GPTQ relies on the naïve, mix-aggregated Hessian matrix, overlooking the varying importance across tasks, which results in suboptimal outcomes. These drawbacks make the direct application of GPTQ in multi-task scenarios ineffective.
Our MLGPTQ (Multi-LoRA GPTQ) Algorithm To address these drawbacks, we propose a multi- task quantization algorithm termed MLGGPT. Our algorithm enables joint quantization of multiple tasks to retain only one quantized base model, while effectively maintaining the model accuracy by capturing the numerical distributions of all tasks. The goal of MLGPTQ is to minimize the errors of activations before and after quantization, i.c.,
T
arg min Q(W)
Σ (W + BAA.X. (Q(W) B.A)).))
(I)
NNot that idea our solution can also be applied to other quantization algorithms which follow the same paradigm (like AWQ) since the drawbacks exist generally. Considering that the choice of backbone algorithm is orthogonal to our goal, we focus on GPTQ in this work due to its widespread adaption, and present how to adapt
solution to AWQ in Appendix D

---

GPTQ
Naive mix- aggregated
GPTQ
Hess Malrix
LoRA Taske1
Base Model
Qaamtized
Noive Modified Weight
Model
IoRA
T
LoRA
Task-3 3
Inpot
DOForward
Activation f
@okerregate Info
@DModilyy
Weight
LoRA
Task-44
Min-etrur
Base Model
Modiffid Weight
Quantieed
Model
LoRA
Task-T
Max-ageregated
MLOPPQ Purward
Hese Matrix
MLGPTQ
Figure 2: Process MLGPTQ vs GPTQ. Both MLGPTQ and GPTQ follow the Forward Aagger-- Info- Modify Weight- Saaattttuanttaa paradigm. MLGPTQ primarily improves the first three steps, aiming to better gather and highlight critical information for all tasks.
where T denotes the number of tasks, A, and B, are the low-rank adapter matrices of the 1-th task, X₁ is the input of t-th task, and W and ((W) denote the original and quantized weights of a layer.
As shown in Figure 2 During Forward, MLGPTQ loads the corresponding LoRA adapters based on each task, accurately computing the activations. In Aggregate Info, unlike GPTQ's naive mix- aggregation that disrupts task task-specific information, MLGPTQ derives the max aggrregatnn to solve the objective Eq. (the derivation can be found in the Appendix A). which has the following form:
Wy Q(wg) H Eqz where t" arg max (H, 199 (E(1.2.--TT
VW
(H..) )ag
where H, denotes the Hessian matrix of the t-th task. the g-th parameter in W. To be formal, there are primarily two steps in Aggregate Info. First, it calculates the Hessian matrix information for each task individually (i.e., compute ()-,)) Second, it aggregates the most important information from each one into max-aggregated Hessian matrix (i.e., Hemp = MazAgg (H, I),_)) "H,") In Modify Weight, MLGPTQ utilizes the max-aggregated Hessian matrix to adjust the weights according to Eq.22 2 Finally in Quant, we utilize the modified weights for quantization. Due to space constraints, we only present the core concept of MLGPTQ here. Interested readers are referred to Appendix [A] complete derivation well as the detailed algorithm.
3.2 Dynamic Task Addition
In real-world online services, there need for dynamic task addition (i.e., adding new LoRA adapters). In single-task scenarios, adding new tasks typically requires launching more services with extra hardware resources, which does not affect the services for existing tasks. In multi-tast scenarios, there would be interference since all tasks share the same base model. However, we find that none of the existing multi-task serving systems address this problem, lacking proper solution.
Nevertheless, adding new LoRA adapters on the fly in LoRA-Inlaid is inherently far from trivial since the multi-task quantization poses two challenges: (1. Unseen Distributions) Since the MI.GPTQ algorithm is invoked before the new tasks are involved, the quantized model has not captured the distribution information about the new tasks, making it infeasible work with the new LoRA adapters directly. (2. Serving Interruption) Directly re-quantizing the model requires a substantial amount of memory, it necessitates pausing the ongoing serving for while reserve available space for re- re-quantization, harming the stability of online services. To support dynamic task addition in mulli-task scenarios, LoRA- -nlaid tackles these two obstacles, respectively.
To capture the information of new tasks, a naive solution is to perform full quantization once there new tasks. Denote Tı, runs the two steps of Aggregate Info above with 7 = Ti + Tz. However, this leads to redundant 1-1,. In addition, given the commutative property of the max-aggregation
numbers of existing and new tasks, respectively. The naïve solution
computation of 1H, operation, we have MaxAgg( (H, (T=))= = MaxAgg(MaxAgg( (H 1(11), MaxAgg( ( - +1), +1), where the first term MaxAgg( (H, ')²)) has already been computed as Hemp in the previous quantization. Inspired by this, LoRA-Inlate caches Hemp so that the incremental quantization can be done as follows. In Forward, it capture the activation information of new task Ti + 1,..., ,2. In Aggregate
5

---

Input Len
Output Len
Total Len
Z
Log(# (#tokens)
10a
Tosks num
30
10?
in
Log
36-
101
500
750
1250
1.503
Trench-Engliste Table
Code Transiation Summary Summary Generation
Text
Schedule Iteration
Figure 4: Number of tasks in each scheduling step of different scheduling strategies.
Figure Length distributions of different tasks.
Info, it computes the Hessian matrices for new tasks "H, It=Ty+1 and then max-aggregates the
T- +11 matrices (i.e., (H, 1 It=m,+1 +1 and the cached Hmp (oockee)) ). At last, it performs Modify Weight and
Quant, as introduced in 1 By doing so, incremental quantization with T2 tasks is identical to full quantization with T₃ + T2 tasks, while avoiding redundant computation.
To avoid halting the ongoing services, LoRA-Inlaid spawns a background thread for incremental quantization. Moreover, it is done in a layer-by-layer manner to reduce the memory consumption for each (unquantized) model weight, we load it from CPU memory to GPU memory, perform incremental quantization, remove it from GPU memory, and proceed to the next model weight. The I0 between CPU-GPU is overlapped with computation. Thus, LoRA-Inlate supports seamless task addition on the fly and has very little influence on the ongoing services, as evaluated in $4.4
Putting them together, LoRA-Inlaid develops an asynchronous, layer-wise re-quantization mechanism, which accomplishes incremental quantization with the tasks and cached Hessian matrices asynchronously, without interrupting the serving.
3.3 Multi-task Scheduling
Despite extensive research on scheduling strategies for LLM serving, these approaches primarily focus on single-task serving, leaving the unique characteristics in the multi-task scenarios neglected. Below we analyze two limitations of existing scheduling strategies in multi-task serving. Besides, due to the space constraint, we briefly introduce the corresponding solutions in LoRA-Inlaid, while leaving the details of our multi-task scheduling algorithm in Appendix B
Limitation 1: Divergent Output Length Distributions Leuding to High Average Completion Time. As shown in Figure 3 the distributions of input and output lengths vary significantly across different tasks, while requests of the same task exhibit clustering effects. Current strategies mainly rely on comi-information (e.g.. input length, processed time, etc.) to make the scheduling decisions, but do not consider the information of output length since it is not the prior knowledge. Intuitively, this may work fine for single-task scenarios where the vast majority of requests fall within the same workload and thus the clustering effect exists. However, it is unsuitable for multi-task scenarios due to the divergent output length distributions across different tasks. Eventually, find that existing scheduling strategies suffer from heavy performance degradation when applied to multi-task serving.
Solution 1: Scheduling Guided by Ouipul Length Prediction. Existing research has shown that the output lengths can be accurately predicted by small, distilled model given the requests 49] Inspired by this, we leverage a number of small models, to predict the output lengths of incoming requests. Particularly, upon receiving a new request, we predict its output length on CPU using a small model (255MB). Note that the output length prediction takes about 16 milliscconds for one request CPU, while it takes about 200 milliseconds or more to finish the inference of one request on GPU. Hence, we can completely overlap the prediction, without occupying any GPU computing resources. Based on the predictions, we employ a Shortest Remaining Time First (SRTF) scheduling, which prioritizes requests with the shortest remaining processing time and has been proven minimize the average completion time the field of job scheduling [37].
Limitation 2: Excessive Tasks Involved in each Step Leading to Expensive Memory Access Overhead Due to the randomness and dynamicity of request arrivals, multiple tasks are to scheduled each step. However, owing to the lack of consideration upon the task for each request, existing scheduling

---

Table 2: Model quality of different approaches under different tasks (std-dev given in parentheses). QPTQ and AWQ are in gray background color since the quantized models produced by them cannot be shared acToss different tasks. We mark the best multi taallt-tsskkae quantization approaches (i.c., the best among MLGPTQ, GPTQ
AWQreeaked and RTN) in bold.
Dataset
trans-r-
transees
tram-in
tranc-nl
trans-da
trans-ow
QTamm
shum
tiny-rndes
GSMBK
med-qa
mallicion
Metrie
S_BLEU
S. S_BLEU
S. BLEUU
S. BLEUU
S. BLEUU
S_BLEU
ROUGE-1
ROUGE-1
ROUGE-1
Acc (%)
Acc (%)
(%)
51 11399 (000000 [8354
(0:00 19:34
35,64 (0)
51.
(1.10
40.35 (0)
41.33 [0.01
51.72
02)
32.
(0)
5039 (0)
37.44 (0)
MUGPTQ
(4,05.000331 31.16 (3,62)3363 00.01)30.73
(0.04
34.39 (0.0))000)) 31.20
(0.01)
49,02 (0)
(0.96.8.02)
30.85
(0,5)
31662 (0)
76.7 (0.68)
36,44 (0.6)
13.91 (0.02)
28 (0 21 12:58 (0.08
10:48
(0.07)
33.47 (0
28
94 (0.06
48.23 00.0044..800
(0:02)
19.77 0..06
29
25 10)
31.51 (0)
74.81 (2.53)
15.05 (1.15)
(0.80
(0.11)33.00
(0.05)29.99)
10.09
33.34 (0.11 90.11
47.96 (0.03)
10.12
(0.0))
(1.13
25.68 (0.33)
RIN
33.79 (0) 29.64 (2.0) (0.01) 20.35
33.96 (0) 20.46 (0.02)
(0.01)
(0.02)
30.5 (0.0)
31.01
76.15
33.76
GPTO
34.07 (1.19.000)) 33.79
(0.02)20.86000
54.57 03.57-0002) 31.08
49.26 (0:02)
30.92 (0.06)
31.35
76.7
36.25
AWU
34.17 31,19 (0.05) 33.72 (0.07)20.00
(0.0X)
54.21
31,07
49.04
(1.12)
(0.02)
33,45
75.42
36.18
MI MUGITQQ
51.72
(0.11 (0.63)27.07
36.77 (0.50) 28.06
(0.36)
39.07
27.62
28.74
(7.94)
31,00
GPTQQreaae
21.3-066)
25.89 (0:7) 28.18
(1.01
24.09
15
38.52
(0:15)
21.80:0-46)
28.30
54,02 (7:91
20.90 (0.24)
2 Q.Quss
$1.57
(0.56)))
26.77 (0.7)
(9.7
(1:83:0-46)
28.24
53.66 (19)
31.00 (0.34)
" KIN
26.12 (0.01)
(0)
0.00 (0)
0.35
0.15 (0)
0.30
(0)
26.38 (0)
51.3 (0)
31.4
GPTO
50.83
(0.19
28.65) (0
52.93
(0 29:75
(0.3)
17.17
99.53 (0.17)
12
28.16 (0)
(12.39)
10.50
AWQ
(0.23 (5.35(00 30
(0.73
11.23.0
32)
28
(0:5)
47.13
(0.
57)
28.63 (0)
53.66 (4.08)
11.77 (0.20)
strategies typically involve a great number of tasks, and the system has to load lots of LoRA adapters in each step, as shown in Figure 4 As it is well known that the decoding phase of LLM inference usually bounded by the memory bandwidth, the need for loading more LoRA adapters further exacerbates this issue. Worst still, we observe that the sets of involved tasks vary significantly between consecutive steps. As introduced in $ multi-task serving systems only preserve limited GPU memory space for LoRA adapters, and swap them between CPU and GPU memory when necessary. Consequently, existing scheduling strategies force the system to frequently swap LoRA adapters, rendering performance degradation due to the expensive memory swapping overhead.
Solution 2: Reducing Tasks Involved via Grouping. To address the limitation, we adopt a simple yet effective grouping-basee approach, which partitions requests into groups according to their tasks to guide scheduling. On one hand, to avoid involving excessive tasks in each step, we set a grouping coefficient/ 3 (10 by default) and keep the number of involved tasks below ,3 in each step. On the other hand, to alleviate the memory swapping overhead, we prioritize tasks involved in the previous step, aiming to use the LoRA adapters for more consecutive steps. In addition, we maintain a starvation queue based on the waiting time to get rid of starvation, striking a good balance among the tasks.
4 Experiments
4.1 Experimental Setup
Hardware Environment. All experiments are conducted on one RTX 4090 GPU or one RTX 3090 GPU, with GPU memory capacity of 24GB. Detailed specifications can be found in Appendix C.I
Datasets and Workloads. For accuracy tests, we consider 12 tasks in total, including six translation
tasks (38) (trans-fr, trans-cs, trans-id, trans-nl, trans-da, trans-sw), one text summarization task [14] (xlsum), one table summarization task (48) (QTsum), one code generation task [3] (tiny-codes), one math QA task (6] (GSM8K), one medical QA task (2) (med-qa), and one malicious detection task [ (malicious). Detailed descriptions are provided in Appendix [22] For efficiency tests, we follow prior to generate different levels of request rates using the Gamma process.
works 19 35
Models. We conduct experiments with LLaMA2-7B and LLaMA2-13B (25). For accuracy tests, open-source fine-tuned models are used, as detailed in Appendix C.2 For efficiency tests, following S-LoRA 355. we consider LoRA adapters with different ranks (8, 16. 32, 64) across the served tasks to simulate diverse serving scenarios.
Metrics. For the accuracy test, we focus on Sacre (S BLEU) 18 ROUGE [1] and Accuracy (Ace). Details of these metries are provided in Appendix C.3 For efficiency tests, the considered metrics include throughput, average request latency, job completion time (JCT), and SLO (service level objective) Attainment (33) (the percentage of requests completed within the expected latency). By default, test the serving for I minute, with an expected latency of 6 seconds.
Baselines. For accuracy tests, we focus on two kinds of baselines introduced in $3.1 i.e., (i) the Moating-point round-to-neareds quantization without calibration (denoted as RTN), and (ii) GPTQ [2] and AWQ 22 with a mixed calibration set from all tasks (denoted as GPTQuvoked and AWQ WQwoolled).
Source code is available httttss://gtthut com/PKU- BEOUUUDARRLLRRAAInn..
7

---

To be fair, these baselines quantize the base model without any LoRA adapters to ensure the quantized model is shareable. Besides, facilitate the comparison to single-task quantization, we further consider GPTQ and AWQ for each task individually with the corresponding calibration set and LoRA adapter (denoted as GPTQ and AWQ), which cannot generate shareable quantized model though.
For efficiency tests, we compare LoRA-Inlaid with vLLM [19 and S-LoRA [35]. For vLLM, we quantize each model into 4-bit and launch multiple processes (each with one quantized model) on the same GPU to achieve multi-task serving. For S-LoRA, which does not support deploying quantized models, we deploy one half- aalf-precision (16-bit) model and let multiple LoRA adapters share it. For LoRA- MiRaidd we use 4-bit quantization in all efficiency tests. Note that although the model is quantized, the computation during inference is still executed in half- aalf-precision (i.e., the model weights are dequantized before computation).
4.2 Model Quality after Quantization
We first assess the model quality after quantization of different approaches. Table presents the model quality of different tasks under different metrics. It can be seen that all quantization methods incur accuracy drops compared to no quantization. For 4-bit quantization, the average accuracy drops for MLGPTQ. Gm AWQwwelled and RTN 1.70%, 4.72%, 4.50%, 4.02%, respectively. For 3-bit quantization, MLGPTQ consistently achieves the best results, outperforming GPTQ and AWQnweaked by 59.30%. 69. 69.98% average. While RTN suffers from up 74.41% accuracy drops. In addition, for GPTQ and AWQ, which could not produce a shareable model, the average accuracy drops are very close to MIGPTQ (e.g., 1. 1.00%, 1 1.89% respectively for 4-bit quantization). This proves that our work achieves comparable model quality against single-task quantization while enabling multi-task sharing after quantization.
We also conduct experi- ments to anatomize the effectiveness of MLGPTQ. To be more comprehen- sive, five additional metrics (ROUGEI, ROUGE2 (2T]. NIST MT 19. METEOR [9] detailed
GFTO
GPTO
ML2ITO
trans de
trans-ce
trans-nl
Azudt1
ROUGE
ApUGEL
ROU3E2
ADUGI3
HIST
MAT MT
ALL
MST
ALEI
and G BLEU in Appendix uating machine translation quality are included. In addition, we further consider variant of MLGPTQ termed MLGPTQ,e _ sargers which intentionally excludes the calibration set and LoRA adapter of target task during quantization (the other tasks not affected).
C for eval-
trans-fr
AOUGEE
A2UGE1
ROUGE
RDUSE
AOUGI1
METEOR
5 SiCu
NETEOR
SLEU
NETEOR
BUEU
Figure 5: Effectiveness anatomy. The radar charts show the relative accu- racy drops compared no quantization (outer better).
According to the results in Figure 5 we point out that the effectiveness of MLGPTQ stems from two factors: ! whether the information (e.g., activation and Hessian matrix) of each task is correctly captured, and 2 whether the data distribution of each task involved during the forward pass of quantization. In particular, GPTQweeted (missing ①) and MLGPTQ,, ntrrger (missing 2) exhibit higher accuracy drops compared to MLGPTQ and GPTQ (both fulfilling 1 and ()) in almost all metrics. These results verify that the design of MLGPTQ fits multi-task quantization well.
4.3 End-to-end System Performance
Throughput, Latency, and JCT. We evaluate the system performance with various numbers of tasks and request arrival rates. The throughput, latency, and JCT are shown in Figure 6 Overall, LoRA-Inlaid consistently outperforms S-LoRA and vL.MM S-LoRA fails to serve LLaMA2-133 13B due to the lack of support for deploying quantized models. Since vL..M necessitates launching multiple processes to achieve multi-task serving, the memory consumption grows linearly w.r.t. the number

---

(T=S:
(T=10)
LoRA (T=
(T-100)
VLLM(T-2)
VLLM (T=4)
LL 789RTX4990 2468
LLaM42 76-R TX390 24GB
LLaMA2 TX490
TX3000
Latenoy Throughout
5.0
(Fee/t))
2.5-
207 com
10
20
30
10
20
30
10
20
10
20
500
200
(nc)
200
200-
1000
100
10
20
30
10
20
30
10
20
10
20
30
500
1000
JC1 ())
500
1000
250
10
20
10
20
10 Request rate
20
10
20
Request
rate
Request rate
Request rate
Figure 6: System performance in terms of throughput (higher is better), latency (lower is better), and JCT (lower better) under various request rates (z-axis) and numbers of tasks (T).
Ours
S-LORA
VLLM
SLD Attalmmunt
200%
LLMMA2-7B(2 tasks)
LLaMA2-7B(LOO tasks]
LLaMAR-138(100 taaka)
LLaMA2 138 tanks)
50%
50%
.LLM OOM!!!
S-LRA DCM!
S-LuRA and vLLM OOM
0%
(8,0))
Das02
(8108)
(8.200)
(o_200)
(am
(10.22))
(8-2000)
(a-2000
(ku
06.266
(L
(Laegg)
freqs rate, max length?
Figure 7: SL. Attainment (higher is better) under various serving loads (RTX 4090).
Table 3: Scalability comparison in terms of throughput (reqs/s, higher is hetter) under different request rates
and number of LoRA adapters (I.L.a (...aMA--27 7B @ RTX 4090).
Task num
2
100
1000
Reqs rate
10
20
10
20
3
16
16
26
10
20
Imladd
3,89
4.70
4.86
3.78
4.66
4.81
3.82
4.77
4.89
3.71
4.61
4.73
3.60
4.25
4.58
3.42
4.02
4.22
S-LoRA
2.93
3.45
3.45
3.51
2.97
3.38
3.54
2.91
3,36
3.58
2.97
3.40
3.55
2.87
3.35
3.36
2.78
3.26
3.28
VLLM
1.77
2.46
2.98
1.02
1.68
2.27
0.77
0.80
OOM
UOM
DUM
OOM
[OM
0OM
0OM
OOM
OUM
of tasks, and it encounters out-of-memory (OOM) errors in several cases. In contrast, LoRA-Inlaid supports all cases well. More importantly, since LoRA-Inlate is able to reserve more memory for intermediate results (c.g., KV cache) in serving. it achieves higher performance than the baselines. For instance, LoRA-Inlaid surpasses S-LoRA by 26.5%, 31.3%, 24.1% 1..%% on average, and up to 58. %, 76.3%, and 99.9%, in terms of throughput, latency, and JCT, respectively.
SLO Attainment. We also assess the SLO Attainment under different serving loads by varying the request rates and maximum request lengths. The results are shown in Figure 7 In short, compared to S-LoRA and vLLM, LoRA-Inlaid improves the SLO Attainment by 3.9x, 8.5x on average, and up to 10x, 38x, respectively. Furthermore, we observe that as the request rate or maximum sequence length increases, S-LoRA and vLLM experience a steep decline in SLO Attainment while LoRA- does not. This demonstrates the excellent adaptability of -Inladd to various serving loads.
4.4 More Experiments
Scalability. We investigate the scalability w.r.t. number of tasks. As shown in Table 3 from significant performance decline, dropping by 56%-73% when the number of tasks increases from 2 to 4, and eventually encountering out-of-memory (OOM) errors when the number of tasks reaches 5. In contrast, under all experimented request rates, the throughput of -nlaidd hardly declines, even with 1000 tasks served simultaneously. S-LoRA also supports large number of tasks, while LoRA- consistently achieves better performance across all kinds of workloads.
vLLMM suffers
Ablation Studies of Multi-task Scheduling and Multi-task Quantization. We compare different scheduling strategies on LoRA-Inlaid, The results are shown the left of Figure 8 "Ours (w/o group)", "Ours (w/o prediction)" and "Ours (w/o SRTF)" represent three variants of our multi-task scheduling strategy without task grouping, without output length prediction and without the prediction- based SRTF, respectively. "FIFO" is the strategy adopted in S-LoRA and vLLM, and "Skip-join

---

Ours
Gurswwo SRTF)
Ours Oursialo quant LURA
S10 Attainnunt
Oursiet
0.6
predction)
Throughout
(reqa/s)
0.4
0.4
0.2
0.2-
tenorgrput decresaed 10%
throughout
0.0
decrezsed by 10%
decressedity by
7B
138
78
Serving Process
Figure 8: Left: Ablation studies of scheduling strategies on LoRA-Inlaid (100 tasks). Right: Effectiveness of quantiza- Figure 9: Impact of dynamic task addition on tion to SLO Attainment (Llama-2-13B is not shown due to online throughput (LLaMA2-7B8 RTX 4090 with OOM of the other two methods).
100 tasks initially, and the request rate is 30 reqs/s).
MLFQ" represents the strategy in FastServe [40]. It is evident that multi-task scheduling strategy achieves the best performance in terms of SLO Attainment. The designs of task grouping, output length prediction, and SRTF increase the SLO Attainment by 1.16x, 1.23 and 2.27 on average, respectively, We also explore the individual impact of multi-task quantization as shown in the right of Figure 8 Specificically, we consider a variant of LoRA-Inlaid, which disables quantization (i.e., the served model is not quantized), denoted as "Ours (w/o quant)". ". The results show that multi-task quantization brings 39% improvement ("Ours" vs. "Ours (w/o quant)") when serving the 7B model. Additionally, without quantization, it will lead to OOM when serving the 13B model.
Dynamic Task Addition. We evaluate the ability of dynamic task addition LoRA- Inlaid by adding 1, 5, and 10 tasks to heavily loaded service on the fly. The re-
Forward=Cal
Agg lless matrix
Peak
+Quant
Total
Memory
Full Quant
1403(421
415(1668
1818(422)s
9.2GB
Incr Quant
663(=11)s
416(+5)s
1079(+12)s
9.2GB
sults in Figure 1 show that the throughput
(offline)
undergoes 10%-13% of degradation during
Iner Quant
889
469(3668
13586+131s
2,5GB
the task addition, regardless of the number
Table 4: Time cost of quantization.
of tasks added. This is worthwhile given that the online service need not be interrupted. Meanwhile, to evaluate the time consumption of dynamic task addition, we conducted an experiment where there are 5 tasks in the ongoing service and another tasks need to be added. We measured the time cost of three approaches: "Full Quant", which halts the serving and performs full quantization with 10 tasks, "Iner Quant offline", an offline variant (which halts the serving) of our incremental quantization on the new tasks without by-y-e quantization, and "Incr Quant", our incremental quantization with the 5 new tasks, which works concurrently with the ongoing service. As shown in Table by avoiding the redundant computation, the time cost of forward process and calculation of Hess matrix can be reduced greatly, accelerating quantization. Moreover, although the mayer-by-lyyee mechanism slows down the quantization by 1.26 x due to the extra IO, it reduces the memory greatly and does not halt the serving. These empirical results validate the flexibility and robustness of LoRA-Inlaid for multi-task serving.
5 Conclusion and Limitations
In this work, we focused on LLM serving in multi-task scenarios and developed a multi-LoRA task serving system, namely LoRA-Inlaid. On one hand, we designed flexible and efficient dynamic multi-task quantization algorithm that supports the joint quantization of models for multiple tasks, significantly reducing the memory requirements for model deployment. We also facilitated real- time dynamic task addition, enhancing the stability and flexibility of online services. On the other hand, we introduced a novel multi-task scheduling strategy based on output length prediction and grouping, effectively resolving the issues of high memory overhead and frequent memory swapping when applying existing strategies in multi-task scenarios. Extensive experiments demonstrated that LoRA significantly outperforms existing LLM serving systems.
Despite the effectiveness of -nlaid, still has several limitations. First, our quantization does not detect the existence of malicious or poisoning tasks, which might be intentionally crafted to harm the other tasks. Second, our scheduling does not consider the fairness among tasks (e.g., balancing the total numbers of output tokens for all tasks), which may be essential for shared service platforms. Third, it only supports language tasks while requiring some system re-designs for multi-modal tasks. We wish to leave the exploration of these issues as future works.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (U22B2037, U23B2048), China National Postdoctoral Program for Innovative Talents (BX20230012), China Postdoctoral Science Foundation (2024M750103), Beijing Natural Science Foundation (4244080), research grant No. SH-2024JK29, the Fund of Kunpeng and Ascend Center of Excellence (Peking University), and High-perforrance Computing Platform of Peking University. Fangcheng Fu and Bin Cui are the corresponding authors.
References
[1] malicious-600k. 2023.
[2] Medical mmlu.
mmm lu 2023.
(3] Satanjeev Banerjee and Alon Lavie. METEOR: automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
[4] Hong Chen, Kang Yuan, Yanjun Huang, Lulu Guo, Yulei Wang, and Jie Chen. Feedback is all you need: from chatgpt autonomous driving. Sci. China Inf. Sci., 66(6), 2023.
[5] Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. CoRR, abs/2310.185477 2023.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. CoRR, abs/21 14.14168, 2021.
[7] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3.int8() 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems (NeurIPS 2022), 2022.
[8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems (NeurIPS 2023), 2023.
[9] George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research (HLLT 2002), 2002.
[10] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint urXiv:/808 1808.0988,, 2018.
[11] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. Advances in Neural Information Processing Systems (NeurIPS 2022), 2022.
[12] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post- pest-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323 2022.
[13] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi. al. Textbooks are all you need. arXiv preprint urXiv:2306.1 1644, 2023.
[14] Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Rahman, Shafiq Joty Zadeh, Eduard Hovy, Naeemul Hoque, and Hamdy Mubarak. Xlsum: A large-scale multilingual abstractive sum- marization dataset. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), 2021.

---

[15] Cunchen Hu, Heyang Huang. Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng. Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate LLM inference for mixed downstream workloads. CoRR, abs/2401.11181, 2024.
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations 2022 (ICLR 2022), 2022.
[17] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference Machine Learning (ICML 2021), 2021.
[18] William JF Keenan. Sacre bleu: faith, fashion and freedom: Marist foundation garments 1862-1862. In Materializing Religion, pages 132-153. Routledge, 2017.
[19] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023), pages 611-626, 2023.
[20] Zhi Lei, Guixian Zhang, Lijuan Wu, Kui Zhang, and Rongjiao Liang. A multi-level mesh mutual attention model for visual question answering. Data Sci. Eng. 7(4):339-353. 2022.
[21] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, pages 74-81, 2004.
[22] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. AWQ: activation- aware weight quantization for LLM compression and acceleration. CoRR, abs/2306.00978. 2023.
[23] Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng. Llm-fp4: 4-bit floating-poin quantized transformers. arXiv preprint arXiv:2310. 16836, 2023.
[24] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08.0 08345, 2019.
[25] Meta. Llama 2: Advancing ai through collaboration. 07/1lama- 2/ July 2023. Accessed: 2024-04-21.
[26]
Meta. Introducing meta llama 3: The most capable openly available Ilm to date. 7/ai www...eeaacmmmm wtw.mta..ccm/mooppp./mmmm www../oo/oooo.aa......oooooaaaaa 1aaiaaa111aa.-//. 3/ 2024.
https
[27] Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. Hetu: a highly efficient automatic parallel distributed deep learning system. Sci. China Inf. Sci., 66(1), 2023.
[28]
NVIDIA, Fastertransformmer: A fast and memory-efficient library for transformer models. https://gittub:: wttpth/bb..ccc//i/A/A///aaaeaa/e
2023. Accessed: 2024-05-17.
[29]
NVIDIA. Tensortt-llm. httpp://githab..cm//DDD/t///
2023.
[30]
OpenAl. ChatGPT: Optimizing Language Models for Dialogue, 2022. http:: com/
blog/chase chatgpt
[31] OpenAl. GPT-4 technical report. CoRR, abs/2303.0877/ 2023.
[32] Abhay K. Parekh and Robert G. Gallager. A generalized processor sharing approach to flow control in integrated services networks: the single-node case. JEEE/ACM Trans. Netw., 1(3):344- 357, 1993.
[33] Muhammad Raza. Service level objectives (slos) explained, 2018. Accessed: 2024-05-17.
[34] Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Kurt Shuster, Erie M Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint 23X1: 2020.
12

---

[35] Ying Sheng, Shiyi Cao. Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer. Joseph E. Gonzalez, and lon Stoica. S-lora: Serving thousands of concurrent lora adapters. CoRR, abs/2311.000 103288,, 2023.
[36] Xian-He Sun and Xiaoyang Lu. The memory-bounded speedup model and its impacts in computing. J. Comput. Sci. Technol., 38(1):64-79) 2023.
[37] Andrew S. Tanenbaum and Herbert Bos. Modern Operating Systems. Pearson Education, edition, 2014. Includes comprehensive coverage of various scheduling algorithms, including Shortest Remaining Time First (SRTF).
[38] Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214-2218. Citeseer, 2012.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Annual Conference on Neural Information Processing Systems 2017 (NeurIPS 20/7), 2017.
[40] Bingyang Wu, Yinmin Zhong. Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXhv:2305.05922 2023.
[41] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young. Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation, 2016.
[42] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning (ICML 2023), 2023.
[43] Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving BERT fine-tuning via self-ensemble and self-distillation. J. Comput. Sci. Technol., 38(4))853-866, 2023.
[44] Yuemei Xu, Han Cao, Wanze Du, and Wenqing Wang. A survey of cross-lingual sentiment analysis: Methodologies, models and evaluations. Data Sci. Eng., 7(3):279-299 2022.
[45] Gyeong-In Yu, Joo Seong Jeong, Geon-Won Kim, Soojeong Kim, and Byung- Gon Chun. Orca: A distributed serving system for transformer-base generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
[46] Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International conference machine learning (ICML 2020), 2020.
[47] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jian- feng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint
2019.
[48] Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Xiangru Tang, Yumo Xu, Arman Cohan, et al. Qtsumm: new benchmark for query-focused table summarization. arXiv preprint arXiv:2305, 14303, 2023.
[49] Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An llm-empowered Ilm inference pipeline. Ad- vances in Neural Information Processing Systems (NeurIPS 2023), 2023.
[50] Zhe Zhou. Xuechao Wei, Jiejing Zhang, and Guangyu Sun. Pets: A unified framework for parfmeter-efffcientt transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 489-504, 2022.
13

---

A Details of MLGPTQ
A.1 Derivation of MLGPTQ
MLGPTQ is based on GPTQ, adapting the corresponding LoRA adapter according to the activation values of the imput data minimize the error in activation values before and after quantization. For only one task, GPTQ aims to solve the following problem:
arg min Q(W))
[ Q(
(3)
where Wt represents the weight of (-th layer in the base model, Q(Wi) represents the quantized version of Wt, X, is the input of I-th layer. In other words, the objective of GPTQ is find Q()i) for each layer's weight W, through layer-wise quantization, order to minimize the changes in activation values. Since are discussing quantization within a single layer, will omit the subscript for simplicity in the rest of this section.
As proved by solving Eq. can be transformed into solving Eq. 4 as follows.
dem
arg min E Q(W)
JW Q(W)((W))))) X
(4)
2m1
Since the model has converged through training before quantization, existing works generally assume the model has reached local minimum. Thus, when we add small adjustment VW to the parameter W, according to Taylor expansion, we have
OE @W
VE =
VW
VW WWW H VW + 1(1)))
(5)
2
8 2 represents the Hessian matrix. Again, since the model has converged, existing works generally assume its first-order partial derivative close to zero and thus negligible. By neglecting the first-order partial derivative and higher-order terms, we have
where H
VE VWT. H VW.
(6)
Recall that goal is to quantize W (WW). Denote Vw, = Q(wy) wig where Wa represents the q-th element W. Then, the problem to solve can be re-written as
VW H HH -.WW)
[e] ve + w. Q((a))
arg min arg min VW
where ey represents unit vector with value of 1 at position q and 0 elsewhere. Since this is a constrained convex optimization problem, based on the method of Lagrange multipliers, it is necessary to solve the following equation:
L= VW H. VW + A(e VW + w, - Q(w,))
(8)
By taking the partial derivatives of VW and X, and setting them solution, we have
find the steady-state
1
2 H H VW + xea = 0
(9)
c VW + Wq Q(wg) =0
Solving this, we get
A Wy -Q(wg)
(10)
H 1)m
We Q(wa)
VW
H
C.g.
(11)
H 1)
VE (w, Q(w,))2
(12)
2(H 4)g
where (H- represents the value the diagonal position (g.g) of H which is the inverse of the Hessian matrix.
For the Hessian matrix, say that H 2XX T by proving the following Lemma.
14

---

Lemma A.1. Let a be 1 x n row vector and X be an n x m matrix. The Hessian matrix of the quadratic form ||aXX] 2 is 2XX".
Proof. Let (a1: 02. On be 1 x row vector, and let X -may] be matrix.
Let's denote y = a.X. Then, y a x m row vector with elements yy defined as
yj
a.Tij-
Thus, we have
|aX (aX) (aX)"
We expand this expression and re- re-write it as quadratic form, i.e.,
|ax1B ΣΣΣ
To find the Hessian matrix of this quadratic form, we treat it as quadratic form in Let Q be the coefficient matrix of this quadratic form.
|aX12 =aQaa.
The ki element of Q is given by
m Qik
Zij Tkj
j=1
Thus, the matrix Q can written as And the Hessian matrix is twice Q:
Q ==XX".
H =2 2Q = 2X
Therefore, the Hessian matrix of ||aX 12 is 2XXx", which completes the proof.
GPTQ quantizes the weight W row by row. For each row, according to Eq. 12 mentioned above, finds the minimum wi that leads
increase in the loss function due quantization, then calculates scales via a maz(W:.)) (WW.)-ma((W.... performs quantization using and finally update
Qmaz
the remaining values using Eq. This process repeats until all parameters have been updated.
MLGPTQ considers the scenario of quantization for multiple tasks. During the forward propagation process, it dynamically loads the corresponding LoRA adapter for each task to simulate the correct activation values for the respective tasks. Consequently, the problem need to solve is as follows.
arg min ((W+BAA)X, (Q(W) + B,A.)x,)) Q(W)
(13)
where T denotes the number of tasks, A, and B, the low-rank adapter matrices of the 1-th task, X, is the input of t-th task, and W and Q(W) denote the original and quantized weights of a layer.
Denote W, = W + B, then the problem is re-written as
arg min Q(W) 1=1
W t.. X. - Q(W.)...))
(14)
15

---

Same as GPTQ, we could get T' Hessian matrix Hμ, where t [1.T]. To minimize the objective function, we can obtain the updating formulas for W and in MLGPTQ:
W/q Q(wig) (H )49
(Wy (wig)) 2 (H )99)
VW
H Eqg VE
where t" arg max
(HH, )qg) (15)
rE[],]]
The updating method described here leads to the max aggregation method proposed 3.1 which always selects the Hessian matrix of the task that minimizes EE for updating, ultimately reducing the overall error and thus better-guide geater-giddig parameter updates.
A.2 Pseudocode of MLGPTQ
We provide the pseudocode of MLGPTQ in Algorithm I and we also highlight the differences compared directly applying GPTQ multi-task scenarios, which is termed GPTQweeded in our work (i.e., with mixed calibration set and without any LoRA adapters).
Due to the high complexity and numerical instability of the process described in Appendix A.1 leverage the following optimizations to accelerate the quantization, partly inspired by the practical implementation of GPTQ [1]]
Random Order Optimization. GPTQ requires updating weights in the order that produces the smallest quantization error E. For W E Rmxx, the complexity of GPTQ is ((m)"). However, using a random order achieves similar results and facilitates GPU parallel optimization [2]
non-redundant, Since weight updates between different columns of the same matrix W are batch processing and delayed updates, with 128 columns processed at time, to enhance computation speed.
Batch Processing,
Using numerically stable Cholesky decomposition to pre-compute the necessary information increases computational stability.
Cholesky Decomposition.
Algorithm Routines of MLGPTQ and GPTQ to quantize one layer in multi-task scenarios.
Input: (Xe)
The inputs of different tasks of this layer Forward pass
I: X. <- (W + B.A.)X, Xmixed ΣЛ WX,
2: H Cholesky((2X.xX AD)) for AA [1,2]
H + (2olesky(2X Taized + AI) yy
D. Cal inv of Hessian matrix
Hemp f- zero (H,,
D Init the tmp Hessian matrix Store quantized results b Store quantization errors in blocks
Q < Odoa
5: + Odwm
6: for i + 0, B, 2B, do
7: for j i,... B do
8:
Q.i quant (W.))
Column-wise quantization
9:
E. (W.,, Q.,)/ maxign.m ((Hr')s) )ss)
E.j- + (W., Q..)/(H )33
> Update quantization error
10:
(Happ).. (H,-'). )... where t" arg g(
for [j, B
W W. < W ()(+B) E.j-i (Him)).
W W) W Wi((HB E E.g- (H (1()((B))
Update weights in current block
11:
end for
12:
W ((+R) + W ()+#) E- Htmp)a (14
W. W.(+) W. (1+B))
> Update weights remaining blocks
13: end for
16

---

B Details of our Multi-task Scheduling Strategy
Our multi-tass scheduling strategy is based on task grouping and prediction- based SRTF. Com- pared to other scheduling strategies, it is well-suited for multi-task scenarios, achieving excellent results. The pseudocode is shown Algorithm 2 with the helper functions generate_nent batch batch and schedule_nen _ batch in Algorithm 3 and There are four queues maintaining different requests in our system: ( prefill_eeg reqs: requests that have not yet entered the prefill stage (i.e., new requests that have not yet been served). 2 decoding reqq: requests in the decoding stage. ② hungry seaggdddooddggggeeee requests in starving state in the decoding stage.
requests in starving state that have not yet been served.
First, we check running batch is empty (line to determine whether the system in cold start phase (i.e., the previous scheduling step did not have any requests to serve). If yes, we perform cold start by calling generale_newe nes__eeewbahhhhhhheeeeeeee batch to get a new batch to schedule from pre fil_rees (line new_batch
8). If new wew_hacch is not empty, we proceed to filfill with new batch (line 10-11). Otherwise, the system remains idle since there are no requests (line 13). If homning_botten is not empty, we check if we have consecutively executed max stte__ot_tttee _ont_decooe decoding steps (line 16). If yes, we call generate cetteratteeewaaeeeeeee new new__wtwll batch to schedule new requests from prefill_regy and preggrggppeeflll.. (line 17-20). Otherwise, we check if the continuous scheduling for a batch has reached pre-defined threshold mar_ sat__ctt_deee cont_decode____ pescode_oe__ee _ot_bbatchhhehhbeeooeeeeee (line 22). If yes, we call schedule new new_batch batch to schedule new requests in the decoding stage from decoding reqs and hungry decoding deegsinggregs according to the SRIF and grouping strategy (line 23-25). Otherwise, we continue processing the current running batch (line 27-28). This two-level threshold strategy effectively reduces the frequent swapping LoRA adapters and KV caches due to frequent batch switching. Additionally, by adjusting the threshold size, we can ensure the immediacy and flexibility of schedulingg
Algorithm 2 Our multi-task scheduling strategy based on grouping and SRTF.
Input: Four queues: prefill prelresreqs, decoding_regs, hungry_prefill_g
I: running batch <- 0
2: max__onl,
cont decode cont_decode: Threshold value for decoding
3: _e__cot__dccod_____e _ottcd__ooeeeahcchhhheeeeee batch +-
Threshold value for decoding one batch
decode_count +0
5: decode_count www.ttttoorrrooc catt__one_bccchhhheeeeeeeeee batch +
6: while not terminated do
7: if rrunning batch is empty then
batch new mot_batchh - www..euutchhhprefllllrrr if new batch o 0 then
8:
10:
Perform prefill(new batch)
11:
decoding regs +- decoding regs + new batch
12:
else
13:
Keep idle end
14:
15:
else
16:
if decode, count 2 mar __te___t______eeeeeeeeeeeeeee_ decode then
17:
new batch generate pes___ee__baae battubbbcpppppfllll reggr hew.gllle....gggggrrrrrrreee preftry Perform prefill(new decoding_regs +- decoding_regs + new new_batch decode_count +
18:
19:
20:
21:
else
22:
decodeecounnt____ batch 2 max_cont, mar__cmt__eecode_mme dee_dddeodd__e_eeeeeeee batca then
23:
+ schedule schedule_neww new__eewwwee bwtchbbeecddggggeeeeeee Perform decode(runningg_hete hatch)
new batch new_batch
24:
25:
decode cound. one. butch
26:
else
27:
Perform decode(runniing batch) _onettto_____eeeett batch + hoatt_ooeebbachh
28:
29:
end if
30:
decode_come count decode_count
31:
end
32:
end if
33:
end while
17

---

Algorithm 3 The utility function generale_new__baee for Algorithm 2
1: function GENERATE_ BATCH(prefill hungry_pree patrrrpprreflll.e rilg))
2:
Sort prefill_reqs by len(prompt) predict Sort
predicctootpptt outppttuupptt _lentteeeeeee ascending
3:
hatll/_rreeellleeeeeeeeeeeeeeeee by waiting_ting time descending, then len(prompt) + prredicttoutputt len
ascending
new new__atee botehbhthh.. t-
5:
for each req in parngry_prefilll regs do
6:
if canaa_at add req(reg, new batch) and meet max Jora(rer,, new batch) then _e__aadddrrr__ee_
7:
new www.bbttch.appp aapchhappend(reg)) hungry pecooyppreefeeee frefill ret.ttllrrsseeaaeeee removeer)))))))
8:
end if end for
10:
for each req in prefill_regy do
11:
if can can_add add req(reg, new batch) and meet max lora(reg, new balch) then
12:
new batch www.bbtthhhpppeee appendddppdddr)) pre rre.ill_eeg...emvveerrer reqs.eemov)((ee)
14:
end for
15:
if new__ateh not full then
16:
for each reg in parggrrrpreellllee if can add req(req, new baacch) then new (etech..ppend(reg)) preiil reeeeovrr)s.. end if end for end if
prefill_rege do
17:
18:
19:
20:
21:
22:
for each reg prefitl reqs and hungry_preffiee prefill reqs
23:
req. waitiagiig time +- req. wei end for
24:
25:
for each reg in prefill_reqs do
26:
if req. Mree.waiite waiting lime Threshold then pre frefill trefill_reqqee ret._rrrggreeoerrre. hungry end end for
28:
29:
30:
return new_batch
31:
end function
Algorithm 4 The utility function achedule_new.r for Algorithm 2
I: function SCHEDULF, NEW
batthiionn_athhhhdeddinggggggg Beciiinnn_reseeeee pegsshungry_decooinnggg
2:
Sort decoding reqs by predict_outpee _ottput__teeeet (ou(out ascending Sort weworrddddddngnrroeem reqs by waiting time descending, then thenpredict, perddiic_ouuuuuu len -len (output))
ascending
4: new batch +- 0
5:
for each reg in
de
6:
if can_add, aad__rq((rq, new_ batch) and reg.lora running batch then new hew_batchh.om hawp__atch..ppnd((ee... append(ree): hungry sunggry_ecooding_rge reegooooerrg....
7:
end if end for 10: for each reg decoding reqs
8:
9:
deqs do
if can_add le___add_rrrreee req(reg, new batch) and lora E rrnninggbbo batch then new new__atchh.o append(reg). decoding senadinggreqsremeeeee removee))))( end if end for
12:
13:
14:
15:
if new-_aaa batch not full then
for each req in hungry_deeeee j___eeodiingreree decoding_rege do can__dde aad_dddeee aad_req(reg. news_atte new__atcch) then new end if for end if
17:
18:
decoodinggreqsseemooee reqv.e))
19:
20:
21:
22:
for each reg in decoding_regs do
23:
req. wwiwwaingggittoo time waiting
24:
end for
25:
return new batch
26:
end function
18

---

C More Experimental Details
C.1
Experimental Environment
The GPU platforms for evaluation are shown in Table 3 The RTX 3090 platform is equipped with InteI(R) Core(TM) 19-10900X CPU @ 3.70GHz and 256GB host memory, while the RTX 4090 platform equipped with Intel(R) (R) Gold 6330 CPU 2.00GH and 512GB host memory.
Table 5: Experimental GPU platforms in detail.
Platform
RTX 4090
RTX 3090
GPU cores
16384
10496
GFLOPS
51640
35580
Memory Capacity
24 GB
24 GB
Memory Access Bandwidth
1024 GB/s
936 GB/s
C.2 Summary of Evaluated Tasks
We primarily select 12 major datasets for testing, covering tasks such as machine translation, text summarization, table summarization, code generation, math QA, medical QA and malicious detection. For each task, we use open-source models from Hugging Face that have been fine-tuned using the training set of the corresponding dataset and evaluated on the test set. The summary is presented in Table 6
Table 6: Dataset Summary
Avg. Input Length
Avg. Output Length
Dataset Name
Abbreviation
Type of Task
OPUS-French-Englist
trans-fr
121
105
Machine Translation
OPUS-Czech Ezech-English
trans-es
47
47
Machine Translation
English
trans-id
47
38
Machine Translation
OPUS- Vietnamese- Vietnamese-English
trans-nl
Machine Translation
OPUS-Danish-Ennghh
trans-da
Machine Translation
SPUUSSSweiih--Enllinn
trans-sw
64
65
Machine Translation
XI.Sum
xlsum
2595
125
Text Summarization
QTSUMM
QTsum
1350
339
Table Summarization
tiny-codes
tiny-codes
328
1890
Code Generation
gsm8k
GSMSk
240
293
Math Question Answer
medical_ medical_mcadow www.ddwwmmmmluuuu mmmmlu
med-qa
367
1
Medical Question Answer
malicious-600k
malicious
59
1
Malicious Detection
For the machine translation tasks of different languages, we consider the classic bilingual translation dataset OPUS 338, specifically choosing six translation tasks: French-to Crench-to-Engiee -tng-gsis,h, Frech-to-Englinn Indonesian Indonesian-to-EEnniihh, Vietnamese-to-Be to-English Danish-to-Engge Ennglis,, and Swedish-to- tog-llish, consider- ing the diversity of languages.
For the text translation task, we consider the XL.Sum dataset 14. comprehensive and diverse dataset comprising 1.35 million professionally annotated text-summary pairs extracted from the BBC.
For the table summarization task, consider the QTSumm dataset (48), which is large-scale dataset for query-centric summarization tasks on tabular data. contains 7,111 numan-annolan query-summary pairs and 2,934 tables covering various topics.
For the code generation task, we consider the tiny-codes dataset million short and clear code snippets, aiding LLM models in learning how reason using both natural and programming languages. The dataset includes a variety programming languages such as Python, TypeScript, JavaScript, Ruby, Julia, Rust, C++, Bash, Java, C#, and Go.
This dataset consists of 16
19

---

For the math QA task, we choose the GSM8k (Grade School Math 8K) dataset [6], which consists of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning.
For the medical QA task, we choose multiple choice question.
[2]. which contains 3.79k medical
For malicious detection task, we choose malicious-600k which contains 641k URLs to determine whether they malicious.
C.3 Summary of Metries
(1) SacreBLEU represented as BLEU, a classic machine translation evaluation standard. scores by comparing the n-gram overlap between the machine translation output and one more reference translations, while also considering a brevity penalty to prevent favoring overly short translation outputs.
(2) rougel 21 represented ROUGE1, calculates the overlap ratio of words between the machine- generated text and the reference text, i.e., the unigram (single word) match rate.
(3) rouge2 21 represented as ROUGE2, similar to rougel, indicates the bigram (two- word sequence) match rate.
(4) nist mt [9] represented as NIST __T,, is an improvement on BLEU that gives higher weight to less common words, encouraging diversity and accuracy in translation.
(5) meteor 3 abbreviated as METEOR, calculates the score based on the harmonic mean of precision and recall and introduces penalty factor to penalize excessive mismatches.
(6) google bleu [41) abbreviated as G BLEU, is an improvement on BLEU that adjusts for smooth- ing methods, n-gram weights, and brevity penalties to optimize its performance.
20

---

D Multi-LoRA AWQ Migration
Algorithm 5 Routines of MLAWQ
AWQ AWQ Doeated
quantize on layer in multi-task scenarios.
Input: _x])= aatio_search__pppe_e space
>> The inputs of different tasks of this layer
1: M + (W BBA
Maxxed f Σi WX
Forward to get monitoring matrix
W mean f W. (.mean(0)
3: X X' mCan(() for Vt [1,
X f- {X') M mean(()
Aggregate Info: Per-channel
mean of X
4:
best_s None
5: min errors + 00 Lám(X shope-1]]
min_error error f- 00
Initialize minimum error(s)
6: for ratio in ratio search space do
Aggragate Info: Calculate best
S1
X pow(catio W ness pow! (1 ratio)
for vi [1,
Kucas X (ro(io)))) W
menn- paw maoo)
Calculate
8: W scaled f- W. 81
Wacsted +- W
Scale W
X X
//
Xscaied
x 12
/t=1/s
Scale
10: emTS: + M (a (round (clamp waaeea/ ox min_ val, max_val))) )+1)) B' A' " X 2.0.ll/..commm
CTT Mmixed (round (clamp (W watee// min_ val, max_vall)) Xxsieed
Use pscudo quantized
forward cal quant of this ratio, where is the scale factor of pseudo quant
11:
min_errs|]]] best_s[j] + mine [j),, ), Sang min, (e1m) [2] 4 j E X shapes--11 11, vt E [1
min eir <- (min err). best <- (err < min err) ? best s
D> Aggregate the min error to get best_s
12: end for
Modiified best
Modify weight
quant <- quantize( W modifedd
Quantize modified weight
15:
Return W quant
Our work centers on GPTQ due to its widespread use, but our solution can also adapt to AWQ. We presented the differences between MLAWQ and AWQuweaked in Alg/5 As introduced in $3.1 most quantization methods follow the forward-Aggregen Info-Modify Weight-Quant paradıgm. In essence, AWQwwekeed smooths outliers by multiplying weights with smoothing factor, best_s, to minimize per-channel quant error:
In Forward, the input is multiplied by the weights create unquantized monitor, guiding min error quantization (line 1).
In Aggregate Info, the average of all samples and weights calculated for each channel (lines 2&3) to determine smoothing factor (line 7). W and X are smoothed to remove outliers (lines 8&9). Then, smoothed W is pseudo-quantized (quantize- (quantize-then-dqqquee -equunnee to simulate round loss) and compared to the unquantized monitor for quantization error (line 10). This process iterates over various ratios (line 6), selecting the factor with the smallest error as best_s s (line 11). Then, this best_s is used to Modify the weight (line 12), followed by the Quant (line 13) process using the modified weight.
The drawbacks discussed in 3.1 also exist for AWQuecked in multi-task quantization:
Forward (line 1): It can't pass LoRA adapters during activation distribution simulation, causing quantization bias during inference.
Aggregate Info: It uses Xmean = X Xmean(0), naive mixed average of multi-tasks' info. Since each task affects each channel differently, simply averaging blurs distributions, ignor- ing individual effects.
As explained in 3.1 and shown in Fig 2 MLGPTQ mainly improves the first three steps tackle GPTQ's issues in multi-task scenarios. Similarly, we can fix AWQ's issues to create better multi-task quantization algorithm MLAWQ:
21

---

Forward: MLAWQ loads corresponding LoRA adapter for each task to participate in forward propagation, accurately simulating real activation distribution (line
ggregate Info: Instead of mixing and averaging features of each column across all tasks to compute MLAWQ computes the average for each task separately to get (line 3&7). Then it calculates quantization error for each column rather than the entire matrix (line 10). If the i-th task results in the smallest quantization error for the j-th column, it sets best_s [5] = sز (line 11). This approach allows optimal error minimization, showing each task's individual effect on different channels, enhancing Aggregate Info (lines 3&6-11), and improving the Modify Weight (line 12) and Quant (line 13) processes.
In summary, work identifies common drawbacks of current single-task quantization methods in multi-task scenarios. By addressing these issues, we can develop more precise multi-task quantization algorithms.
22

---

E Broader Impact and Future Work
This work proposes LoRA-Inlaid, a brand new LLM serving system for the multi-task scenario. LoRA-Inlate is featured with a series of innovations, specifically the multi-task joint quantization algorithm, the dynamic task addition mechanism, and the multi-task scheduling strategy. Considering the booming research and applications of LLMs in various downstream tasks, we believe LoRA-Inlai has the potential to gain widespread adoption and shed light on the high-performance and resource- efficient system designs for follow-up works in the field of LLM serving. However, there are several issues that LoRA-Inlaid does not consider currently.
On the one hand, the multi-task quantization algorithm in LoRA-Inlaid does not involve the detection of malicious poisoning tasks that may bring negative impacts on the other tasks. One of the most typical use cases of LoRA-Inlaid is the personalization of LLMs, where clients can upload their data create personalized LoRA adapters using the same base model (or directly upload their self-tuned LoRA adapters). The server is responsible for serving requests from all these clients using the proposed LoRA-Inlaid system. Fortunately, these LoRA adapters are independently manufactured, so we can apply malicious detection to them individually. For instance, the server can prepare rich set of evaluations assess the security risks of each LoRA adapter, including violence, discrimination, unlawful responses, etc. If any LoRA adapters fail to pass the evaluation, the server can reject serving them.
On the other hand, the multi-task scheduling strategy in LoRA-Inlaid ignores the fairness among different tasks (e.g., controlling the number of output tokens to be close), which may make our work ineffective under some settings. To measure fairness among tasks, we can compute a weighted combination of numbers of input and output tokens for each task. This is because the prefilling and decoding phases in LLM inference have different workload characteristics (also why these tokens differ in online service pricing). Then, we can borrow the idea of Weighted Fair Queueing (WFQ) 32 for scheduling different tasks. We wish to address these issues in the future.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section 3 and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5 and Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section source code.
Appendix
C.1
Appendix
C.2 and we have provided the link to
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Section 4.1
Appendix
C.2 and we have provided the link to source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cessionnooccyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwticccguie//eeeeeeeeeii CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.1 Appendix
C.1
Appendix
C.2 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section and Appendix C.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: wwtprrips.ccpuuuuiiee http://pp....pppooooocccccccaoea
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix
E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "EGSDUP7BFp", "reviewer": "Reviewer_v1aq", "summary": "This paper introduces LoRA-Inlaid, an innovative and efficient system for quantizing and serving Large Language Models (LLMs) in multi-task environments. By utilizing the Multi-LoRA GPTQ (MLGPTQ) algorithm, LoRA-Inlaid facilitates sharing a unified quantized model across various LoRA adapters, significantly reducing memory usage for model deployment. The platform also features a dynamic task addition mechanism that enhances the stability and reliability of online services. Moreover, it introduces a novel multi-task scheduling approach guided by predicted output lengths and task grouping, significantly reducing memory consumption and increasing overall system efficiency. Experimental results demonstrate that LoRA-Inlaid outperforms current state-of-the-art (SOTA) LLM serving systems in terms of throughput, average request latency, Job Completion Times (JCT), and Service Level Objectives (SLO) Attainment without detracting from the model's performance.", "questions": "1. In Figure 5, why is there a lack of error bars for RTN?\n\n2. In Figure 5, why does the accuracy drop for some tasks with 3-bit quantization appear to be less than that for 4-bit quantization?\n\n3. In the three charts of Figure 6, MLGPTQ and GPTQ have their respective strengths and weaknesses across various metrics. It is suggested that additional tasks be added to demonstrate the superiority of MLGPTQ in terms of accuracy.\n\n4. Currently, there are only experiments on accuracy drops. Please supplement with accurate experiments.\nIn the efficiency tests, only S-LoRA used half-precision, which significantly increased its computational overhead, leading to an unfair comparison experiment.\n\n5. In addition to the End-to-end system performance, could you independently analyze the individual impacts of components such as multi-task quantization, incremental quantization, and predicted output length on each performance of LLM services?\n\n6. Please add formalized descriptions of dynamic task addition and multi-task scheduling in the main text and a brief outline of the specific method.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper introduces LoRA-Inlaid, a novel multi-task serving system for Large Language Models (LLMs), featuring innovations such as multi-task joint quantization, dynamic task addition, and multi-task scheduling. These advancements significantly propel the current field of LLM deployment and services.\n\nSignificance: LoRA-Inlaid reduces the memory requirements for model deployment through its multi-task joint quantization and scheduling strategies while maintaining model quality. This is of great importance for resource-constrained environments. Additionally, the addition of dynamic tasks ensures the stability and robustness of online services.\n\nClarity: The paper has a clear structure and is easy to comprehend, providing detailed procedures of the algorithms and facilitating reproduction. The experiments encompass a variety of metrics, including throughput, average request latency, JCT, and SLO Attainment, demonstrating a well-designed and persuasive set of results.", "weaknesses": "1. The paper introduces multi-task quantization, incremental quantization, and predicted output length, which naturally will incur additional computational overhead. However, the paper needs an analysis of the extra costs associated with introducing these techniques.\n\n2. The paper is missing absolute accuracy comparison experiments for the unquantized model. Both Figure 5 and Figure 6 represent relative accuracy drops compared to the quantization model without providing a direct comparison to the baseline performance of the unquantized model.\n\n3. An excessive amount of content from Chapter 3 has been relegated to the appendices, with the main text providing a succinct description of the specific methods and also needing formulas to aid in explanation, which affects the clarity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W2 & Q4 (first half)\nSince divergent metrics (SacreBLEU and ROUGE-1) are used for different tasks, Fig 5 shows the relative performance to align the y-axis. We have presented the detailed results in _Table A_ of the one-page PDF, showing that MLGPTQ consistently outperforms the baselines.\n## Q1\nThanks for pointing out the issue. The std-dev is also provided in _Table A_. RTN has a lower std-dev than the other quantization approaches. This is because RTN is deterministic while the other approaches introduce randomness as we shuffle the calibration sets for every time of quantization. The randomness in RTN (as well as Unquantized) only comes from the non-deterministic outputs (the LLM may produce different outputs for the same input to enhance creativity), however, such randomness hardly affects the evaluation.\n## Q2\nFig 5 uses different value ranges for the y-axes of 3-bit and 4-bit quantization, leading to misunderstanding. _Table A_ shows 3-bit quantization is consistently worse than 4-bit quantization.\n## Q3\nIn Fig 6, GPTQ is the baseline that quantizes the model individually for each task, so the quantized model of GPTQ cannot be shared among different tasks. In contrast, MLGPTQ quantizes the model jointly for all tasks, ensuring the quantized model is shareable. Hence, it is reasonable that GPTQ is better on some metrics. We considered GPTQ as a reference since it fulfills the two key factors (discussed in lines 283-286 of Section 4.2), aiming to better anatomize the effectiveness.\n\nDue to the space constraint, we only provided the results on 3 datasets in Fig 6. To address the comment, we have presented the results on the other 3 datasets of the translation task in _Fig D_ of the one-page PDF (the other tasks are not considered since metrics like G_BLEU, S_BLEU, and NIST_MT do not apply). The results are consistent with Fig 6 --- both MLGPTQ and GPTQ, which fulfill the two factors, outperform the other approaches in almost all metrics. Moreover, since MLGPTQ produces a shareable quantized model while GPTQ cannot, the results verify that MLGPTQ is suitable for multi-task quantization.\n## Q4 (second half)\nAlthough the model is quantized, the computation during inference is still executed in half-precision. Before the computation of each layer, the corresponding model weights are temporarily dequantized (e.g., `MatMul4Bit`in `bitsandbytes`: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.43.0/bitsandbytes/autograd/_functions.py#L516). Thus, mmodel quantization does not decrease computational overhead but introduces a minor overhead of dequantization. \n\nBesides, S-LoRA does not support deploying quantized models since existing quantization methods do not fit multi-task scenarios, as discussed in Section 3.1 of our manuscript, so we could only use half-precision for S-LoRA. (LoRA-Inlaid is the first system that supports deploying quantized models for multi-task serving.) Thus, we believe the system comparison in our work is fair.\n## W1 & Q5\nBelow we analyze the individual impact of each component.\n\n**Multi-task Quantization**\n\nAs discussed in the response to _Q4 (second half)_, moodel quantization does not decrease computational overhead. However, it reduces the memory consumption of storing model weights so that we can preserve more memory for KVCache to improve overall efficiency. \n\nTo assess the impact of multi-task quantization, we considered a variant of LoRA-Inlaid, which disables quantization (i.e., the served model is not quantized), denoted as \"Ours (w/o quant)\" in the right of _Fig C_ of the one-page PDF. The results show that multi-task quantization brings 39% improvement (\"Ours\" vs. \"Ours (w/o quant)\") when serving the 7B model, and disabling it leads to OOM when serving the 13B model.\n\nThe time cost of quantization is discussed in the analysis of incremental quantization below. Note that we can quantize the model before serving, so it does not affect the performance of serving.\n\n**Incremental Quantization**\n\nThe incremental quantization aims to support dynamic task addition without halting the serving. As introduced in our _Global Responses_, there are two steps in quantization, and we avoid redundant computation in the first step of incremental quantization. Meanwhile, a layer-by-layer mechanism is developed to reduce the memory consumption of incremental quantization. \n\nTo evaluate its impact, we conducted an experiment where there are 5 tasks in the ongoing service and another 5 tasks need to be added. We measured the time cost of three approaches:\n- Full quantization with 10 tasks, which halts the serving.\n- An offline variant of our incremental quantization with the 5 new tasks, which halts the serving. However, it does not need to perform the layer-by-layer quantization.\n- Our incremental quantization with the 5 new tasks, which works concurrently with the ongoing service.\n\nAs shown below, by avoiding the redundant computation, the time cost of the first step can be reduced greatly, accelerating quantization. Moreover, although the layer-by-layer mechanism slows down the quantization by 1.26 times due to the extra IO, it reduces the memory greatly and does not halt the serving. \n||Step 1|Step 2|Total|Peak Memory (GB)|\n|-|-|-|-|-|\n|Full Quant|1403(±21)s|415(±6)s|1818(±22)s|9.2|\n|Incr Quant (offline)|663(±11)s|416(±5)s|1079(±12)s|9.2|\n|Incr Quant|889(±11)s|469(±6)s|1358(±13)s|2.5|\n\n**Output Length Prediction**\n\nAs introduced in our _Global Responses_, the output length prediction is done on CPU and overlaps with the LLM inference on GPU. To measure its impact, we experimented with a variant of LoRA-Inlaid without the output length prediction (instead, the averaged output length of each task is used), denoted as \"Ours (w/o prediction)\" in the left of _Fig C_ of the one-page PDF. The results show that it brings 18% and 27% improvement (\"Ours\" vs. \"Ours (w/o prediction)\") for the 7B and 13B models.\n## W3 & Q6\nPlease refer to the _Global Responses._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YXwyAzt5r", "reviewer": "Reviewer_RQ33", "summary": "The paper addresses the need for efficient fine-tuning and deployment of large language models (LLMs) in multi-task scenarios, which has been largely overlooked in favor of single-task scenarios. Existing quantization methods, such as GPTQ and AWQ, and parameter-efficient fine-tuning techniques like LoRA, are commonly used, but they do not support the integration of multiple tasks due to their limitations in sharing the base model across tasks and handling dynamic task addition. To tackle these issues, the authors propose LoRA-Inlaid, a multi-task LLM serving system that combines a flexible and efficient multi-task quantization algorithm with a novel multi-task scheduling strategy. This system significantly reduces memory consumption, supports real-time task addition, and enhances the stability of online services.\n\nThe major contributions of this paper are as follows. The authors introduce an innovative multi-task quantization algorithm, MLGPTQ, which enables the joint quantization of models for multiple tasks, allowing a single quantized base model to be shared across tasks and supporting incremental quantization for new tasks. Additionally, they develop a novel scheduling strategy based on output length prediction and grouping, which minimizes memory consumption and reduces memory swapping overhead in multi-task scenarios. These techniques are integrated into the LoRA-Inlaid system, which demonstrates significant performance improvements over existing LLM serving systems, achieving up to 1.58× higher throughput, 1.76× lower average latency, 2× faster job completion time, and 10× better SLO attainment, all while maintaining model quality. Despite some limitations, such as the need for improved detection of malicious tasks and fairness considerations among tasks, LoRA-Inlaid represents a significant advancement in multi-task LLM serving, highlighting its potential for resource-constrained environments.", "questions": "Detection of Malicious Tasks: Can you provide more details on how you plan to enhance the detection of malicious tasks? Are there any preliminary strategies or methods you are considering to address this vulnerability?\n\nFairness Among Tasks: How do you measure fairness among tasks in the LoRA-Inlaid system? Can you elaborate on the strategies you plan to implement to ensure equitable resource distribution?\n\nIncremental Quantization Process: Could you provide more information on the incremental quantization process? How are new tasks integrated into the system without impacting the performance of existing tasks?\n\nGeneralizability of Results: The experiments demonstrate significant improvements, but are these results consistent across a broader range of tasks and datasets? Can you share any additional experimental results or plans for future testing?\n\nScalability and Real-Time Performance: How does the system scale with a large number of tasks added in real-time? Are there any performance benchmarks or case studies that illustrate the system’s scalability in real-world scenarios?", "limitations": "The authors have identified some limitations, such as the need for improved detection of malicious tasks and fairness considerations among tasks. However, these limitations are not fully addressed in the paper.\n\nAddressing Malicious Task Detection: The paper mentions the need for better detection of malicious tasks but does not provide concrete strategies. Constructive suggestion: Develop and describe specific methods for detecting and mitigating malicious tasks, potentially through anomaly detection techniques or secure task validation protocols.\n\nFairness Among Tasks: Fairness is noted as a limitation, but there is little discussion on how to ensure it. Constructive suggestion: Elaborate on fairness metrics and propose strategies for fair resource allocation among tasks, possibly through dynamic scheduling algorithms that prioritize based on task urgency and resource consumption.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces an approach to addressing the overlooked area of multi-task fine-tuning and deployment of LLMs. By proposing LoRA-Inlaid, the authors present an innovative multi-task quantization algorithm (MLGPTQ) that allows for joint quantization of models across multiple tasks, supporting incremental quantization for new tasks. This results in their novel multi-task scheduling strategy, which efficiently manages memory consumption and task addition, marking a departure from existing single-task-focused methods.\n\nQuality: The research is evaluated through comprehensive experiments and performance evaluations. The authors provide evidence of significant improvements over existing LLM serving systems, including up to 1.58× higher throughput, 1.76× lower average latency, 2× faster job completion time, and 10× better Service Level Objective (SLO) attainment. \n\nClarity: The paper is well-structured and clearly presents the problem, methodology, and results. The authors provide detailed explanations of their innovative multi-task quantization algorithm and scheduling strategy, making complex concepts accessible. The use of figures and tables to illustrate performance improvements enhances the clarity and readability of the paper.\n\nSignificance: This work addresses a critical gap in the efficient deployment of LLMs in multi-task scenarios. The LoRA-Inlaid system has the potential to greatly improve the efficiency and stability of online services, particularly in resource-constrained environments. The advancements presented in this paper can help advance deployment and scalability of LLMs across various applications, making it an important contribution to the field.", "weaknesses": "Editorial comments: Abstract: Avoid excessive use of undefined acronyms in the abstract. Can you explain in the abstract briefly how is LoRA-Inlaid related to MLGPTQ? References: Capitalize proper names and acronyms properly in the References. Some references are incomplete — their publication details (or URL) are missing (e.g., [10])\n\nOutliers: (Figure 3) There are a significant number of outliers represented by circles, especially in the French-English Translation and Table Summary tasks. It might be useful to provide a brief explanation or context for these outliers.\nY-Axis Scale: The y-axis scale goes up to 4000 tokens, but most data points fall well below this range. This could make it harder to see differences in distributions for tasks with shorter lengths. Using a logarithmic scale or breaking the y-axis into two parts could provide better clarity.\n\nOutliers and Variability (Figure 4): There seems to be significant variability in the number of tasks for Skip-join MLFQ (FastServe) and FIFO (S-LoRA). Providing statistical summaries such as mean or median lines could help interpret the data more effectively. Including error bars or confidence intervals would give a better understanding of the variability and reliability of the scheduling strategies.\n\nMalicious Task Detection: The paper acknowledges the need for improved detection of malicious tasks but does not provide detailed solutions or strategies to address this issue. This represents a potential vulnerability in the proposed system that could be exploited in practical applications.\n\nFairness Considerations: Fairness among tasks is briefly mentioned as a limitation, but the paper lacks an in-depth discussion on how fairness is measured and what specific strategies could be employed to ensure equitable resource distribution among tasks. This is crucial for practical deployment in environments where multiple users or tasks compete for limited resources.\n\nExperimental Scope: While the results are promising, the scope of the experiments is somewhat limited. The paper would benefit from additional experiments across a wider range of tasks and more diverse datasets to fully validate the generalizability and robustness of the proposed system.\n\nIncremental Quantization Details: The paper does not provide extensive details on the incremental quantization process. More information on how new tasks are integrated into the system and the potential impact on existing tasks would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Editorial Comments\n**Undefined Acronyms**\n\nThere are three undefined acronyms in our abstract:\n- **LoRA**, short for Low-Rank Adaptation, is one of the most widely used parameter-efficient fine-tuning techniques for LLMs. \n- **GPTQ** and **AWQ** are state-of-the-art quantization algorithms for LLMs. GPTQ (Frantar et al., ICLR 2023) merges the name of GPT model family with \"Q\"uantization. AWQ (Lin et al., MLSys 2024) stands for Activation-aware Weight Quantization.\n\n**How is LoRA-Inlaid related to MLGPTQ?**\n\nLoRA-Inlaid consists of two major techniques: the multi-task quantization algorithm and the multi-task scheduling algorithm. We term the multi-task quantization algorithm as MLGPTQ (Multi-LoRA GPTQ). MLGPTQ supports joint quantization of multiple tasks and allows the quantized model to be shared across tasks. It also supports incremental quantization, facilitating dynamic task addition without impacting performance. We'll add the name of our MLGPTQ method to the abstract for clarity.\n\n**References**\n\nThanks for pointing out the issue. We will update the references in our manuscript accordingly.\n## Outliers and Variability\n**Explanation for the Outliers in Fig 3**\n\nThe outliers are due to the long-tail property of sequence lengths, where few sequences are significantly longer than others. This property is common in NLP datasets (e.g., see section 2 of this article: https://www.harmdevries.com/post/context-length/).\n\n**Scale of the y-axis in Fig 3**\n\nThanks for the suggestion. We used the same y-axis value range in Fig 3 to clarify differences in sequence length distributions across tasks. For instance, it's clear that the table summary task has shorter output lengths than the code generation task. Thus, we are afraid that using a logarithmic scale or breaking the y-axis into two parts may not achieve clarity as expected.\n\nTo address the concern, we have redrawn Fig 3 as suggested, shown in _Fig B_ of the PDF. We will update our manuscript if the reviewer prefers the new figure.\n\n**Variability in Fig 4**\n\nThanks again for the suggestion. We present the mean and std-dev for each strategy in Fig 4 below. As discussed in Section 3.3, FIFO (S-LoRA) and Skip-join MLFQ (FastServe) lack consideration of tasks scheduled in each step, leading to significant variability. In contrast, our approach shows a smaller std-dev, so our strategy is more suitable for multi-task scheduling.\n\n|  | Ours | FIFO (S-LoRA) | Skip-join MLFQ (FastServe) |\n| --- | --- | --- | --- |\n| mean | 10.11 | 22.22 | 31.49 |\n| std-dev | 1.57 | 2.07 | 4.27 |\n\n\n## Malicious Task Detection\nOne of the most typical use cases of LoRA-Inlaid is the personalization of LLMs, where clients can upload their data to create personalized LoRA adapters using the same base model (or directly upload their self-tuned LoRA adapters). The server is responsible for serving requests from all these clients using the proposed LoRA-Inlaid system. Fortunately, these LoRA adapters are independently manufactured, so we can apply malicious detection to them individually. For instance, the server can prepare a rich set of evaluations to assess the security risks of each LoRA adapter, including violence, discrimination, unlawful responses, etc. If any LoRA adapters fail to pass the evaluation, the server can reject serving them.\n\n## Fairness Considerations\nTo measure fairness among tasks, we can compute a weighted combination of numbers of input and output tokens for each task. This is because the prefilling and decoding phases in LLM inference have different workload characteristics [a] (also why these tokens differ in online service pricing). Then, we can borrow the idea of Weighted Fair Queueing (WFQ) [b] for scheduling different tasks.\n\n[a] Hu et al. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads.\n\n[b] Parekh and Robert. A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case.\n\n## Experimental Scope/Generalizability of Results\nWe evaluated our work's effectiveness with 9 datasets from 4 task types, as detailed in Appendix D of our manuscript. We believe these datasets are with significant diversity:\n\n- We considered six languages (French, Czech, Indonesian, Vietnamese, Danish, and Swedish) for the machine translation task, covering various language families.\n- Beyond texts, we also considered two other kinds of inputs (tables and codes) to evaluate our work over diverse tasks.\n\nTo address reviewer concerns, we added three datasets with distinct tasks to enhance experimental diversity: grade school math problems (GSM8K), medical QA (Medical_MMLU), and anomaly detection (malicious-600k). As shown in _Table A_ of the one-page PDF, our work consistently outperforms baselines across all tasks. Thus, we believe our work is effective and robust across a wide range of tasks.\n\nGSM8K: https://huggingface.co/datasets/openai/gsm8k\n\nMedical_MMLU: https://huggingface.co/datasets/medalpaca/medical_meadow_mmmlu\n\nmalicious-600k: https://huggingface.co/datasets/bgspaditya/malicious-600k\n\n## Scalability and Real-Time Performance\nIn Table 2 of our manuscript, we evaluated the scalability of our work under different numbers of tasks. The results show a small decrease in LoRA-Inlaid throughput (less than 10%) when tasks increase from 2 to 100, even under three request rate levels. To address the comment, we further increase the number of tasks to 1000, and the throughputs under request rates of 5, 10, 20 are 3.42, 4.02, and 4.22 reqs/s, respectively, which are even better than that of S-LoRA serving only 2 tasks. Thus, we believe LoRA-Inlaid has a sound scalability to support both small and large scale workloads in real-world scenarios.\n\n## Details of the Incremental Quantization Process\nPlease refer to the _Global Responses_.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBSfs8TFRv", "reviewer": "Reviewer_gmzG", "summary": "This paper mainly focuses on the quantization problem of large language models and the problem of low-rank decomposition solvers, and proposes a method for quantizing large language models for multiple tasks and integrating multiple low-rank decomposition solvers. The article first analyzes that the current mainstream quantization method will make it impossible to share model parameters among tasks when processing multiple tasks. Therefore, the current LLM service system cannot integrate LLM quantization with multiple LoRA solvers to achieve memory-efficient multi-task support. At the same time, the existing LLM service system lacks support for dynamic task addition. Therefore, the LoRA-Inlaid proposed in this paper designs a flexible and efficient multi-task quantization algorithm, which is convenient for multiple LoRA solvers to share a single quantization model, greatly reducing the memory consumption of model deployment. On the other hand, LoRA-Inlaid develops a new multi-task scheduling algorithm based on output length prediction and grouping, which effectively reduces memory consumption and avoids frequent switching of LoRA solvers. This innovative multi-task quantization algorithm MLGPTQ uses multi-task data to perform joint quantization on the basic model. This allows the basic model of quantization to be shared between multiple tasks. In addition, it also supports incremental quantization of newly added tasks without affecting the performance of the service.", "questions": "This article is a very excellent and innovative algorithmic article for quantitative model improvement. The algorithm process is clearly explained, the process is clear, and the proof process is very detailed. The proof ideas are clear and complete. However, the following small issues can be considered:\n1. The symbols of the proof process in this article need to be further sorted out, for example, does the q-th parameter refer to the qth parameter or the qth row parameter?\n2. This article mentions that the Lagrange multiplier method needs to be used to solve max-aggregation, but the solution process, including the final result, will have multiple matrix inversions. Then the condition number of the matrix needs to be further considered to see if it will affect the accuracy and speed of the solution.\n3. Figure 2 mentions the difference between the method in this article and the GPTQ method, which is very clear, but this article also proposes other quantitative methods. Can you explain why GPTQ is simply compared with MLGPTQ, or the AWQ method is also represented in this way to show the highlights of this method?\n4. This article has less or less explanation of the multi-task scheduling strategy. It is recommended that the author add more comparative experiments on this module, otherwise the content is relatively lacking as an innovation point.\n5. For large model tasks, there may be some priorities among the tasks. So based on the multi-task scheduling method in this article, how should we consider the situation where there are multiple tasks and the task levels have an order?", "limitations": "This article mentions possible limitations of this method at the end. The first is that the quantitative method in this article does not detect the existence of malicious or poisoned tasks, which may be intentionally used to harm other tasks. Second, our scheduling does not consider fairness among tasks, which may be essential for shared service platforms. \nThird, it only supports language tasks, while requiring some system redesign for multimodal tasks. In addition to the three possible limitations mentioned in the article, the priority of tasks and the order between tasks may also be points that need to be considered. \nSecondly, the calculation time of the intermediate process of MLGPTQ may vary significantly depending on the model. In addition, if some tasks may need to be exited midway during the training process, how should the model proposed in this article handle this situation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes an innovative multi-task LLM quantization method for large language models, which can be used without introducing any other memory calculation consumption.\nIt handles multiple quantification tasks with the following main advantages:\n1. This innovative multi-task quantization algorithm MLGPTQ uses multi-task data to perform joint quantification on the basic model. This allows the quantified base model to be shared across multiple tasks.\n2. Supports incremental quantification of adding new tasks, which improves the defect that most current systems can only support a constant number of tasks. Tasks can be added dynamically without having a great impact on the current task process. No need Pause or restart the current service process.\n3. A multi-task scheduling strategy based on output length prediction and grouping is proposed. This effectively reduces memory consumption and memory swapping overhead in multi-tasking scenarios, significantly improving the overall performance of the system.\n4. LoRA-Inlaid integrates multi-task quantification, realizes dynamic task addition, and adopts a multi-task scheduling strategy to achieve high-performance and flexible computing services for multi-task LLM in an environment with limited resources.", "weaknesses": "This article is a very excellent and innovative algorithmic article for quantitative model improvement. The algorithm process is clearly explained, the process is clear, and the proof process is very detailed. The proof ideas are clear and complete. However, the following small issues can be considered:\n1. The symbols of the proof process in this article need to be further sorted out, for example, does the q-th parameter refer to the qth parameter or the qth row parameter?\n2. This article mentions that the Lagrange multiplier method needs to be used to solve max-aggregation, but the solution process, including the final result, will have multiple matrix inversions. Then the condition number of the matrix needs to be further considered to see if it will affect the accuracy and speed of the solution.\n3. Figure 2 mentions the difference between the method in this article and the GPTQ method, which is very clear, but this article also proposes other quantitative methods. Can you explain why GPTQ is simply compared with MLGPTQ, or the AWQ method is also represented in this way to show the highlights of this method?\n4. This article has less or less explanation of the multi-task scheduling strategy. It is recommended that the author add more comparative experiments on this module, otherwise the content is relatively lacking as an innovation point.\n5. For large model tasks, there may be some priorities among the tasks. So based on the multi-task scheduling method in this article, how should we consider the situation where there are multiple tasks and the task levels have an order?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1\nIt refers to the $q$-th parameter, i.e., $w_q$ denotes the $q$-th element of $\\mathbf{W}$ after it is flattened.\n\n## Q2\nIn Appendix A.2, we implement MLGPTQ using **Cholesky decomposition** to increase speed and computational stability, similar to GPTQ's implementation (https://github.com/AutoGPTQ/AutoGPTQ/blob/v0.7.0/auto_gptq/quantization/gptq.py#L117). To address the comment, we record the condition numbers of the Hessian matrices for different layers, providing the mean and std-dev below. Results show that condition numbers are consistently within $10^3$, which is a reasonable value range in practice.\n\n| | trans-fr | trans-cs | trans-id | trans-nl | trans-da | trans-sw | QTsum | xlsum | tiny-codes |\n| ----- | ------ | -------- | -------- | -------- | -------- | -------- | ----- | ----- | ---------- |\n|mean| 285.1    | 316.3    | 345.7    | 321.0    | 318.3    | 324.1    | 258.3 | 203.2 | 232.9     |\n|std-dev| 101.0    | 108.7    | 94.8     | 107.9    | 108.2    | 103.9    | 92.2  | 92.9  | 89.5       |\n\n## Q3\nOur work centers on GPTQ due to its widespread use, but our solution can also adapt to AWQ. We presented the differences between MLAWQ and $AWQ_{tweaked}$ in _Alg A_ of the one-page PDF (similar to Alg 1 of our manuscript). As introduced in Section 3.1, most quantization methods follow the **_Forward-Aggregate Info-Modify Weight-Quant_** paradigm. In essence, $AWQ_{tweaked}$ smooths outliers by multiplying weights with a smoothing factor, best_s, to minimize per-channel quant error:\n\n- In **_Forward_**, the input is multiplied by the weights to create an unquantized monitor, guiding min error quantization (line 1). \n- ​In **_Aggregate Info_**, the average of all samples and weights is calculated for each channel (lines 2&3) to determine smoothing factor s (line 7). $W$ and $X$ are smoothed to remove outliers (lines 8&9). Then, smoothed $W$ is pseudo-quantized (quantize-then-dequantize to simulate round loss) and compared to the unquantized monitor for quantization error (line 10). This process iterates over various ratios (line 6), selecting the factor with the smallest error as best_s (line 11). Then, this best_s is used to **_Modify the weight_** (line 12), followed by the **_Quant_** (line 13) process using the modified weight.\n\nThe drawbacks discussed in lines 146-154 of Section 3.1 also exist for $AWQ_{tweaked}$ in multi-task quantization:\n\n- **_Forward_** (line 1): It can't pass LoRA adapters during activation distribution simulation, causing quantization bias during inference.\n- **_Aggregate Info_**: It uses $X_{mean}=X.mean(0)$, a naive mixed average of multi-tasks' info. Since each task affects each channel differently, simply averaging blurs distributions, ignoring individual effects.\n\nAs explained in lines 155-172 of Section 3.1 and shown in Fig 2, MLGPTQ mainly improves the first three steps to tackle GPTQ's issues in multi-task scenarios. Similarly, we can fix AWQ's issues to create a better multi-task quantization algorithm MLAWQ:\n\n- **_Forward_**: MLAWQ loads corresponding LoRA adapter for each task to participate in forward propagation, accurately simulating real activation distribution (line 1).  \n- **_Aggregate Info_**: Instead of mixing and averaging features of each column across all tasks to compute $s$, MLAWQ computes the average for each task separately to get $s_i$ (line 3&7). Then it calculates quantization error for each column rather than the entire matrix (line 10). If the $i$-th task results in the smallest quantization error for the $j$-th column, it sets best_s$[j]=s_{i}[j]$ (line 11). This approach allows optimal error minimization, showing each task's individual effect on different channels, enhancing **_Aggregate Info_** (lines 3&6-11), and improving the **_Modify Weight_** (line 12) and **_Quant_** (line 13) processes.\n\nIn summary, our work identifies common drawbacks of current single-task quantization methods in multi-task scenarios. By addressing these issues, we can develop more precise multi-task quantization algorithms. Due to time constraints, we couldn't complete the coding and experiments for MLAWQ during rebuttal. However, since the backbone algorithm choice is orthogonal to our work, and GPTQ is popular in LLM, we believe it doesn't diminish our work's significance.\n\n## Q4\nAs discussed in Section 3.3 and _Global Responses_, our multi-task scheduling strategy involves two key techniques: prediction-based SRTF and task grouping. Ablation studies in Figure 9 show that prediction-based SRTF and task grouping increase SLO Attainment by 2.27x and 1.16x, respectively. To address the comment, in _Fig C_ of the PDF, we conducted two more experiments:\n\n- We considered a variant of LoRA-Inlaid, enabling multi-task scheduling strategy while disabling multi-task quantization (i.e., served model not quantized), denoted as \"Ours (w/o quant).\" Results show \"Ours (w/o quant)\" increases SLO attainment by 16% compared to S-LoRA, showing the power of our scheduling strategy.\n- We considered a variant of LoRA-Inlaid without output length prediction (instead, using averaged output length of all requests in the same task for sorting), denoted as \"Ours (w/o prediction).\" Results indicate 18% and 27% efficiency improvements (\"Ours\" vs. \"Ours (w/o prediction)\") for 7B and 13B models, respectively.\n\n## Q5\nAs discussed in Section 5 and Appendix F, our work does not consider task fairness currently and we leave it as a future work. When tasks have different priorities, it becomes a case of weighted fairness. A potential solution is using Weighted Fair Queueing (WFQ) [a]. For example, compute a weighted combination of input and output tokens for each task and scale it by the task's priority factor. This information can be seen as the service amount each task receives. Then, we can integrate WFQ to schedule different tasks.\n\n[a] Parekh and Robert. A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Responses

We are grateful to all reviewers for the careful reviews. We provide _Global Responses_ to common questions, followed by individual responses. Please refer to the attached one-page PDF for related figures and tables.  

## Details of the Incremental Quantization and Multi-task Scheduling

We acknowledge the oversight that these two modules are not introduced comprehensively in the main text of our manuscript due to the space constraint. To address the reviewers' concerns, we would like to elaborate below.

### **Incremental Quantization**

In Appendix A.2 of our manuscript, we provided the full quantization process of MLGPTQ in Alg 1. To quantize each model weight $W$ with $T$ tasks, there are two steps: 

- [Lines 1-2 of Alg 1] Compute the Hessian matrices for all tasks (i.e., compute $H_1^{-1},H_2^{-1},\cdots,H_T^{-1}$).
- [Lines 3-13 of Alg 1] Max-aggregate these $T$ Hessian matrices (i.e., $H_{tmp}=MaxAgg(H_1^{-1},\cdots,H_T^{-1})$)  and update the model weight.

When there are new tasks, a naive solution is to perform full quantization again. Denote $T_1,T_2$ as numbers of existing and new tasks, respectively. The naive solution runs the two steps above with $T=T_1+T_2$. However, this leads to **redudant computation** of $H_1^{-1}, H_2^{-1}, \cdots, H_{T_1}^{-1}$. Furthermore, given the **commutative property of the max-aggregation operation**, we have 

$MaxAgg(H_1^{-1},\cdots, H_{T_1+T_2}^{-1})=MaxAgg(MaxAgg(H_1^{-1},\cdots, H_{T_1}^{-1}),MaxAgg(H_{T_1+1}^{-1},\cdots,H_{T_1+T_2}^{-1})),$

where the first term $MaxAgg(H_1^{-1},\cdots,H_{T_1}^{-1})$has already been computed as $H_{tmp}$ in the previous quantization. 

Inspired by this, we can cache $H_{tmp}$, so the incremental quantization can be done as follows:

- [Same as lines 1-2 of Alg 1] Compute the Hessian matrices for new tasks $H_{T_1+1}^{-1},\cdots,H_{T_1+T_2}^{-1}$.
- [Same as lines 3-13 of Alg 1] Max-aggregate the $T_2+1$ matrices (i.e., $H_{T_1+1}^{-1},\cdots,H_{T_1+T_2}^{-1}$ and the cached $H_{tmp}^{(cached)}$) and update the model weight.

Obviously, incremental quantization with $T_2$ tasks is identical to full quantization with $T_1+T_2$ tasks, while **avoiding redundant computation**.

To avoid halting the ongoing services, LoRA-Inlaid spawns a background thread for incremental quantization. In addition, to reduce the memory consumption of incremental quantization, it is done in a **layer-by-layer** manner: for each (unquantized) model weight, we load it from CPU memory to GPU memory, perform incremental quantization, remove it from GPU memory, and proceed to the next model weight. The IO between CPU-GPU is overlapped with computation. Thus, LoRA-Inlaid supports seamless task addition on the fly and has very little influence on the ongoing services.

### **Multi-task Scheduling**

In Appendix B of our manuscript, we illustrated multi-task scheduling in Alg 2~4 and discussed its workflow in lines 529-542. For better understanding, we included a flow chart in _Fig A_ of the one-page PDF.

Given the prompt of a request, there are two phases for LLM inference: _the prefilling phase_ takes the prompt to compute the key-value (KV) cache and generates the first output token in a single step, and _the decoding phase_ takes the last generated token and KV cache to generate subsequent tokens. The decoding phase needs to be executed for multiple steps for each request, with each step generating only one token, until the EOS token is generated.

In the field of LLM serving, the scheduling strategy is responsible for determining what should be done in each scheduling step. There are three choices for each step: 

- [lines 7-11&16-20 of Alg 2] generate a batch from the prefilling queues;
- [line 22-25 of Alg 2] schedule a batch from the decoding queues;
- [lines 27-28 of Alg 2] continue decoding for the current batch.

The determination in our work follows the standard rules in the field of LLM serving (e.g., switch to prefilling if we have done decoding for several consecutive steps). 

To enhance multi-task scheduling, there are two key techniques in our work (i.e., the two solutions in Section 3.3) when generating/scheduling a batch.

**Scheduling Guided by Output Length Prediction**: Considering the sequence length variation across tasks, we take the remaining output length information into account.
-  [lines 214-216 of Section 3.3] Upon receiving a new request, we predict its output length on CPU using a small model (255MB) and enqueue the request into the _prefill_reqs_. Note that the output length prediction takes about 16ms for one request on CPU, while it takes about 200ms to finish the inference of one request on GPU. Hence, we can completely overlap the prediction, without occupying any GPU computing resources.
-  [lines 2-3 of Alg 3 & lines 2-3 of Alg 4, also in lines 217-219 of Section 3.3] For each scheduling step, we sort the queues to achieve the Shortest Remaining Time First (SRTF).

**Reducing Tasks Involved via Grouping**: To avoid expensive memory access overhead in each step, there are two efforts.
- [lines 6&11 of Alg 3, also in lines 233-234 of Section 3.3] We restrict the number of involved tasks below a threshold (i.e., $\beta$ in Section 3.3) when generating a new batch from the prefilling queues;
- [lines 6&11 of Alg 4, also in lines 235-236 of Section 3.3] We prioritize tasks involved in the previous step to avoid swapping LoRA adapters frequently.

Besides, our work also maintains the waiting time of each request to avoid starvation (i.e., by the hungry queues in Alg 2-4), which is common in LLM serving. 

To conclude, the scheduling in our work is not complex. Compared with existing scheduling strategies, we leverage two techniques considering the characteristics of multi-task serving. Thus, in our manuscript, we focused on the two techniques in Section 3.3, while the detailed routine (e.g., the determination of the three choices for each step) is deferred to the appendix.

Meta-review: The paper introduces LoRA-Inlaid, an innovative multi-task large language model serving system that optimizes memory usage and supports dynamic task addition through novel quantization and scheduling algorithms. Reviewers praised its practical impact and theoretical contributions but raised concerns about computational overhead, clarity, and limited empirical validation. Authors responded with detailed clarifications and additional experiments, effectively addressing most concerns. The reviewers' consensus was positive, recommending acceptance due to the system's significant advancements in LLM deployment.  The AC concurs with this recommendation and the paper may be accepted to NeurIPS.
Decision: Accept (poster)