Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters

Authors: yifei  xia
fangcheng  fu
wentao  zhang
jiawei  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:32:15

Abstract: With the remarkable achievements of large language models (LLMs), the demand for fine-tuning and deploying LLMs in various downstream tasks has garnered widespread interest. Parameter-efficient fine-tuning techniques represented by LoRA and model quantization techniques represented by GPTQ and AWQ are of paramount significance. However, although these techniques have been widely adopted in single-task scenarios, research is scarce in multi-task scenarios. To be specific, we find that mainstream quantization methods would prevent the base LLM from being shared among tasks, so current LLM serving systems are infeasible to integrate LLM quantization with multiple LoRA adapters to achieve memory-efficient multi-task serving. Moreover, existing LLM serving systems lack support for dynamic task addition and overlook the workload differences among tasks, leading to inefficiencies in multi-task scenarios.

This work proposes LoRA-Inlaid, an efficient multi-task LLM serving system. On the one hand, LoRA-Inlaid designs a flexible and efficient multi-task quantization algorithm (MLGPTQ) that facilitates the sharing of a single quantized model for multiple LoRA adapters, which significantly reduces the memory consumption for model deployment. Meanwhile, it supports adding LoRA adapters for new tasks on the fly, without sacrificing the stability of online services. On the other hand, LoRA-Inlaid develops a novel multi-task scheduling algorithm guided by output length prediction and grouping among different tasks, which effectively shrinks the memory consumption and avoids frequent switching of LoRA adapters. Empirical results verify that LoRA-Inlaid outperforms existing state-of-the-art LLM serving systems by up to 1.58 times in terms of throughput, 1.76 times in terms of average latency, 2 times in terms of job completion time, and 10 times in terms of SLO Attainment, while maintaining the same level of model quality.

Main paper: 
 introduction: 
 Large language models (LLMs) have demonstrated impressive effectiveness in various domains [26, 30, 31, 39] , and the demand of deploying LLMs in downstream tasks continues to grow [4, 10, Yifei Xia, Fangcheng Fu, and Bin Cui are with the School of Computer Science and Key Lab of High Confidence Software Technologies (MOE), Peking University. Bin Cui is also with the Institute of Computational Social Science, Peking University (Qingdao). Wentao Zhang is with the Center for Machine Learning Research, Peking University. Jiawei Jiang is with the School of Computer Science, Wuhan University. Multi-task Scheduling vLLM [19] & TensorRT-LLM [29] ✗ ✗ ✗ ✗ S-LoRA [35] & Punica [5] ✓ ✗ ✗ ✗ LoRA-Inlaid (this work) ✓ ✓ ✓ ✓ 20, 24, 34, 43, 44, 46, 47] . Given the explosive increase in model size and the limitations of hardware resources, "parameter-efficient fine-tuning" (PEFT) and "quantization-then-deployment" have become the most common pathways for deploying LLMs in downstream tasks [50] . On the one hand, PEFT techniques, represented by LoRA (Low-Rank Adaptation) [16] , only train small-scale adapters to adapt the base model to a specific task, significantly reducing the cost of model fine-tuning.On the other hand, low-bit quantization techniques like GPTQ and AWQ [12, 22] can substantially reduce the memory requirements of model deployment and alleviate memory access overhead during inference, while maintaining model quality.Although mainstream LLM serving systems like vLLM and TensorRT-LLM [19, 29] have integrated support for the quantized deployment of fine-tuned models, these systems focus on single-task serving scenarios. With the rising demand for various downstream tasks, efficiently supporting multi-task servicing scenarios has become increasingly crucial. This has led to the emergence of multi-task serving systems supporting multiple LoRA adapters concurrently, such as S-LoRA and Punica [5, 35] . These systems share a unified base model across different tasks and activate different LoRA adapters based on the incoming requests, enabling the simultaneous processing of multiple tasks in a single batch. However, in multi-task scenarios, existing systems still face three major challenges.First, existing multi-task serving systems cannot effectively incorporate mainstream model quantization methods such as GPTQ and AWQ. Specifically, these quantization methods require calibration of numerical distributions using task-specific datasets, and the quantization process for each task necessitates activating the corresponding LoRA adapter. Consequently, the base models after quantization are divergent across different tasks, and thus it is infeasible to share a unified quantized model. This limitation leads to performance deficiencies or even unavailability in resource-constrained scenarios.Second, in practical multi-task serving scenarios, it would be necessary to add new tasks in real time. However, existing systems only support a static number of tasks and are incapable of dynamically adding LoRA adapters. More importantly, after a quantized model is deployed, current solutions do not support any subsequent quantization and deployment for new tasks without affecting the existing tasks. In contrast, adding new tasks typically requires suspending and restarting the serving process, which severely harms the stability and robustness of online services.Third, incoming requests for different tasks inevitably have workload variations (such as request length, processing time, etc.) and require loading different LoRA adapters for processing. Existing systems overlook these issues during the scheduling for multi-task requests, and thus necessitate loading a large number of adapters in a single scheduling step as well as frequently switching adapters between adjacent scheduling steps, leading to significant efficiency degradation.To address these challenges, we develop LoRA-Inlaid, a resource-efficient and high-performance system for multi-task LLM serving. The main contributions of this paper are as follows.To begin with, we propose an innovative multi-task quantization algorithm termed MLGPTQ (Multi-LoRA GPTQ), which utilizes multi-task data to perform joint quantization on the base model. This allows the quantized base model to be shared across multiple tasks. In addition, it supports incremental quantization for newly added tasks without impacting the performance of online services.Subsequently, we introduce a novel multi-task scheduling strategy based on output length prediction and grouping. This effectively reduces memory consumption and memory swapping overhead in multi-task scenarios, significantly enhancing overall system performance.Based on these two techniques, we develop a brand new multi-task LLM serving system, namely LoRA-Inlaid. As shown in Table 1 , LoRA-Inlaid integrates multi-task quantization, enables dynamic task addition, and employs the multi-task scheduling strategy, achieving high-performance and flexible multi-task LLM serving in resource-constrained environments.Finally, experimental results demonstrate that, compared to existing systems, LoRA-Inlaid can increase throughput by up to 1.58×, reduce average latency and job completion time by up to 1.76× and 2×, improve SLO Attainment by up to 10×, and support larger-scale language models under the same resource constraints, all while maintaining nearly the same level of model quality. 
background and related works: 
 Low-Rank Adaptation. LoRA [16] , short for Low-Rank Adaptation, is one of the most widely used parameter-efficient fine-tuning (PEFT) techniques. Unlike full-parameter fine-tuning, LoRA fine-tunes only a small adapter, which consists of much fewer parameters than the base model, significantly reducing the training cost. The key idea behind LoRA is that the fine-tuning process should only introduce small changes to the weight matrix of the base model (denoted by W ∈ R m×n ), so we can learn two small, low-rank matrices (denoted by A ∈ R r×n , B ∈ R m×r where r ≪ m, n), and approximate such changes with the product of two matrices (i.e., ∆W ≈ BA).Low-bit Quantization. Low-bit quantization [7, 8, 12, 22, 23, 42] shrinks the model size effectively and thus reduces the memory requirement when deploying the model. In addition, it usually helps to improve efficiency by decreasing the memory access overhead of the model weights. Consequently, it has been widely adopted in LLM serving. There are various quantization paradigms, with post-training quantization (PTQ) being among the most popular ones. Typically, PTQ computesX INT = Round(α Clip(X R /α, Q min , Q max )), where X R represents the real-valued parameters before quantization, X INT represents the parameters after quantization to integers, Q min and Q max denote the minimum and maximum values of the quantization range, and α represents the scaling factor. Various PTQ methods calculate the quantization knobs like α with diverse approaches or implement different approximation methods. While mainstream PTQ methods (e.g., GPTQ [12] , AWQ [22] ) have a common ground that they need to calibrate the numerical distribution based on a small task-specific dataset (a.k.a. the calibration set), since numerous studies have revealed the accuracy after quantization with dataset calibration is usually significantly higher than that without dataset calibration [17] . Therefore, this paper focuses on quantization with dataset calibration.Scheduling in LLM Serving. With the explosive applications of LLMs, more and more studies try to evolve the scheduling strategies in LLM serving for better performance. Early systems like FasterTransformer [28] rely on request-level scheduling. Notably, Yu et al. [45] introduced Orca, the first iteration-level scheduling with first-come-first-serve (FCFS) order for better batching. Building on this, mainstream LLM serving systems leverage various batching approaches, such as continuous batching in vLLM [19] and in-flight batching in TensorRT-LLM [29] . FastServe [40] takes the semi-information of requests (e.g., input length, processed time, etc.) into account and tries to minimize average job completion time. However, none of these scheduling strategies consider the characteristics of multi-task scenarios, as we will discuss in §3.3.Multi-task Serving Systems. Since the LoRA fine-tuning technique keeps the base model unaltered, it is feasible to share the same base model across multiple LoRA adapters, so that we can serve requests from multiple tasks within a single batch. Punica [5] and S-LoRA [35] are two notable multi-task serving systems, putting forward the initial efforts to support multi-task LLM serving with multiple LoRA adapters. Specific optimization techniques are proposed. For instance, the Segmented Gather Matrix-Vector (SGMV) kernel is developed to enhance memory and computation efficiency when processing requests from different tasks together. In addition, to allocate more GPU memory to intermediate results (typically, KV cache), existing systems maintain the LoRA adapters in CPU memory and only preserve a relatively small number of LoRA adapters in GPU memory. When a LoRA adapter outside GPU memory is needed, it is necessary to perform memory swapping between the CPU and GPU memory.3The overview of LoRA-Inlaid is depicted in Figure 1 . Given an LLM with multiple LoRA adapters for various downstream tasks, LoRA-Inlaid initiates a joint quantization process ( §3.1), which produces a unified quantized base model that can be shared across the adapters. During the online serving, if new tasks are to be included on the fly, LoRA-Inlaid facilitates a dynamic task addition process ( §3.2) that efficiently conducts incremental re-quantization and seamlessly deploys the added tasks. Furthermore, LoRA-Inlaid employs a multi-task scheduling strategy ( §3.3) that takes the workload differences into account for better efficiency. 
multi-task joint quantization: 
 As introduced in §2, mainstream quantization methods require task-specific datasets for calibration. In addition, they mostly follow the Forward-Aggregate Info-Modify Weight-Quant paradigm in Figure 2 . This paradigm first simulates the activation distribution for a given task through Forward propagation and Aggregates Information of this specific task. Subsequently, it uses the aggregated information to Modify model Weights to adapt to the task. Finally, the quantization knobs like scales α are calculated based on the modified weights to Quantize the base model. However, in multi-task scenarios, since different tasks should provide diverse calibration sets and necessitate unique LoRA adapters for computation, the quantized models of different tasks are inevitably divergent. Intuitively, if we wish to tweak existing quantization methods to make the quantized model shareable across tasks, we should quantize the model without any LoRA adapters. In addition, we should either (i) quantize the model without calibration or (ii) quantize the model with a mixed calibration set consisting of the datasets from all tasks.However, these approaches fail to accurately capture the unique numerical distribution of each task, and suffer from severe accuracy loss (as evaluated in §4.2). Below we first elaborate on the reason why these approaches fail with the widely used GPTQ [12] and then propose our solution1 .Drawbacks of GPTQ in multi-task scenarios. Directly applying GPTQ in multi-task scenarios has the following drawbacks. First, as aforementioned, GPTQ can only quantize the model without any LoRA adapters, which is infeasible to accurately capture the correct activation information for multiple tasks during Forward. Second, in Aggregate Info, since the calibration sets from all tasks are mixed, GPTQ simply accumulates the information from different tasks into one Hessian matrix, making each task's specific information diluted and losing the emphasis on critical information from different tasks. Third, in Modify Weight, GPTQ relies on the naïve, mix-aggregated Hessian matrix, overlooking the varying importance across tasks, which results in suboptimal outcomes. These drawbacks make the direct application of GPTQ in multi-task scenarios ineffective.Our MLGPTQ (Multi-LoRA GPTQ) Algorithm To address these drawbacks, we propose a multitask quantization algorithm termed MLGPTQ. Our algorithm enables joint quantization of multiple tasks to retain only one quantized base model, while effectively maintaining the model accuracy by capturing the numerical distributions of all tasks. The goal of MLGPTQ is to minimize the errors of activations before and after quantization, i.e., arg min where T denotes the number of tasks, A t and B t are the low-rank adapter matrices of the t-th task, X t is the input of t-th task, and W and Q(W) denote the original and quantized weights of a layer.EQUATIONAs shown in Figure 2 , During Forward, MLGPTQ loads the corresponding LoRA adapters based on each task, accurately computing the activations. In Aggregate Info, unlike GPTQ's naïve mixaggregation that disrupts task-specific information, MLGPTQ derives the max-aggregation to solve the objective in Eq. 1 (the derivation can be found in the Appendix A), which has the following form:EQUATIONwhere H t denotes the Hessian matrix of the t-th task,w q is the q-th parameter in W. To be formal, there are primarily two steps in Aggregate Info. First, it calculates the Hessian matrix information for each task individually (i.e., compute {H -1 t } T t=1 ) Second, it aggregates the most important information from each one into a max-aggregated Hessian matrix (i.e., Htmp = MaxAgg({H -1 t } T t=1 )). In Modify Weight, MLGPTQ utilizes the max-aggregated Hessian matrix to adjust the weights according to Eq. 2. Finally in Quant, we utilize the modified weights for quantization. Due to space constraints, we only present the core concept of MLGPTQ here. Interested readers are referred to Appendix A for a complete derivation as well as the detailed algorithm.In real-world online services, there is a need for dynamic task addition (i.e., adding new LoRA adapters). In single-task scenarios, adding new tasks typically requires launching more services with extra hardware resources, which does not affect the services for existing tasks. In multi-task scenarios, there would be interference since all tasks share the same base model. However, we find that none of the existing multi-task serving systems address this problem, lacking a proper solution.Nevertheless, adding new LoRA adapters on the fly in LoRA-Inlaid is inherently far from trivial since the multi-task quantization poses two challenges: (1. Unseen Distributions) Since the MLGPTQ algorithm is invoked before the new tasks are involved, the quantized model has not captured the distribution information about the new tasks, making it infeasible to work with the new LoRA adapters directly. (2. Serving Interruption) Directly re-quantizing the model requires a substantial amount of memory, so it necessitates pausing the ongoing serving for a while to reserve available space for re-quantization, harming the stability of online services. To support dynamic task addition in multi-task scenarios, LoRA-Inlaid tackles these two obstacles, respectively.To capture the information of new tasks, a naïve solution is to perform full quantization once there are new tasks. Denote T1, T2 as numbers of existing and new tasks, respectively. The naïve solution runs the two steps of Aggregate Info above with T = T1 + T2. However, this leads to redundant computation of {H -1 t } T 1 t=1 . In addition, given the commutative property of the max-aggregation operation, we have MaxAgg({H -1t } T t=1 ) = MaxAgg(MaxAgg({H -1 t } T 1 t=1 ), MaxAgg({H -1 t } T 2 t=T 1 +1), where the first term MaxAgg({H -1 t } T 1 t=1 ) has already been computed as Htmp in the previous quantization. Inspired by this, LoRA-Inlaid caches Htmp so that the incremental quantization can be done as follows. In Forward, it capture the activation information of new task Info, it computes the Hessian matrices for new tasks{H -1 t } T 2t=T 1 +1 , and then max-aggregates the T2 + 1 matrices (i.e., {H -1 t } T 2 t=T 1 +1 and the cached H (cached) tmp ). At last, it performs Modify Weight and Quant, as introduced in §3.1. By doing so, incremental quantization with T 2 tasks is identical to full quantization with T 1 + T 2 tasks, while avoiding redundant computation.To avoid halting the ongoing services, LoRA-Inlaid spawns a background thread for incremental quantization. Moreover, it is done in a layer-by-layer manner to reduce the memory consumption -for each (unquantized) model weight, we load it from CPU memory to GPU memory, perform incremental quantization, remove it from GPU memory, and proceed to the next model weight. The IO between CPU-GPU is overlapped with computation. Thus, LoRA-Inlaid supports seamless task addition on the fly and has very little influence on the ongoing services, as evaluated in §4.4.Putting them together, LoRA-Inlaid develops an asynchronous, layer-wise re-quantization mechanism, which accomplishes incremental quantization with the new tasks and cached Hessian matrices asynchronously, without interrupting the serving.Despite extensive research on scheduling strategies for LLM serving, these approaches primarily focus on single-task serving, leaving the unique characteristics in the multi-task scenarios neglected. Below we analyze two limitations of existing scheduling strategies in multi-task serving. Besides, due to the space constraint, we briefly introduce the corresponding solutions in LoRA-Inlaid, while leaving the details of our multi-task scheduling algorithm in Appendix B.Limitation 1: Divergent Output Length Distributions Leading to High Average Completion Time. As shown in Figure 3 , the distributions of input and output lengths vary significantly across different tasks, while requests of the same task exhibit clustering effects. Current strategies mainly rely on semi-information (e.g., input length, processed time, etc.) to make the scheduling decisions, but do not consider the information of output length since it is not the prior knowledge. Intuitively, this may work fine for single-task scenarios where the vast majority of requests fall within the same workload and thus the clustering effect exists. However, it is unsuitable for multi-task scenarios due to the divergent output length distributions across different tasks. Eventually, we find that existing scheduling strategies suffer from heavy performance degradation when applied to multi-task serving.Solution 1: Scheduling Guided by Output Length Prediction. Existing research has shown that the output lengths can be accurately predicted by a small, distilled model given the requests [49] . Inspired by this, we leverage a number of small models, to predict the output lengths of incoming requests. Particularly, upon receiving a new request, we predict its output length on CPU using a small model (255MB). Note that the output length prediction takes about 16 milliseconds for one request on CPU, while it takes about 200 milliseconds or more to finish the inference of one request on GPU. Hence, we can completely overlap the prediction, without occupying any GPU computing resources. Based on the predictions, we employ a Shortest Remaining Time First (SRTF) scheduling, which prioritizes requests with the shortest remaining processing time and has been proven to minimize the average completion time in the field of job scheduling [37] .Limitation 2: Excessive Tasks Involved in each Step Leading to Expensive Memory Access Overhead. Due to the randomness and dynamicity of request arrivals, multiple tasks are to be scheduled in each step. However, owing to the lack of consideration upon the task for each request, existing scheduling Table 2 : Model quality of different approaches under different tasks (std-dev given in parentheses). GPTQ and AWQ are in gray background color since the quantized models produced by them cannot be shared across different tasks. We mark the best multi-task quantization approaches (i.e., the best among MLGPTQ, GPTQtweaked, AWQtweaked, and RTN) in bold. strategies typically involve a great number of tasks, and the system has to load lots of LoRA adapters in each step, as shown in Figure 4 . As it is well known that the decoding phase of LLM inference is usually bounded by the memory bandwidth, the need for loading more LoRA adapters further exacerbates this issue. Worst still, we observe that the sets of involved tasks vary significantly between consecutive steps. As introduced in § 2, multi-task serving systems only preserve limited GPU memory space for LoRA adapters, and swap them between CPU and GPU memory when necessary. Consequently, existing scheduling strategies force the system to frequently swap LoRA adapters, rendering performance degradation due to the expensive memory swapping overhead.Solution 2: Reducing Tasks Involved via Grouping. To address the limitation, we adopt a simple yet effective grouping-based approach, which partitions requests into groups according to their tasks to guide scheduling. On one hand, to avoid involving excessive tasks in each step, we set a grouping coefficient β (10 by default) and keep the number of involved tasks below β in each step. On the other hand, to alleviate the memory swapping overhead, we prioritize tasks involved in the previous step, aiming to use the LoRA adapters for more consecutive steps. In addition, we maintain a starvation queue based on the waiting time to get rid of starvation, striking a good balance among the tasks. Datasets and Workloads. For accuracy tests, we consider 12 tasks in total, including six translation tasks [38] (trans-fr, trans-cs, trans-id, trans-nl, trans-da, trans-sw), one text summarization task [14] (xlsum), one table summarization task [48] (QTsum), one code generation task [13] (tiny-codes), one math QA task [6] (GSM8k), one medical QA task [2] (med-qa), and one malicious detection task [1] (malicious). Detailed descriptions are provided in Appendix C.2. For efficiency tests, we follow prior works [19, 35] to generate different levels of request rates using the Gamma process.Models. We conduct experiments with LLaMA2-7B and LLaMA2-13B [25] . For accuracy tests, open-source fine-tuned models are used, as detailed in Appendix C.2. For efficiency tests, following S-LoRA [35] , we consider LoRA adapters with different ranks (8, 16, 32, 64) across the served tasks to simulate diverse serving scenarios.Metrics. For the accuracy test, we focus on SacreBLEU (S_BLEU) [18] , ROUGE [21] and Accuracy (Acc). Details of these metrics are provided in Appendix C.3. For efficiency tests, the considered metrics include throughput, average request latency, job completion time (JCT), and SLO (service level objective) Attainment [33] (the percentage of requests completed within the expected latency). By default, we test the serving for 1 minute, with an expected latency of 6 seconds.Baselines. For accuracy tests, we focus on two kinds of baselines introduced in §3.1, i.e., (i) the floating-point round-to-nearest quantization without calibration (denoted as RTN), and (ii) GPTQ [12] and AWQ [22] with a mixed calibration set from all tasks (denoted as GPTQ tweaked and AWQ tweaked ).To be fair, these baselines quantize the base model without any LoRA adapters to ensure the quantized model is shareable. Besides, to facilitate the comparison to single-task quantization, we further consider GPTQ and AWQ for each task individually with the corresponding calibration set and LoRA adapter (denoted as GPTQ and AWQ), which cannot generate a shareable quantized model though.For efficiency tests, we compare LoRA-Inlaid with vLLM [19] and S-LoRA [35] . For vLLM, we quantize each model into 4-bit and launch multiple processes (each with one quantized model) on the same GPU to achieve multi-task serving. For S-LoRA, which does not support deploying quantized models, we deploy one half-precision (16-bit) model and let multiple LoRA adapters share it. For LoRA-Inlaid, we use 4-bit quantization in all efficiency tests. Note that although the model is quantized, the computation during inference is still executed in half-precision (i.e., the model weights are dequantized before computation). 
model quality after quantization: 
 We first assess the model quality after quantization of different approaches. We also conduct experiments to anatomize the effectiveness of MLGPTQ.To be more comprehensive, five additional metrics (ROUGE1, ROUGE2 [21] , NIST_MT [9] , METEOR [9] , and G_BLEU [41] , detailed in Appendix C.3) for evaluating machine translation quality are included. In addition, we further consider a variant of MLGPTQ termed MLGPTQ no_target , which intentionally excludes the calibration set and LoRA adapter of a target task during quantization (the other tasks are not affected).According to the results in Figure 5 , we point out that the effectiveness of MLGPTQ stems from two factors: 1 ⃝ whether the information (e.g., activation and Hessian matrix) of each task is correctly captured, and 2⃝ whether the data distribution of each task is involved during the forward pass of quantization. In particular, GPTQ tweaked (missing 1 ⃝) and MLGPTQ no_target (missing 2 ⃝) exhibit higher accuracy drops compared to MLGPTQ and GPTQ (both fulfilling 1 ⃝ and 2 ⃝) in almost all metrics. These results verify that the design of MLGPTQ fits multi-task quantization well.Throughput, Latency, and JCT. We evaluate the system performance with various numbers of tasks and request arrival rates. The throughput, latency, and JCT are shown in Figure 6 . Overall, LoRA-Inlaid consistently outperforms S-LoRA and vLLM. S-LoRA fails to serve LLaMA2-13B due to the lack of support for deploying quantized models. Since vLLM necessitates launching multiple processes to achieve multi-task serving, the memory consumption grows linearly w.r. SLO Attainment. We also assess the SLO Attainment under different serving loads by varying the request rates and maximum request lengths. The results are shown in Figure 7 . In short, compared to S-LoRA and vLLM, LoRA-Inlaid improves the SLO Attainment by 3.9×, 8.5× on average, and up to 10×, 38×, respectively. Furthermore, we observe that as the request rate or maximum sequence length increases, S-LoRA and vLLM experience a steep decline in SLO Attainment while LoRA-Inlaid does not. This demonstrates the excellent adaptability of LoRA-Inlaid to various serving loads.Scalability. We investigate the scalability w.r.t. number of tasks. As shown in Table 3 , vLLM suffers from significant performance decline, dropping by 56%-73% when the number of tasks increases from 2 to 4, and eventually encountering out-of-memory (OOM) errors when the number of tasks reaches 5. In contrast, under all experimented request rates, the throughput of LoRA-Inlaid hardly declines, even with 1000 tasks served simultaneously. S-LoRA also supports a large number of tasks, while LoRA-Inlaid consistently achieves better performance across all kinds of workloads.Ablation Studies of Multi-task Scheduling and Multi-task Quantization. We compare different scheduling strategies on LoRA-Inlaid. The results are shown in the left of Figure 8 . "Ours (w/o group)", "Ours (w/o prediction)" and "Ours (w/o SRTF)" represent three variants of our multi-task scheduling strategy without task grouping, without output length prediction and without the predictionbased SRTF, respectively. "FIFO" is the strategy adopted in S-LoRA and vLLM, and "Skip-join MLFQ" represents the strategy in FastServe [40] . It is evident that our multi-task scheduling strategy achieves the best performance in terms of SLO Attainment. The designs of task grouping, output length prediction, and SRTF increase the SLO Attainment by 1.16×, 1.23× and 2.27× on average, respectively. We also explore the individual impact of multi-task quantization as shown in the right of Figure 8 . Specificically, we consider a variant of LoRA-Inlaid, which disables quantization (i.e., the served model is not quantized), denoted as "Ours (w/o quant)". The results show that multi-task quantization brings 39% improvement ("Ours" vs. "Ours (w/o quant)") when serving the 7B model. Additionally, without quantization, it will lead to OOM when serving the 13B model. Dynamic Task Addition. We evaluate the ability of dynamic task addition in LoRA-Inlaid by adding 1, 5, and 10 tasks to a heavily loaded service on the fly. The results in Figure 9 show that the throughput undergoes 10%-13% of degradation during the task addition, regardless of the number of tasks added. This is worthwhile given that the online service need not be interrupted. Meanwhile, to evaluate the time consumption of dynamic task addition, we conducted an experiment where there are 5 tasks in the ongoing service and another 5 tasks need to be added. We measured the time cost of three approaches: "Full Quant", which halts the serving and performs full quantization with 10 tasks, "Incr Quant offline", an offline variant (which halts the serving) of our incremental quantization on the 5 new tasks without layer-by-layer quantization, and "Incr Quant", our incremental quantization with the 5 new tasks, which works concurrently with the ongoing service. As shown in Table 4 , by avoiding the redundant computation, the time cost of forward process and calculation of Hess matrix can be reduced greatly, accelerating quantization. Moreover, although the layer-by-layer mechanism slows down the quantization by 1.26 × due to the extra IO, it reduces the memory greatly and does not halt the serving. These empirical results validate the flexibility and robustness of LoRA-Inlaid for multi-task serving. 
conclusion and limitations: 
 In this work, we focused on LLM serving in multi-task scenarios and developed a multi-LoRA task serving system, namely LoRA-Inlaid. On one hand, we designed a flexible and efficient dynamic multi-task quantization algorithm that supports the joint quantization of models for multiple tasks, significantly reducing the memory requirements for model deployment. We also facilitated realtime dynamic task addition, enhancing the stability and flexibility of online services. On the other hand, we introduced a novel multi-task scheduling strategy based on output length prediction and grouping, effectively resolving the issues of high memory overhead and frequent memory swapping when applying existing strategies in multi-task scenarios. Extensive experiments demonstrated that LoRA-Inlaid significantly outperforms existing LLM serving systems.Despite the effectiveness of LoRA-Inlaid, it still has several limitations. First, our quantization does not detect the existence of malicious or poisoning tasks, which might be intentionally crafted to harm the other tasks. Second, our scheduling does not consider the fairness among tasks (e.g., balancing the total numbers of output tokens for all tasks), which may be essential for shared service platforms. Third, it only supports language tasks while requiring some system re-designs for multi-modal tasks. We wish to leave the exploration of these issues as future works. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Not applicable. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: Not applicable. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: MLGPTQ is based on GPTQ, adapting the corresponding LoRA adapter according to the activation values of the input data to minimize the error in activation values before and after quantization. For only one task, GPTQ aims to solve the following problem: arg minEQUATIONwhere W l represents the weight of l-th layer in the base model, Q(W l ) represents the quantized version of W l , X l is the input of l-th layer. In other words, the objective of GPTQ is to find a Q(W l ) for each layer's weight W l through layer-wise quantization, in order to minimize the changes in activation values. Since we are discussing quantization within a single layer, we will omit the subscript l for simplicity in the rest of this section.As proved by [11] , solving Eq. 3 can be transformed into solving Eq. 4 as follows.arg minEQUATIONSince the model has converged through training before quantization, existing works generally assume the model has reached a local minimum. Thus, when we add a small adjustment ∇W to the parameter W, according to Taylor expansion, we haveEQUATIONwhere H = ∂ 2 E/∂W 2 represents the Hessian matrix. Again, since the model has converged, existing works generally assume its first-order partial derivative is close to zero and thus negligible. By neglecting the first-order partial derivative and higher-order terms, we haveEQUATIONRecall that our goal is to quantize W to Q(W). Denote ∇w q = Q(w q ) -w q , where w q represents the q-th element of W. Then, the problem to solve can be re-written asEQUATIONwhere e q represents a unit vector with a value of 1 at position q and 0 elsewhere. Since this is a constrained convex optimization problem, based on the method of Lagrange multipliers, it is necessary to solve the following equation:EQUATIONBy taking the partial derivatives of ∇W and λ, and setting them to zero to find the steady-state solution, we haveEQUATIONSolving this, we getEQUATIONEQUATIONEQUATIONwhere (H -1 ) qq represents the value at the diagonal position (q, q) of H -1 , which is the inverse of the Hessian matrix.For the Hessian matrix, we say that H = 2XX T by proving the following Lemma.Lemma A.1. Let a be a 1 × n row vector and X be an n × m matrix. The Hessian matrix of the quadratic form ∥aX∥ 2 2 is 2XX T .Proof. Let a = [a 1 , a 2 , . . . , a n ] be a 1 × n row vector, and let X = [x ij ] be an n × m matrix.Let's denote y = aX. Then, y is a 1 × m row vector with elements y j defined asy j = n i=1 a i x ij .Thus, we have∥aX∥ 2 2 = (aX) • (aX) T = m j=1 y 2 j = m j=1 n i=1 a i x ij 2 .We can expand this expression and re-write it as a quadratic form, i.e.,∥aX∥ 2 2 = m j=1 n i=1 n k=1 a i a k x ij x kj .To find the Hessian matrix of this quadratic form, we treat it as a quadratic form in a. Let Q be the coefficient matrix of this quadratic form.∥aX∥ 2 2 = aQa T .The (i, k) element of Q is given byQ ik = m j=1x ij x kj .Thus, the matrix Q can be written asQ = XX T .And the Hessian matrix is twice Q:H = 2Q = 2XX T .Therefore, the Hessian matrix of ∥aX∥ 2 2 is 2XX T , which completes the proof.GPTQ quantizes the weight W row by row. For each row, according to Eq. 12 mentioned above, it finds the minimum w q that leads to an increase in the loss function due to quantization, then calculates scales via α = max(Wi,:)-max(Wi,:) Qmax , performs quantization using α, and finally update the remaining values using Eq. 11. This process repeats until all parameters have been updated. MLGPTQ considers the scenario of quantization for multiple tasks. During the forward propagation process, it dynamically loads the corresponding LoRA adapter for each task to simulate the correct activation values for the respective tasks. Consequently, the problem we need to solve is as follows.arg minEQUATION)where T denotes the number of tasks, A t and B t are the low-rank adapter matrices of the t-th task, X t is the input of t-th task, and W and Q(W) denote the original and quantized weights of a layer.Denote W t := W + B t A t , then the problem is re-written as arg minEQUATION)Same as GPTQ, we could get T Hessian matrix H t , where t ∈ [1, T ]. To minimize the objective function, we can obtain the updating formulas for W and E in MLGPTQ:∇W = - w q -Q(w q ) (H -1 t * ) qq H -1 t * e q , ∇E = (w q -Q(w q )) 2 2 (H -1 t * ) qq, where t * = arg maxEQUATION)The updating method described here leads to the max-aggregation method proposed in § 3.1, which always selects the Hessian matrix of the task that minimizes ∇E for updating, ultimately reducing the overall error and thus better-guiding parameter updates.We provide the pseudocode of MLGPTQ in Algorithm 1, and we also highlight the differences compared to directly applying GPTQ to multi-task scenarios, which is termed GPTQ tweaked in our work (i.e., with a mixed calibration set and without any LoRA adapters).Due to the high complexity and numerical instability of the process described in Appendix A.1, we leverage the following optimizations to accelerate the quantization, partly inspired by the practical implementation of GPTQ [12] .Random Order Optimization. GPTQ requires updating weights in the order that produces the smallest quantization error ∇E. For W ∈ R m×n , the complexity of GPTQ is O(mn 3 ). However, using a random order achieves similar results and facilitates GPU parallel optimization [12] .Batch Processing. Since weight updates between different columns of the same matrix W are non-redundant, we use batch processing and delayed updates, with 128 columns processed at a time, to enhance computation speed.Cholesky Decomposition. Using numerically stable Cholesky decomposition to pre-compute the necessary information increases computational stability.Algorithm 1 Routines of MLGPTQ and GPTQ tweaked to quantize one layer in multi-task scenarios.Input: {Xt} T t=1 ▷ The inputs of different tasks of this layer1: Xt ← (W + BtAt)Xt X mixed ← T 1 WXt ▷ Forward pass 2: H -1 t ← Cholesky((2XtX T t + λI)) T , for ∀t ∈ [1, T ] H -1 ← T 1 Cholesky((2X mixed X T mixed + λI) -1 ) T ▷ Cal inv of Hessian matrix 3: Htmp ← zero_like(H -1 t )▷ Init the tmp Hessian matrix 4: Q ← 0 drow×d col ▷ Store quantized results 5: E ← 0 drow×B ▷ Store quantization errors in blocks 6: for i ← 0, B, 2B, . . . do 7:for j ← i, . . . , i + B -1 do 8:Q:,j ← quant(W:,j) ▷ Column-wise quantization 9: First, we check if running_batch is empty (line 7) to determine whether the system is in a cold start phase (i.e., the previous scheduling step did not have any requests to serve). If yes, we perform a cold start by calling generate_new_batch to get a new batch to schedule from pref ill_reqs (line 8). If new_batch is not empty, we proceed to pref ill with new_batch (line 10-11). Otherwise, the system remains idle since there are no requests (line 13). If running_batch is not empty, we check if we have consecutively executed max_cont_decode decoding steps (line 16). If yes, we call generate_new_batch to schedule new requests from pref ill_reqs and hungry_pref ill_reqs (line [17] [18] [19] [20] . Otherwise, we check if the continuous scheduling for a batch has reached a pre-defined threshold max_cont_decode_one_batch (line 22). If yes, we call schedule_new_batch to schedule new requests in the decoding stage from decoding_reqs and hungry_decoding_reqs according to the SRTF and grouping strategy (line [23] [24] [25] . Otherwise, we continue processing the current running_batch (line [27] [28] . This two-level threshold strategy effectively reduces the frequent swapping of LoRA adapters and KV caches due to frequent batch switching. Additionally, by adjusting the threshold size, we can ensure the immediacy and flexibility of scheduling.E:,j-i ← (W:,j -Q:,j)/ max t∈[1,T ] (H -1 t )jj E:,j-i ← (W:,j -Q:,j)/(H -Algorithm 2 Our multi-task scheduling strategy based on grouping and SRTF. The GPU platforms for evaluation are shown in Table 5 . The RTX 3090 platform is equipped with Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz and 256GB host memory, while the RTX 4090 platform is equipped with Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GH and 512GB host memory. For the machine translation tasks of different languages, we consider the classic bilingual translation dataset OPUS [38] , specifically choosing six translation tasks: French-to-English, Czech-to-English, Indonesian-to-English, Vietnamese-to-English, Danish-to-English, and Swedish-to-English, considering the diversity of languages.For the text translation task, we consider the XLSum dataset [14] , a comprehensive and diverse dataset comprising 1.35 million professionally annotated text-summary pairs extracted from the BBC.For the table summarization task, we consider the QTSumm dataset [48] , which is a large-scale dataset for query-centric summarization tasks on tabular data. It contains 7,111 human-annotated query-summary pairs and 2,934 tables covering various topics.For the code generation task, we consider the tiny-codes dataset [13] . This dataset consists of 16 million short and clear code snippets, aiding LLM models in learning how to reason using both natural and programming languages. The dataset includes a variety of programming languages such as Python, TypeScript, JavaScript, Ruby, Julia, Rust, C++, Bash, Java, C#, and Go. Our work centers on GPTQ due to its widespread use, but our solution can also adapt to AWQ. We presented the differences between MLAWQ and AWQ tweaked in Alg 5. As introduced in §3.1, most quantization methods follow the Forward-Aggregate Info-Modify Weight-Quant paradigm. In essence, AWQ tweaked smooths outliers by multiplying weights with a smoothing factor, best_s, to minimize per-channel quant error:• In Forward, the input is multiplied by the weights to create an unquantized monitor, guiding min error quantization (line 1). • In Aggregate Info, the average of all samples and weights is calculated for each channel (lines 2&3) to determine smoothing factor s (line 7). W and X are smoothed to remove outliers (lines 8&9). Then, smoothed W is pseudo-quantized (quantize-then-dequantize to simulate round loss) and compared to the unquantized monitor for quantization error (line 10). This process iterates over various ratios (line 6), selecting the factor with the smallest error as best_s (line 11). Then, this best_s is used to Modify the weight (line 12), followed by the Quant (line 13) process using the modified weight.The drawbacks discussed in §3.1 also exist for AWQ tweaked in multi-task quantization:• Forward (line 1): It can't pass LoRA adapters during activation distribution simulation, causing quantization bias during inference. • Aggregate Info: It uses X mean = X.mean(0), a naive mixed average of multi-tasks' info.Since each task affects each channel differently, simply averaging blurs distributions, ignoring individual effects.As explained in §3.1 and shown in Fig 2 , MLGPTQ mainly improves the first three steps to tackle GPTQ's issues in multi-task scenarios. Similarly, we can fix AWQ's issues to create a better multi-task quantization algorithm MLAWQ:• Forward: MLAWQ loads corresponding LoRA adapter for each task to participate in forward propagation, accurately simulating real activation distribution (line 1). • Aggregate Info: Instead of mixing and averaging features of each column across all tasks to compute s, MLAWQ computes the average for each task separately to get s i (line 3&7). Then it calculates quantization error for each column rather than the entire matrix (line 10). If the i-th task results in the smallest quantization error for the j-th column, it sets best_s [j] = s i [j] (line 11). This approach allows optimal error minimization, showing each task's individual effect on different channels, enhancing Aggregate Info (lines 3&6-11), and improving the Modify Weight (line 12) and Quant (line 13) processes.In summary, our work identifies common drawbacks of current single-task quantization methods in multi-task scenarios. By addressing these issues, we can develop more precise multi-task quantization algorithms.This work proposes LoRA-Inlaid, a brand new LLM serving system for the multi-task scenario.LoRA-Inlaid is featured with a series of innovations, specifically the multi-task joint quantization algorithm, the dynamic task addition mechanism, and the multi-task scheduling strategy. Considering the booming research and applications of LLMs in various downstream tasks, we believe LoRA-Inlaid has the potential to gain widespread adoption and shed light on the high-performance and resourceefficient system designs for follow-up works in the field of LLM serving. However, there are several issues that LoRA-Inlaid does not consider currently.On the one hand, the multi-task quantization algorithm in LoRA-Inlaid does not involve the detection of malicious or poisoning tasks that may bring negative impacts on the other tasks. One of the most typical use cases of LoRA-Inlaid is the personalization of LLMs, where clients can upload their data to create personalized LoRA adapters using the same base model (or directly upload their self-tuned LoRA adapters). The server is responsible for serving requests from all these clients using the proposed LoRA-Inlaid system. Fortunately, these LoRA adapters are independently manufactured, so we can apply malicious detection to them individually. For instance, the server can prepare a rich set of evaluations to assess the security risks of each LoRA adapter, including violence, discrimination, unlawful responses, etc. If any LoRA adapters fail to pass the evaluation, the server can reject serving them.On the other hand, the multi-task scheduling strategy in LoRA-Inlaid ignores the fairness among different tasks (e.g., controlling the number of output tokens to be close), which may make our work ineffective under some settings. To measure fairness among tasks, we can compute a weighted combination of numbers of input and output tokens for each task. This is because the prefilling and decoding phases in LLM inference have different workload characteristics [15] (also why these tokens differ in online service pricing). Then, we can borrow the idea of Weighted Fair Queueing (WFQ) [32] for scheduling different tasks. We wish to address these issues in the future.Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Appendix E. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Note that the idea of our solution can also be applied to other quantization algorithms which follow the same paradigm (like AWQ) since the drawbacks exist generally. Considering that the choice of backbone algorithm is orthogonal to our goal, we focus on GPTQ in this work due to its widespread adaption, and present how to adapt our solution to AWQ in Appendix D.Source code is available at https://github.com/PKU-DAIR/LoRA_Inlaid.https://huggingface.co/kaichuup 


Decision: The paper introduces LoRA-Inlaid, an innovative multi-task large language model serving system that optimizes memory usage and supports dynamic task addition through novel quantization and scheduling algorithms. Reviewers praised its practical impact and theoretical contributions but raised concerns about computational overhead, clarity, and limited empirical validation. Authors responded with detailed clarifications and additional experiments, effectively addressing most concerns. The reviewers' consensus was positive, recommending acceptance due to the system's significant advancements in LLM deployment.  The AC concurs with this recommendation and the paper may be accepted to NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper introduces LoRA-Inlaid, an innovative and efficient system for quantizing and serving Large Language Models (LLMs) in multi-task environments. By utilizing the Multi-LoRA GPTQ (MLGPTQ) algorithm, LoRA-Inlaid facilitates sharing a unified quantized model across various LoRA adapters, significantly reducing memory usage for model deployment. The platform also features a dynamic task addition mechanism that enhances the stability and reliability of online services. Moreover, it introduces a novel multi-task scheduling approach guided by predicted output lengths and task grouping, significantly reducing memory consumption and increasing overall system efficiency. Experimental results demonstrate that LoRA-Inlaid outperforms current state-of-the-art (SOTA) LLM serving systems in terms of throughput, average request latency, Job Completion Times (JCT), and Service Level Objectives (SLO) Attainment without detracting from the model's performance.
strength_weakness: strength_weakness: 1. The paper introduces multi-task quantization, incremental quantization, and predicted output length, which naturally will incur additional computational overhead. However, the paper needs an analysis of the extra costs associated with introducing these techniques.

2. The paper is missing absolute accuracy comparison experiments for the unquantized model. Both Figure 5 and Figure 6 represent relative accuracy drops compared to the quantization model without providing a direct comparison to the baseline performance of the unquantized model.

3. An excessive amount of content from Chapter 3 has been relegated to the appendices, with the main text providing a succinct description of the specific methods and also needing formulas to aid in explanation, which affects the clarity of the paper.
questions: questions: 1. In Figure 5, why is there a lack of error bars for RTN?

2. In Figure 5, why does the accuracy drop for some tasks with 3-bit quantization appear to be less than that for 4-bit quantization?

3. In the three charts of Figure 6, MLGPTQ and GPTQ have their respective strengths and weaknesses across various metrics. It is suggested that additional tasks be added to demonstrate the superiority of MLGPTQ in terms of accuracy.

4. Currently, there are only experiments on accuracy drops. Please supplement with accurate experiments.
In the efficiency tests, only S-LoRA used half-precision, which significantly increased its computational overhead, leading to an unfair comparison experiment.

5. In addition to the End-to-end system performance, could you independently analyze the individual impacts of components such as multi-task quantization, incremental quantization, and predicted output length on each performance of LLM services?

6. Please add formalized descriptions of dynamic task addition and multi-task scheduling in the main text and a brief outline of the specific method.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper addresses the need for efficient fine-tuning and deployment of large language models (LLMs) in multi-task scenarios, which has been largely overlooked in favor of single-task scenarios. Existing quantization methods, such as GPTQ and AWQ, and parameter-efficient fine-tuning techniques like LoRA, are commonly used, but they do not support the integration of multiple tasks due to their limitations in sharing the base model across tasks and handling dynamic task addition. To tackle these issues, the authors propose LoRA-Inlaid, a multi-task LLM serving system that combines a flexible and efficient multi-task quantization algorithm with a novel multi-task scheduling strategy. This system significantly reduces memory consumption, supports real-time task addition, and enhances the stability of online services.

The major contributions of this paper are as follows. The authors introduce an innovative multi-task quantization algorithm, MLGPTQ, which enables the joint quantization of models for multiple tasks, allowing a single quantized base model to be shared across tasks and supporting incremental quantization for new tasks. Additionally, they develop a novel scheduling strategy based on output length prediction and grouping, which minimizes memory consumption and reduces memory swapping overhead in multi-task scenarios. These techniques are integrated into the LoRA-Inlaid system, which demonstrates significant performance improvements over existing LLM serving systems, achieving up to 1.58× higher throughput, 1.76× lower average latency, 2× faster job completion time, and 10× better SLO attainment, all while maintaining model quality. Despite some limitations, such as the need for improved detection of malicious tasks and fairness considerations among tasks, LoRA-Inlaid represents a significant advancement in multi-task LLM serving, highlighting its potential for resource-constrained environments.
strength_weakness: strength_weakness: Editorial comments: Abstract: Avoid excessive use of undefined acronyms in the abstract. Can you explain in the abstract briefly how is LoRA-Inlaid related to MLGPTQ? References: Capitalize proper names and acronyms properly in the References. Some references are incomplete — their publication details (or URL) are missing (e.g., [10])

Outliers: (Figure 3) There are a significant number of outliers represented by circles, especially in the French-English Translation and Table Summary tasks. It might be useful to provide a brief explanation or context for these outliers.
Y-Axis Scale: The y-axis scale goes up to 4000 tokens, but most data points fall well below this range. This could make it harder to see differences in distributions for tasks with shorter lengths. Using a logarithmic scale or breaking the y-axis into two parts could provide better clarity.

Outliers and Variability (Figure 4): There seems to be significant variability in the number of tasks for Skip-join MLFQ (FastServe) and FIFO (S-LoRA). Providing statistical summaries such as mean or median lines could help interpret the data more effectively. Including error bars or confidence intervals would give a better understanding of the variability and reliability of the scheduling strategies.

Malicious Task Detection: The paper acknowledges the need for improved detection of malicious tasks but does not provide detailed solutions or strategies to address this issue. This represents a potential vulnerability in the proposed system that could be exploited in practical applications.

Fairness Considerations: Fairness among tasks is briefly mentioned as a limitation, but the paper lacks an in-depth discussion on how fairness is measured and what specific strategies could be employed to ensure equitable resource distribution among tasks. This is crucial for practical deployment in environments where multiple users or tasks compete for limited resources.

Experimental Scope: While the results are promising, the scope of the experiments is somewhat limited. The paper would benefit from additional experiments across a wider range of tasks and more diverse datasets to fully validate the generalizability and robustness of the proposed system.

Incremental Quantization Details: The paper does not provide extensive details on the incremental quantization process. More information on how new tasks are integrated into the system and the potential impact on existing tasks would strengthen the paper.
questions: questions: Detection of Malicious Tasks: Can you provide more details on how you plan to enhance the detection of malicious tasks? Are there any preliminary strategies or methods you are considering to address this vulnerability?

Fairness Among Tasks: How do you measure fairness among tasks in the LoRA-Inlaid system? Can you elaborate on the strategies you plan to implement to ensure equitable resource distribution?

Incremental Quantization Process: Could you provide more information on the incremental quantization process? How are new tasks integrated into the system without impacting the performance of existing tasks?

Generalizability of Results: The experiments demonstrate significant improvements, but are these results consistent across a broader range of tasks and datasets? Can you share any additional experimental results or plans for future testing?

Scalability and Real-Time Performance: How does the system scale with a large number of tasks added in real-time? Are there any performance benchmarks or case studies that illustrate the system’s scalability in real-world scenarios?
limitations: limitations: The authors have identified some limitations, such as the need for improved detection of malicious tasks and fairness considerations among tasks. However, these limitations are not fully addressed in the paper.

Addressing Malicious Task Detection: The paper mentions the need for better detection of malicious tasks but does not provide concrete strategies. Constructive suggestion: Develop and describe specific methods for detecting and mitigating malicious tasks, potentially through anomaly detection techniques or secure task validation protocols.

Fairness Among Tasks: Fairness is noted as a limitation, but there is little discussion on how to ensure it. Constructive suggestion: Elaborate on fairness metrics and propose strategies for fair resource allocation among tasks, possibly through dynamic scheduling algorithms that prioritize based on task urgency and resource consumption.
paper_summary: paper_summary: This paper mainly focuses on the quantization problem of large language models and the problem of low-rank decomposition solvers, and proposes a method for quantizing large language models for multiple tasks and integrating multiple low-rank decomposition solvers. The article first analyzes that the current mainstream quantization method will make it impossible to share model parameters among tasks when processing multiple tasks. Therefore, the current LLM service system cannot integrate LLM quantization with multiple LoRA solvers to achieve memory-efficient multi-task support. At the same time, the existing LLM service system lacks support for dynamic task addition. Therefore, the LoRA-Inlaid proposed in this paper designs a flexible and efficient multi-task quantization algorithm, which is convenient for multiple LoRA solvers to share a single quantization model, greatly reducing the memory consumption of model deployment. On the other hand, LoRA-Inlaid develops a new multi-task scheduling algorithm based on output length prediction and grouping, which effectively reduces memory consumption and avoids frequent switching of LoRA solvers. This innovative multi-task quantization algorithm MLGPTQ uses multi-task data to perform joint quantization on the basic model. This allows the basic model of quantization to be shared between multiple tasks. In addition, it also supports incremental quantization of newly added tasks without affecting the performance of the service.
strength_weakness: strength_weakness: This article is a very excellent and innovative algorithmic article for quantitative model improvement. The algorithm process is clearly explained, the process is clear, and the proof process is very detailed. The proof ideas are clear and complete. However, the following small issues can be considered:
1. The symbols of the proof process in this article need to be further sorted out, for example, does the q-th parameter refer to the qth parameter or the qth row parameter?
2. This article mentions that the Lagrange multiplier method needs to be used to solve max-aggregation, but the solution process, including the final result, will have multiple matrix inversions. Then the condition number of the matrix needs to be further considered to see if it will affect the accuracy and speed of the solution.
3. Figure 2 mentions the difference between the method in this article and the GPTQ method, which is very clear, but this article also proposes other quantitative methods. Can you explain why GPTQ is simply compared with MLGPTQ, or the AWQ method is also represented in this way to show the highlights of this method?
4. This article has less or less explanation of the multi-task scheduling strategy. It is recommended that the author add more comparative experiments on this module, otherwise the content is relatively lacking as an innovation point.
5. For large model tasks, there may be some priorities among the tasks. So based on the multi-task scheduling method in this article, how should we consider the situation where there are multiple tasks and the task levels have an order?
questions: questions: This article is a very excellent and innovative algorithmic article for quantitative model improvement. The algorithm process is clearly explained, the process is clear, and the proof process is very detailed. The proof ideas are clear and complete. However, the following small issues can be considered:
1. The symbols of the proof process in this article need to be further sorted out, for example, does the q-th parameter refer to the qth parameter or the qth row parameter?
2. This article mentions that the Lagrange multiplier method needs to be used to solve max-aggregation, but the solution process, including the final result, will have multiple matrix inversions. Then the condition number of the matrix needs to be further considered to see if it will affect the accuracy and speed of the solution.
3. Figure 2 mentions the difference between the method in this article and the GPTQ method, which is very clear, but this article also proposes other quantitative methods. Can you explain why GPTQ is simply compared with MLGPTQ, or the AWQ method is also represented in this way to show the highlights of this method?
4. This article has less or less explanation of the multi-task scheduling strategy. It is recommended that the author add more comparative experiments on this module, otherwise the content is relatively lacking as an innovation point.
5. For large model tasks, there may be some priorities among the tasks. So based on the multi-task scheduling method in this article, how should we consider the situation where there are multiple tasks and the task levels have an order?
limitations: limitations: This article mentions possible limitations of this method at the end. The first is that the quantitative method in this article does not detect the existence of malicious or poisoned tasks, which may be intentionally used to harm other tasks. Second, our scheduling does not consider fairness among tasks, which may be essential for shared service platforms. 
Third, it only supports language tasks, while requiring some system redesign for multimodal tasks. In addition to the three possible limitations mentioned in the article, the priority of tasks and the order between tasks may also be points that need to be considered. 
Secondly, the calculation time of the intermediate process of MLGPTQ may vary significantly depending on the model. In addition, if some tasks may need to be exited midway during the training process, how should the model proposed in this article handle this situation.

Comments: title: Thanks for your reply
Thanks for your reply
Response to authors' rebuttal
Thanks for your reply
comment: Thank you for your time and consideration! Your detailed comments are extremely helpful, and we believe addressing your comments will significantly improve our paper.
Thanks for the authors' detailed feedback. It has addressed my concerns. In light of these explanations, I will revise my score accordingly.
Many thanks for your acknowledgement! Your insightful comments guide significant enhancement to our paper.
Authors' rebuttal mostly addresses my concerns. I will change the score to "accept".
We are grateful for your acknowledgement in our response and we believe our paper will be substantially improved by addressing your constructive comments.
Thanks for your clarifications. I acknowledge that I have read these comments in the rebuttal in response to my comments and that I have considered these in my review scores.