Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
Authors: Yuchen Zhang
Hanyue Du
Chun Cao
Jingwei Xu
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Primary area: infrastructure

Abstract: Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning (PEFT) technique for adapting large language models (LLMs) to downstream tasks. While prior work has explored strategies for integrating LLM training and serving, there still remains a gap in unifying fine-tuning and inference for LoRA-based models. We present **Loquetier**, a virtualized multi-LoRA framework that seamlessly integrates LoRA fine-tuning and serving within a single runtime. Loquetier introduces two key components: (1) a Virtualized Module that isolates PEFT-based modifications and supports multiple adapters on a shared base model, and (2) an optimized computation flow with a kernel design that merges fine-tuning and inference paths in forward propagation, enabling efficient batching and minimizing kernel invocation overhead. Extensive experiments across three task settings show that Loquetier consistently outperforms existing baselines in both performance and flexibility, achieving up to $3.0\times$ the throughput of the state-of-the-art co-serving system on inference-only tasks and $46.4\times$ higher SLO attainment than PEFT on unified fine-tuning and inference tasks. The implementation of Loquetier is publicly available at https://github.com/NJUDeepEngine/Loquetier.

Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
Yuchen Zhang' Hanyue Du
Chun Cao Jingwei Xul
State Key Laboratory for Novel Software Technology, Nanjing University, China (zfirozen,dh) njj.. edu.edu.coo en,
{caochun, jingweix)Onjul edd.eeu.. edu. cn
Abstract
Low-Rank Adaptation (LoRA) has become widely adopted parameter-efficient fine-tuning (PEFT) technique for adapting large language models (LLMs) to down- stream tasks. While prior work has explored strategies for integrating LLM training and serving, there still remains a gap in unifying fine-tuning and inference for LoRA-based models. We present Loquetier, virtualized multi-LoRA framework that seamlessly integrates LoRA fine-tuning and serving within a single runtime. Loquetier introduces two key components: (I) a Virtualized Module that isolates PEFT-based modifications and supports multiple adapters on a shared base model, and (2) an optimized computation flow with kernel design that merges fine- tuning and inference paths in forward propagation, enabling efficient batching and minimizing kernel invocation overhead. Extensive experiments across three task settings show that Loquetier consistently outperforms existing baselines in both per- formance and flexibility, achieving up 3.0x the throughput the state-of-the-art co-serving system inference-only tasks and 46.4x higher SL. attainment than PEFT on unified fine-tuning and inference tasks. The implementation Loquetier is publicly available at https: http://hhh.ccoo///DDDeeeeee/e///
Introduction
Large Language Models (LLMs) built on stacked transformer blocks Vaaswani et al. 2017 have achieved remarkable success across wide range of text generation tasks. This success has driven the development of ever-larger models, such as the LlaMA series Touvron et al. 2023 Grattafioni et al.) 2024] and the Qwen family [Bai et al.] 2023 Yang et al. 2024 However, the rapid growth in model size has introduced prohibitive costs. For example, LlaMA 3 contains 405B parameters, and DeepSeek-V3 V3 [Bi et al., 2024] scales to 671B. Their computational and memory requirements of full-parameter training now represent a major bottleneck, restricting both the scalability and accessibility of LLM development.
Parameter-efficient fine-tuning (PEFT) has emerged as a practical solution to these challenges. By reducing the number of trainable parameters while retaining the effectiveness of full-model fine- tuning, PEFT offers a balance between efficiency and adaptability [Ding et al.) 2023a]. Recent studies have evaluated PEFT across diverse applications and theoretical dimensions [alne et al. 2024 Xu
ct al. [202a]]. and have surveyed its underlying mechanisms and practical benefits (Han et al. 20024]
Fu et al. 20233 Notice that PEFT approaches such as Prefix Tuning [Li and Liang 2021 and Prompt
Tuning Lester et al.) 2021 have demonstrated strong adaptability by optimizing small task-specific vectors. Moreover, PEFT has shown competitive superior performance in low-resource settings, including zero- and few-shot learning [Liu et al.) 2022
Hu et al. 2023
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Comparison on different LoRA tasks between Loquetier, PEFT and F
Inference
Finetune
Finetune & Inference
Framework or System
Single
Multi
Single
Multi
Single
Multi
Loquetier
PEFT S-LORA+PEFT
X
v
X
X
V
X
FlexLLM
V
X
X
Among PEFT approaches, Low-Rank Adaptation (LoRA) Hu et al. 2022] has become particularly prominent, offering scalable and effective fine-tuning across diverse LLM tasks. Numerous extensions have been proposed to enhance its flexibility, including LoHa [Hyeon-Won Woo et al. 2021]. VeRA
[Kopiezko al. 2023). ), and LoKr (Yeh al.) 2023). LoRA has also shown strong potential for personalization, powering systems in recommendation [Kong et al. 2024 Zhu et al. 2024), and
user-centered content generation (Zhang et al. 2024, Wu et al.) 2024a]. Its modularity makes well-suited for large-scale serving systems where real-time customization is critical.
In practice, systems that serve LoRA often need to support fine-tuning adapters while simultaneously deploying them for inference across diverse tasks. However, no existing framework can seamlessly unify these two capabilities, resulting major obstacles for scaling LoRA into production. Jointly fine- tuning and serving multiple adapters requires minimizing memory and computation overhead while efficiently handling heterogeneous workloads. Prior efforts have mainly optimized the base LLM, such as through KVCache improvements or inference pipeline parallelization [i et al. 2024a with only limited advances in multi-LoRA inference [Chen et al.. 2024 Sheng et al., 2023
integrated fine-tuning and inference [Miao et al. 2024). However, these approaches still face critical challenges: adapters often fused into monolithic instances for efficiency, thus they cannot be dynamically loaded or unloaded; decoding efficiency degrades significantly when fine-tuning and inference run concurrently; and task switching typically requires halting the current job before starting another, causing downtime, bandwidth overhead, and wasted resources. As result, existing frameworks remain inadequate for large-scale, production-ready LoRA applications.
In this paper, we introduce Loquetier a unified virtualization framework that integrates fine-tuning and serving of LLMs with LoRA-based PEFT. Loquetier provides streamlined computation flow that handles both fine-tuning and inference requests within a shared runtime, using kernel-level optimizations to reduce memory overhead and execution latency. Specifically, Loquetier introduces the Segmented Multi-LoRA Mulliplication (SMLM) kernel, which enables mixed-task execution by distinguishing forward-pass behaviors for fine-tuning, evaluation, prefilling, and decoding. To support multiple concurrent LoRA adapters without modifying the base model, Loquetier further incorporates Virtualized Module abstraction that dynamically injects adapter logic while preserving compatibility and isolation across tasks and devices. This design enables seamless co-serving of heterogencous LoRA config aunigurations and supports mstance-to-insteee migration of fine-tuning jobs without kernel restarts or memory duplication. The main contributions are as follows:
We design an SMLM kernel and an unified computation flow that efficiently supports fine-tuning and inference with multiple LoRA adapters on shared base model.
We propose a modular virtualization mechanism that isolates PEFT-based modifications from the base model, enabling flexible instance- migration and seamless adapter management.
We develop Loquetier to unify LoRA fine-tuning and serving. Extensive experiments show that Loquetier outperforms existing systems across diverse scenarios and enables unified practical fine-tunin fine-tuning and serving configurations previously unsupported.
"The name Loquetier is synthesis of "LoRA" and "coquetier", reflecting our design philosophy: the base model serves as the foundational spirit, while LoRA modules act as adjunct ingredients mixps in for task-specific customization.
ingredients-spirtss, spirits. juices, and
FFleXLLM cycles through loading LoRA models during LoRt inference, disregarding the maximum number of resident LoRAs set, which makes its multi-LoRA inference efficiency practically unusable.
"The backward procedure of FlexLLM triggered an error originating from an unsupported operation in its gradient computation logic. We describe our solution in the Appendix

---

2 Related Work
LLM inference optimization via KV cache. Key-value (KV) caching is a core technique for accelerating LLM inference by avoiding redundant computation and memory transfers (Pope et al.) 2023]. Recent work improves cache efficiency and reduces GPU memory overhead through tailored management strategies. Prompt Cache [Gim et al.) 2024] reuses prompt embeddings to reduce duplication. vLLM [Kwon et al. 2023] and vAttention [Prabhu et al. 2025 address fragmentation
using paging and virtual memory, respectively. Infinite-LLM Lin al.) 2024 enables dynamic sharing of cache segments between host and GPU. LoongServe [Wu et al. 2024b] enhances long-
context serving by balancing prefilling and decoding workloads.
LLM inference parallelism. A large amount of work improves LLM inference throughput by optimizing batch scheduling and pipeline execution. Response Length Perception (Zheng al.] 2023] and 5 [Jin et al.) 2023) predict output lengths to batch similar requests, with 5 further refining its predictions over time. Orca YYu al., 2022] uses token-level continuous batching, while DeepSpeed-Fastgen Holmes et al. 2024) adjusts request lengths for better GPU utilization. Sarathi- Serve [Agrawal et al. 2024 co-schedules prefilling and decoding tasks, whereas Tetrilnfer [Tuu et al.,
2024]. Splitwise [Patel et al. 2024]. and DistServe (Zhong et al. 2024] decouple these phases across
threads, machines, or clusters improve parallel efficiency. In kernel, FlashDecoding++ Hong et al.
2024
overlaps computation with data transfer to hide latency. FlashAttention [Dao 2023 Shah et al.
2024
uses warp specialization and asynchronous execution to maximize attention layer throughput
Efficient multi-LoRA inference and unified finettuiigginffrrnnce systems. FlashInfer Ye et al.
2025]
reduces redundant storage and optimizes GPU memory access for efficient multi-LoRA inference. Cutlass (Thakkar et al., 2023). a high-performance CUDA library for GEMM operations, serves as the kernel foundation for Punica [Chen al.) 2024). which combines with FlashInfer to support scalable LoRA serving. Built on this stack, S-LoRA (Sheng et al.] 2023) further improves
GPU memory utilization through dynamic host-device memory transfers. In parallel, FlexLLM (Miao it al. 2024) explores unified token-level computation for co-serving inference and PEFT fine-tuning, though scalability remains limited.
LoRA optimization and variants. There are lots of extensions that have been proposed to enhance
the flexibility, efficiency, and adaptability of LoRA. AdaLoRA (Zhang al. [023a] introduces singular value decomposition for dynamic pruning, while IncreLoRA [Zhang et al. 20236) allocates parameters based on module importance. SoRA (Ding al.,
2023b] adaptively adjusts rank via
gated weight control during training. DiffoRA [Jiang et al.
2025 selects modules to finetune
using& Differential Adaptation Matrix (DAM). To improve expressiveness and stability, DoRA
[Liu et al., 2024) decomposes weights into magnitude and direction. VB-LoRA [Li et al. reduces redundancy by sharing global vector banks across modules. LoRA-XS (Balazy et 2024
2024b
compresses storage with minimal x matrices, and QA-LoRA [Xu et al. 2023b integrates quantization and grouping to increase representational flexibility.
3 Loquetier Framework
In this section, we describe the overall architecture of Loquetier, followed by details of its unified computation flow and the proposed LoRA model virtualization.
3.1 Framework
As illustrated in Figure I Loquetier framework consists of two core components: (I) a model library centered around the Virtualized Module for isolating PEFT modifications, and (2) redesigned kernel and computation flow that jointly support line-tuning and inference workloads. To enable concurrent execution, Loquetier integrates a context management system that coordinates the runtime scheduling of heterogeneous tasks.
When loading base model into CPU or GPU memory, Loquetier instantiates multiple virtual models, each acting as an isolated container for a specific PEFT configuration. These virtual models are bound to distinct adapters, enabling independent and concurrent execution. For LoRA-based adapters, we introduce the MixedLoraModel class to support fine-tuning. Based on our computation flow (see Section 3.3). each MixedLoraModel efficiently fine-tunes its associated LoRA adapter within
3

---

Mixed LoRA Model
Base Model (Library)
Other PEFT Model or Customized Model
Virtual Decoder Layer
Base Decoder Layer
Virtual Module
Virtual Attention
MixedLoRALLnnan Q
Base Attention
LoRAI
LoRA2Q Q
LoRA3
(
Module Modifications
Virtual Linear
Linear Medifications
MLL MLLL V) MLL.
Virtual MLP
Base MLP
Other Modifications
Gate
Down
MLL Up
MLL Gate
MLL Down
Computation Flow Unitied Inputs
Computation Flow in Attention
Inputs
Inputs and Context
Computation Flow
Finetunes
Prefills
Decodes
SMLM Kernel
Q. K, Projection
multi times forwards
with imputs updated
Forward
under high throughiput
PyTorch (with grad) Fine-tuning Forward
FlushInfer Kemee Prefill
FlashInfer Kernel Decode
Backward
Forward
Forward
Backward
Loop
Update Inputs
SMLM Kernel
Finetunes
Prefills
Decodes
Projection
Figure 1: The framework diagram of Loquetier.
its own container from dedicated trainers, while other virtual models can simultaneously remain in inference mode.
Loquetier's modular design is compatible with most existing LLM inference optimizations and training strategies. It fully supports architectures that leverage FlashInfer (Ye et al.) 2025) kernel and remains extensible to those that do not, requiring only localized modification to the inference logic within the computation flow. Section 3.2 further details how the Virtualized Module enables base model sharing across diverse PEFT methods beyond LoRA.
3.2 Virtualized module
The mixing of different LoRA or other PEFT methods in the same model object makes model configurations chaotic and difficult to handle. Moreover, dynamic model loading and unloading should be supported in order apply the fine-tuned and up-to-date LoRA models quickly.
We propose the Virtualized Module that provides methods and data proxies to foundation modules to solve the above problems. Applying the Virtualized Module to the base model is extremely low-cost, with no additional GPU memory overhead and provides independent model instances in an intuitive way. For each module type, the Virtualized Module creates the corresponding virtual module at runtime to provide the correct proxy methods. The Virtualized Module prepares additional properties and methods for Linear and Model to accommodate the architecture of Transformers and PEFT.
Since each virtual module class is created runtime, which means none of these virtual modules can be shared, nor can process that owns any virtual module create child processes from fork or spawn method. Furthermore, any deep copying behavior that includes virtual module will cause the base module linked to it to be copied, thus invalidating base module sharing. For this reason, we provide a non-local class definition for deep copying, serialization, and deserialization. By voiding the containing Virtualized Module, LoRA models and other PEFT models loaded onto the Virtualized

---

Algorithm 1 Computation flow control in attention layer of Loquetier
Input: hidden states matrix X with shape [S, H. list of fine-tuning inputs batch-sequence informa- tion tuples F, list of prefilling and evaluation inputs sequence lengths P, decoding inputs count
D.
Output: attention outputs matrix O. Q Pray [X): K
proj (X); V Vproj (X);
Os
if len > then
Extract Qr.KG. Kr. Vr from Q K. V based on F; Compute Og through the standard forward implementation;
Of is appended to Os;
end
if (P)(P) > 0 then
Compute offset of prefills Offseet based on F;
Extract Kp. Vp from Q, K, V based on P and Offsetp: Initialize KVCache for prefills;
Compute Op through the FlashInfer forward implementation; Op is appended to Os;
end if
D
> then
Compute offistt of decodes Offsetta based F and P: Extract Qa, Ka. Va from Q, K, V based on Offiseta:
Append KVCache for decodes;
Compute Oa through the standard forward implementation; Oa is appended to Os;
end if
Concatenate Os into one tensor as O:
O = O Opraj (O);
return O;
Module can be migrated to other GPU devices after deep-copying and used after unvoiding based on instances of the new Virtual Model.
Based on the above design, the Virtualized Module is compatible with all PEFT methods, as well as any custom model modification methods that do not modify the underlying model's s own data, such as weights and module configurations. For scenarios where the target module overwrites the base module with new data, the module design needs to be normalized so that the target module runs the new forward method based on its own data and the base module's 's data, rather than running the base module's 's forward method after using destructive modification methods.
3.3 Unified computation flow management and SMLM kernel
For multiple LoRA adapters within the same linear layer, traditional methods typically process the computation sequentially, computing the output for one LoRA at a time and iterating through all adapters. This approach significantly slows computation. Since different LoRAs within the same layer usually share identical or similar shapes, it is feasible to compute all outputs in single kernel call. Punica [Chen et al. (2024) leverages this property by implementing a foundational algorithm on top of the Cutlass GEMM (General Matrix Multiplication) library (Thakkar et al.) 2023 enabling simultaneous processing of multiple input- LoRA pairs for efficient multi-adapter computation.
However, the original Punica kernel design is incompatible with fine-tuning, as it statically con- catenates LoRA weights from the same module across different layers. This rigid coupling limits architectural flexibility and prevents selective application of LoRA to specific layers, which is particu- larly problematic in training scenarios where layerwise heterogeneity common. This requires every layer in fine-tuning tasks to adopt the exact same configuration, even when certain linear modules do not actually require LoRA. Meanwhile, this design incurs substantial overhead in tracking and mem- ory management, especially when operating very large tensors during training. For inference tasks,
5

---

Algorithm 2 Computation flow control in causal LM of Loquetier
Input:
LM inputs X, list of fine-tuning inputs batch-sequence information tuples F, list of evaluation inputs sequence lengths E, Labels of fine-tuning and evaluation inputs Labels, Accumulation steps of fine-tuning and evaluation inputs A.
Output: list of losses Loss, logits Logits.
Compute Logits by forward propagation:
Loss =
for (batch-sequence (B, S). accumulation step AFE in (F, E), do
Extract LogitsFE and Labelsfg from Logits and Labels;
Shift LogitsFE and LabelsFE:
Compute LOSSFE of LogitsFE and Lablespe from the given loss function;
LosSa LOSSFE/ AFE;
LosS is appended to Loss;
end for
return Loss, Logits;
it also eliminates the possibility of rapidly swapping LoRAs during runtime. Thus, the computation process must halted before replacement, and the required LoRA must be re-spliced together.
To overcome these limitations, we adapt the Punica kernel to process LoRA weights one linear layer time. This decoupling removes the need to regenerate model files via weight transformation before execution. For static scaling factors in LoRA, we apply the scale directly to the weight tensor at Mi MixedLoraModel instantiation reduce runtime overhead. When dynamic scaling required. is applied on per-request basis during the forward pass. Loquetier then executes forward computation across all active requests unified manner, supporting four types of requests: fine-tuning (training), evaluation, prefilling, and decoding. Evaluation requests are structurally similar to prefilling but compute a loss over labels and execute only single generation pass, while prefilling requests omit the loss computation and transition into decoding after the initial pass.
For the backward pass, since FlashInfer does not support gradient computation, Loquetier falls back to the standard forward implementation backed by PyTorch's 's Autograd when handling fine-tuning requests. This enables full gradient computation through efficient C++-based differentiation. For inference-only requests, Loquetier instead leverages FlashInfer's batch-prefill and batch-decode kernels to maximize throughput and memory efficiency.
The forward computation flow for the attention layer is summarized in Algorithm ! First, the Q. K. and V projections computed jointly for all incoming requests. Attention outputs then computed independently for each request type, concatenated, and passed through a shared output projection O. Algorithm [ outlines the forward method for the causal language model. For each incoming request, Loquetier computes output logits and, when applicable, the loss with respect to provided labels. Fine-tuning and evaluation requests include ground truth labels and thus return both logits and loss values, while prefilling and decoding requests return only logits. Loquetier enables joint forward pass of fine-tuning and evaluation requests within the same batch. Because the losses are tracked separately in Loquetier, this separation allows distinct gradient accumulation strategies for different fine-tuning tasks parallel, without cross-interference. By summing losses across all fine-tuning requests, Loquetier produces shared backward pass, enabling gradients from multiple fine-tuning jobs to be computed efficiently in a single backpropagation step.
We further extend the Transformers' Trainer (Wolf et al.) 2020) to support an interruptible fine-tuning process. Multiple trainers can now share the same computation flow in Loquetier, performing unified forward and backward propagation for fine-tuning different LoRA adapters concurrently. To ensure that each trainer only updates its corresponding parameters, we in- troduce MixedLooRModeeleee ForTraine.. which applies parameter masking on top of a shared MixedLoraModel instance achieve isolation.
Experiments
The Loquetier framework expects enable fine-tuning and reasoning across multiple LoRA models. Based on the objective, we design three experiments to evaluate the performance of Loquetier in

---

Table 2: Comparison on model loading between Loquetier, PEFT, S-LoRA and FMcxLLM. Metrics include time load (Time) and additional storage footprint (Storage).
Base Model
LoRA Model
Total
Framework or System
Time
Storage
Time
Storage
Time
Storage
Loquetier
2.927 s
0
2.409 s
OB
5,336 s
0 B
PEFT
2.877
0 B
1.914
0 B
4.791
O
S-LoRA
33.037
0 B
0.948 s
O
33.985 8
O
FlexLLM
37.933
14.96 GB
0.924
40.04 MB
38.857
15.00 GB
Decode Throughput (tokens/s)
100
400
80
60
SLO Attainment (%)
200
40
20
400
80
60
200
Throughput (tequests/s)
Lequetier (Partial) ezz
Loquetier (Full)
MlexLLM (Partial)
FlexLLM (Full)
S-LoRA (Partial)
ZZZ
S-LoRA (Full)
PEPT (Partial)
DIZ PEFT
Figure 2: Comparison of the performance of Loquetier, FlexLLM, S-LoRA and PEFT in inference tasks. The upper is single LoRA model inference and the lower part is multiple LoRA model inference. Partial means that only 3 modules are enabled for FlexLLM including up, gate, and down. For detailed information on S-LoRA, please refer to the Appendix L Full means that all 7 modules enabled, including 9, v, o. up, gate, and down. indicates that the results were not obtained: FlexLLM does not support enabling LoRA modules for linear layers other than up, gate, and down; FlexLLM cycles through loading LoRA models during multi-Loke inference.
these scenarios: inference-only, fine-tuning ton--uuning-olly and unified fine-tuning and inference. In addition, we perform two experiments to evaluate the performance Loquetier in simulated real-world environments: shorter, approximate simulation for rapidly testing different load conditions across different times of the day, and a 120-minute precise simulation using data extracted from real-world scenarios for design, we conduct a micro-experiment to evaluate the efficiency of the model loading process.
demanding and realistic stress test. To further validate the effectiveness of
4.1 Evaluation settings
Baselines. FlexLLM is an advanced co-serving system for LLM serving and parameter-efficient fine-tuning. We deployed docker images as its runtime environment following the guidelines. S-LoRA designed specifically for large-scale LoRA inference. Therefore, we combine S-LoRA with PEFT as another baseline, which PEFT handles the fine-tuning task. We use HuggingFace Transformers with PEFT as the most basic bascline.
Models. We use the Llama3-8B model as the base model. The LoRA adapter is obtained by training on the Alpaca dataset. For the fine-tuning task, we use the same LoRA configuration as the inference LoRA adapter and initialize the weights from the Gaussian distribution.
Datasets. We use the ShareGPT dataset as input for the inference task, and the Alpaca and GSM8K datasets as input for the fine-tuning task. BurstGPT [Wang et al. 2025) is an LLM service workload dataset comprising over ten million traces collected from Azure OpenAl GPT services. Data spanning more than 60 days is segmented into 20-minute slices according to its provided partitioning scheme.
7

---

Finetunn hpruughuu (tokensis)
Fine- tening time (s)
1,500 2,000 2,500
3,0004,000 0005 ,0000
Finetune Throughput
Fine- tuning time (s)
1,500 2,000 2,500
3,0004,0000 DK35 5,0000
(tokens/))
Single (1) LoRA fine-tuning
Multiple (2) LoRAs line-tuning
Loquetier (Partial, Tmining)
1.qquetier (Partial, Exaluation)
Loquetier (Full, Training)
KZZ
Loquetier (Full. Evaluntion)
LM (Partial, Training)
FlcxLLM (Partial, Evaluation)
M. (Fall, Training)
Flext (Pull, Evaluation)
PEIT (Partial, Training)
PEFT (Partial, Evaluation)
PEFT (Full, Training)
ZZZ
PEFT (Full, Evaluation)
Figure 3: Comparison of the performance of Loquetier, FlexLLM and PEFT in fine-tuning tasks. The meanings of Partial and Full are the same as in Figure indicates that the results were not obtained: FlexLLM does not support backward propagation computations for modules other than up, gate and down. PEFT can only finetune one LoRA adapter at time, so its time cost is cumulative.
Hardware. We test the inference-onle tasks on a server with NVIDIA A6000 48G GPUs. We test the fine-tuning- only tasks and the unified tne-tuming and inference tasks servers with NVIDIA H800 80G GPUs Each test process has at least 128G of host memory available.
The inference-only tasks run inference tasks at different request arrival rates where inference needs to as fast as possible to achieve the Service Level Objective (SLO). The fine-tuning-only tasks need to run the training task for a specified number of epochs to measure its efficiency in processing tokens. The unified fine-tuning and inference tasks need to run the training task along with the inference task, weighing the efficiency of fine-tuning tokens against the SLO of the inference request. The world workload simulation experiment samples six time periods from the BurstGPT dataset, comprising one low-load, two medium-load and three high-load intervals. Each slice was categorized into one of three load tiers based its request rate: low-load periods for average RPS < 1; medium- for 1-1.75; and high-load for > 1.75 (which may include transient spikes exceeding RPS 10). Here, RPS denotes requests per second. The detailed configurations can be viewed in Appendix D
Metries and Tasks. The metrics used for the evaluation are listed in Appendix C
4.2 Evaluation results
Model Loading. The loading speed and additional loading storage overhead are shown in Table 2 Compared to PEFT, Loquetier requires creating Virtualized Modules and applying scaling each LoRA linear when loading LoRA models, resulting in a slight slowdown in this part of the loading. FlcxLLM needs transform and cache the model weights, which leads to significant additional storage footprint. Even with cached transformed models, Ll'xLLM'ss loading is still very slow due to the need read small weight files.
Inference. Figure shows the test results of the inference task. Loquetier maintains the highest SLO attainment at different request arrival rates. As the request rate increases, Loquetier's decoding speed gradually increases. Until at 3 RPS, the decoding speed no longer increases, indicating that the GPU memory access bottleneck has been hit. As the request rate continues to increase, some requests begin to fail to reach their SLOs due to the inability to achieve faster inference on the current GPU.
FLcXLLM's maximum decoding speed is lower than Loquetier's, ss causing its SLO attainment rate to start dropping earlier and fall off a cliff higher request arrival rates. In addition, in conjunction with the findings in Section 4.2 FlexLLM'ss lazy loading mechanism prevents it from handling some of the earliest arriving requests under SLO. Foreing early loading of the model weights improves SLO attainment to or near 100% at 1-2 RPS, but the improvement is very limited for higher request arrival rates due to the limitations of its highest decoding speed. FlexLLM is unable to apply LoRA to all 7 modules, causing it to fail under the corresponding experiments. When loading multiple LoRA models, M trapped in a dead loop for more than minutes, missing SL.Os for all requests. After longer wait, L cannot get out of the trap still, and therefore is marked as failure.
8

---

4,000
600
100
3,000
400
2,000
1,000
200
PEI diops 2216 infer rIQURItY
0
Finetume Throughput (tokens/s)
1,000
Decode Throughpus (tokens/s)
600
100
3,000
80
100
80
2,000
40
1,000
200
20
Attainment
0
0
SLO Atlannn
2,000
600
100
400
1,000
200
0
600
100
2,000
80
400
60
1,000
40
200
20
0
0
5
Throughput (requests/s)
Loqueticr (Partial)
Loquetier (Full)
Flext.1 M (Partial)
M. (Full)
PEPT (Partil)
PEFT (Full)
Figure 4: Comparison of the performance Loquetier and PEFT unified tasks. The 4 subplots correspond respectively to single-finetune & single-infer, single-finetune & multi-infer, multi-fined multi-finetune single-infer, and multi-iinetune multi-infer. The meanings of Partial and Full are the same as in Figure 2 x indicates that the results were not obtained: FlexI.LM and PEFT can only finetune 1 LoRA at a time due to GPU memory limitations, causing it to fail the Lulti-LoRA fine-tuning scenarios; FlexLLM only support 3 target modules as mentioned in previous figures.
Transformers' batch strategy padding different inputs to the same length makes its GPU memory footprint greatly affected by the batch size, making it very easy trigger the error "CUDA out of memory". The processing speed of PEFT constrained by the batch size to avoid exceeding the GPU memory. In multiple LoRA inference tasks, PEFT can only apply LoRAs in serial for different configurations of inputs, making the inference speed further degraded. PEFT's SLO attainment rate unacceptable even under RPS.
Fine-tuning. The test results for the fine-tuning task are shown in Figure The shorter total training time for Loquetier is due to its faster evaluation. Loquetier's fine-tuning is slightly slower than that of PEFT, mainly because of the independent computational calls from the LoRA linears during backward propagation. FlexLLM encounters unsupported operation error during its peft backward propagation, indicating its inability to complete the experiments. The results show that Loquetier leads to almost loss of fine-tuning efficiency.
Unified Fine-tuning and Inference. We test a combination different configurations fine-tuning and inference tasks, and the results are shown in Figure Loquetier is able to provide an average of about 40% fine-tuning efficiency over three different request arrival rates while maintaining similar SLOs in the inference-only tasks. PEFT's inference efficiency is too low, resulting in over 90% of the inference tasks timing out before they even begin. PEFT's fine-tuning tasks only drop about 20% efficiency, but this is due the fact that PEFT has almost computational overhead on the inference tasks, allowing the vast majority of the computing resources to still be used by the fine-tuning tasks. FlexLLM fails to complete the fine-tuning task, so it is not available for unified tests.
Mutable Capacity Allocation Simulation. In order to evaluate performance facing dynamic loads in real-world scenarios, we design inference subtask with dynamic input throughput. As shown Figure 5 Loquetier able to adaptively adjust the efficiency of both fine-tuning and inference tasks under dynamic loads in the mutable unified task. The fine-tuning task makes concessions for

---

Finetune or Evaluste
Throughput (tokens/S)
Decode Throughput
3,000
tne cining to evalnation.
900
(tokens/s)
2,000
600
1,000
K7S
300
0
50
100
150
200
250
300
350
400
Time (S)
Fine-tuning
Evaluation
Decoding
Figure 5: Performance of Loquetier under dynamic load in unified task.
Finctume or Evaluste
Throughput (tokens/))
Decode Throughput
3,000
900
My
(tokens/)
2,000
600
1,000
300
1,000
2,000
3,000
4,000
5,000
6,000
7,000
Time (s)
Fine-tuning
Evaluation
Decoding
Figure 6: Performance of Loquetier under simulated real-world load unified task.
the inference task to ensure the quality of service when request throughput increases, and adjusts back the efficiency by itself when throughput decreases.
Simulated Real-world Workload. To better simulate real-world serving conditions, we construct a composite workload using slices from the BurstGPT dataset, mentioned in Section 4. Each slice contains request arrival times, input lengths, and output lengths. In our simulation, we fully utilize the request arrival times and reference the input length data. For overview of sampling data and preference adjustments, see Appendix [D.]] As shown Figure [6] Loquetier demonstrates strong adaptability well-world workloads, aligning closely with trends evaluated in the earlier simulation experiments. The final SL.O for the entire experiment reaches 92.37%. All requests that failed to meet service metrics occur during transient workload spikes under high-load conditions (RPS 5), which exceeded the load capacity of the hardware. In other periods, Loquetier consistently achieves the defined SLO.
Conclusion
We present Loquetier, a virtualized multi-LoRA framework that runs fine-tuning and inference tasks uniformly. Loquetier performs well in the inference task, with an SLO attainment 20.8x higher than PEFT, and up to 3.0x that of FlexLLM at high request arrival rates, maintaining comparable efficiency the fine-tuning task. In unified task, inference efficiency is maintained as much as possible with an SLO attainment 46.4x higher than PEFT, well balancing the performance of fine-tuning and inference tasks.
Acknowledgments and Disclosure of Funding
We thankful the anonymous reviewers for their helpful comments. This work is supported by Jiangsu Science and Technology Major Project (#BG2025005), the National Natural Science Foundation of China (Grants #62172199), the Collaborative Innovation Center of Novel Software Technology and Industrialization, and Jiangsu Wukong Intelligent Computing Digital Technology. Jingwei Xu (j jjinweeixejjr edu. en) is the corresponding author.
10

---

References
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, and others. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0928. 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and others. The Ilama 3 herd of models. XrXiv preprint arXiv:2407.2178. 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and others. Qwen technical report. arXiv preprint arXiv:2309.1660 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and others. Qwen2. 5 technical report. arXiv preprint 15111:242.151155 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding. Kai Dong, Qiushi Du, Zhe Fu, and others. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401. 029544, 2024.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, and others. Parameter-efficien fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-2333 2023a. Publisher: Nature Publishing Group UK London.
Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. Parameter efficient fine tuning: A comprehensive analysis across applications. arXiv preprint arXiv:2404. 13000 2024.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. 2023a.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficien fine-tuning. Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 12799-12807, 2023. Issue: 11.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 104.08691, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin A Raffel. Few-Shot ceaumeter-Efficienn Fine- Tuming is Better and Cheaper than In-Context Learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 1950- 1965. Curran As- sociates, Inc., 2022. /tRL/attpp://proceeddngg newrriinininnnnnnnnoooe.oer neurips cc/ cappaper........pppp paapppaarfilll files/ www.s//pppee////////// 2022r/202// fileeocceebbssddeBe1666500000663 Cooaapee--ooffeeeeeee
pdf
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Kei Lec. LIm-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304.0193. 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, and others. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
11

---

Nam Hyeon- Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for coamunication-effiienn federated learning. arXiv preprint wxiv:2108.06099, 2021.
Dawid J Kopiezko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.1 2023.
Shih- Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023.
Xiaoyu Kong, Jiancan Wu, An Zhang, Leheng Sheng. Hui Lin, Xiang Wang, and Xiangnan He. Customizing Language Models with Instance-wise LoRA for Sequential Recommendation. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 113072-113095. Cur- ran Associates, Inc., 2024. URL http:: http:ttee/// neurips cc/ wwwpapppp_aaeeeeeeceeee satrrffflleepppae paper/
Jiachen Zhu, Jianghao Lin, Xinyi Dai. Bo Chen, Rong Shan, Jieming Zhu, Ruiming Tang, Yong Yu, and Weinan Zhang. Lifelong personalized low-rank adaptation of large language models for recommendation. arXiv preprint arXiv:2408. 03533, 2024.
You Zhang, Jin Wang. Liang-Chie Chih Yu, Dan Xu, and Xuejie Zhang. Personalized LoRA for human- centered text understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 19588-19596, 2024. Issue: 17.
Yujia Wu, Yiming Shi, Jiwei Wei, Chengwei Sun, Yang Yang, and Heng Tao Shen. Difflora: Generat- ing personalized low-rank adaptation weights with diffusion. arXiv preprint 00774. 2024a.
Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Llm inference serving: Survey of recent advances and opportunities. arXiv preprint arXiv:2407.12391, 2024a.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving. Proceedings of Machine Learning and Systems, 6:1-13, 2024.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng. Kurt Keutzer, and others. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:231 1.03285, 2023.
Xupeng Miao, Gabriele Oliaro, Xinhao Cheng, Mengdi Wu, Colin Unger, and Zhihao Jia. FlexLLM: A System for Co-Serving Large Language Model Inference and Efffcietr-Efficientt Finetuning. arXiv preprint ar 2402 1789,, 2024.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624 2023.
In Gim, Guojun Chen, Scung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, 6:325-338, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611 1266, 2023.
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. vatten- tion: Dynamic memory management for serving llms without pagedattention. In Proceedings of the 30th ACM International Conference on Architectuall Support for Programming Languages and Operating Systems, Volume 1, pages 1133-1150, 2025.
Bin Lin, Chen Zhang, Tao Peng. Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, and others. Infinite-llm: Efficient IIm service for long context with distattention and distributed kveache. arXiv preprint 2xX1v:2401.02669, 2024.
12

---

Bingyang Wu, Shengyu Liu, Yinmin Zhong. Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serving long-contex large language models with elastic sequence parallelism. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 640- 654, 2024b.
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length perception and sequence scheduling: An lIm-empowered Ilm inference pipeline. Advances in Neural Information Processing Systems, 36:65517-65530 2023.
Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. S^ 3: Increasing GPU Utilization during Generative Inference for Higher Throughput. Advances in Neural Information Processing Systems, 36:18015-18027 2023.
Gy Yu, Joo Scong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gov Gon Chun. Orca: A distributed serving system for Saatnsfooerr-Buseee generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and others. Deepspeed- fastege: High-throughput text generation for Ilms via mii and deepspeed-inferenae arXiv preprint (xXx:2440100867.. 2024.
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming (TTrouughput-Lateed ) tradeoff in (LLM) inference with /Sarathi-Serve) In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117-134, 2024.
Cunchen Hu, Heyang Huang. Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang. Sa Wang. Yungang Bao, and others. Inference without interference: Disaggregate Ilm inference for mixed downstream workloads. arXiv preprint arXiv:2401. 11881, 2024.
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative Ilm inference using phase splitting. In 2024 ACM/IEEE 51 Annual International Symposium on Computer Architecture (ISCA), pages 118-132. IEEE, 2024.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. (DistServe): Disaggregating prefill and decoding for goodput-optimizer large language model serving. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193-210, 2024.
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++:: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics. Proceedings of Machine Learning and Systems, 6:148-161, 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2337.0866)) 2023.
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37:68658-68635 2024.
Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv preprint arXi::2501.01005
2025. URL BtRLhhtts::/// //arxivv////ii.///////// orggoo/////ooo abs abs//50111.0 01005
Vijay Thakkar, Pradeep Ramani, Cris Ceeka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS, January 2023. URL https: : //gi thub. com cutta//ssi///tlsssssss..
13

---

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effiient fine- tuning. arXiv preprint arXiv:2303. 105 2023a.
Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. Increlora: Incremental parameter allocation method for parameter-efficient tinettnnig. arXiv preprint arXiv:2308. 12044, 2023b.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311. 16696, 2023b.
Tangyu Jiang, Haodi Wang, and Chun Yuan. DiffoRA: Enabling Parameter-Efffcieet LLM Fine- Tuning via Differential Low-Rank Matrix Adaptation. arXiv preprint arXiv:2502.08905, 2025.
Shih-Yane Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, K Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. Forty-first International Conference Machine Learning, 2024.
Yang Li, Shaobo Han, and Shihao Ji. VB-LoRA: extreme parameter efficient fine-tuning with vector banks. arXiv preprint arXiv:2405.1517, 2024b.
Klaudia Batazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprini arXiv:2405./7604, 2024.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awan low-rank adaptation of large language models. Xiviv preprint arXiv:2309. 14717, 2023b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-lee the Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Compu- tational Linguistics. URL https: www. aclweb. org/ hwrg/anthology/// 2020 demos..eemo666
Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang. Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. BursIGPT:
A Real-Works Workload Dataset to Optimize LLM Serving Systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '25). Toronto, ON, Canada, 2025. ACM. doi: http://....oo001111//71116677711 URL https //doi.org//// 10w.11114/37111 3737443
14

---

A Limitations
Loquetier can be improved by combination with other training and fine-tuning optimization methods, and by applying other cluster management systems to provide clusterized services. We plan to provide a backward propagation kemel operating in concert with the SMLM kernel to accelerate fine-tuning.
Co-operating with forward propagation computations during backward propagation can go some way to balancing the bottleneck of GPU memory bandwidth and computational resources for fine-tuning and inference tasks, and Loquetier can be further improved in this regard. We also consider providing support for other LoRA-like PEFT methods.
B Solutions of FL Backward Procedure Issues
Several cases of operation remain unimplemented in FlcxLLM's gradient computation logic, including OP GELU, OP RELU, OP SIGMOID, OP OP_TANE: and OP ELU. We noticed that their repository contained forward and backward kernels related to these operations, but they had never applied these backward kernels to their computation flow, resulting in its inability We ultimately built a runnable version by instantiating these kernels at the missing locations. This fix was based on our understanding of their framework, and we did not implement any additional computational steps. The correction will not result in performance degradation of FlexLLM.
perform fine-tuning tasks.
C Experimental Metrics
We use the following experimental metires:
Service Level Objective (SL.0): Measuring service satisfaction a level.
SLO Attainment: Percentage of all requests reaching the given SLO.
Request Throughput: Throughput of incoming inference requests. Measured as request per second (RPS).
Throughput of inference requests in decoding. Measured as decode token per second (DTPS).
Decode Throughput:
Finetune Throughput:
Throughput of fine-tuning requests in training forward. Measured finetune token per second (FTPS).
Throughput of fine-tuning requests in evaluation forward. Measured as evaluate token per second (ETPS).
Evaluate Throughput:
D Experimental Settings
D.I SLO
For all inference requests in the experiments, use the following targets in Figure 3 as SLO.
Since PEFT performs batch generation with padding, resulting in additional computational delays in prefilling and decoding, we do not require more decoding aspects for PEFT.
Table 3: SLO settings.
Mean Decoding Latency (ms)
Max Decoding Latency (ms)
Framework or System
Max Waiting Time (s)
Loquetier
200
1,000
PEFT
FlexLLM
200
1,000
15

---

Table 4: Inference-only tasks configurations. The number of requests in multiple (4) LoRAs is
expressed as the total number number of one LoRA.
Single (1) LoRA
Multiple (4) LoRAs
Throughput (RPS)
Requests
Max New Tokens
Requests
Max New Tokens
800
400
800 / 200
400
2
1,600
400
1,600 400
400
2,400
400
2,400 / 600
400
4
3,200
300
3,200 / 800
300
4,000
200
4,000 500
200
D.2 Inference
We the following configurations in Figure to test the inference-only tasks. Note that the maximum tokens supported by FlexLLM is 1024, so all inference requests for FlexLLM do not exceed this limit.
D.3 Fine-tuning
We use the following configurations in Figure 5 to test the fine-tuning-onl tasks.
Table 5: Fine-tuning-only tasks configurations.
Configurations
Single (1) LoRA
Multiple (2) LoRAs
LoRA Config
(rank)
8
lora_alpha
16
16
lora_dropout
0.05
0.05
bias
none
none
task type init lora
CAUSAL LM
CAUSAL LM
lora_ weights
gaussian
gaussian
Training Args
per_device_traies aaain__atch___
1
per_device_eval batchaahhhh size
1
num_train, train_epochs
4
eval_strategy
epoch
epoch
logging_strategs
steps
steps
logging_steps
100
100
eee_sstttegy
epoch
epoch
learning rate
2e-5
2c-5
gradient_ accumulatccnnnnnnncccc.oo steps
4
report_to
none
none
D.4 Unified fine-tuning and inference
We use the following configurations in Figure 6 to test the unified tasks.
Note that the maximum tokens supported by FlexLLM is 1024, so all inference requests for FlexLLM do not exceed this limit.
D.5 Mutable capacity allocation simulation
We use the following configurations in Figure 7 for inference requests and the single LoRA fine-tuning configurations in Figure fine-tuning requests to test the mutable unified tasks.
16

---

Table 6: Unified tasks configurations.
Single (1) LoRA
Multiple (4) LoRAs
Throughput (RPS)
Requests
Max New Tokens
Requests
Max New Tokens
600
400
600 / 150
400
1,200
400
1,200 / 300
400
3
1,800
400
1,800 / 450
400
2,400
300
2,400 600
300
5
3,000
200
3,000 750
200
Configurations
Single (1) LoRA
Multiple (2) LoRAs
LoRA Config
(rank)
8
(ora_alpha
16
16
lora dropout
0.05
0.05
bias
none
none
task_type
CAUSAL_ LM
CAUSAL_IM
init_lora_ _eri__
gaussian
gaussian
Training Args
per_device_train batch_sstii size
I
je__device_eva__oe size
i
num num_train_epochs
1
eval strategy
epoch
epoch
brgging_stratepe
steps
steps
logging_stepp
100
100
save_stree see_strategy
epoch
epoch
learning_ rate
2e-5
2e-5
gradient_acee htt_aaaccclcaoosssseeleeeo
4
report_to
none
none
Table 7: Mutable unified tasks configurations.
Multiple (4) LoRAs
Index
LoRA Index
Requests
Throughput (RPS)
Start at (s)
Duration (s)
120
0
120
1
150
2.5
120
60
2
240
2
180
120
3
120
1
300
120
D.6 Simulated real-world workload
We use the following time periods in Figure to test (the inference task of) the simulated real- world workload. The fine-tuning task use the same configuration mutable capacity allocation simulation.
Based on our analysis of the BurstGPT dataset, less than one-third the time periods correspond to high-load scenarios, with the majority being lowd periods. Given the lower challenge of low-load periods, we adopt the configuration above to ensure representative coverage. Among the selected workloads, hogh-load periods include several minutes where the RPS exceeded 5, with a peak RPS of 11.
E Detailed Information Related to S-LoRA in the Experiment
S-LoRA does not support the LLaMA series models, and repository has been archived. This limitation arises from the Group Query Attention (GQA) architecture used in LLaMA 3, where the shapes of and V differ from those of and O. Consequently. the shape of the weight matrix B in the LoRA linear layers for K and V also differs from Q and O. Current S-LoRA requires all LoRA
17

---

Table 8: Time periods configurations. Peak RPS refers to the highest RPS within 2-second interval.
Time Period
Requests
Mean RPS
Peak RPS
Day 29, 13:00 11:20
676
0.563
1.5
Day 29, 15:00-15:::
2,145
1.788
11.5
Day 29, 16:00 -16:20
1,465
1.226
7
Day 33, 13:40 -14:00
2,823
2.354
10
Day 33, 11:40 -12:00
2,360
1.966
12
Day 33, 11:00 -11:20
1,856
1.547
10.5
weights within the same layer to be concatenated at runtime. However, due the shape discrepancies mentioned above, this concatenation operation fails. As workaround, we replicate K and V weights in advance during model initialization enable S-LoRA start properly.
"Partial" Figure means that only 4 modules are enabled for S-LoRA including q. k, v, and o, as S-LoRA supports applying LoRA only on these 4 linear layers, and does not support the up, gate, and down layers within the MLP. Therefore, its runtime efficiency resembles the Partial scenario described in our paper, where only three linear layers (up, gate, down) are targeted.
In experiments, we observed instability in the S-LoRA kernel, which frequently produced incorrect outputs leading to NaN or Inf values. These errors propagate quickly, causing model generation failures. this time, we did not modify the kernel. Our preliminary analysis suggests this may be due to missing synchronization mechanisms in some computational steps. (Note that this is an initial observation and may not definitive.)
Due to these issues, S-LoRA struggled to complete all inference requests in our scenarios, as it frequently outputs the token directly, making SLO appear better than it actually should be.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providded a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper proposes a redesigned kernel SMLM and unified computation flow for the unified operation of fine-tuning and inference tasks for LoRA models (Section (33). and Virtualized Module implementation to isolate PEFT-based model modifications and flexible instance-to sittnce-to-innsance migration (Section 3.2)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
19

---

Answer: [Yes]
Justification:
The limitations of the work discussed in Section A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper discloses all the information needed for reproducibility (Section 4.1.. This paper releases the code for quick reproducibiliity The implementation code www.hau..oom//aaaaaaaaeeeee available http:p///gggtoooooo/oo/o////////
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The implementation code is available at https: com/ 33o33w55br2dffff//oouttt.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic/guui.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The experimental settings are presented in Section 4.1 More details could be found in our code repository, which is available at https: //g1 thub. com/
Justification:
3Bcosw)55tr2dfffLoqqutt..
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experimental metries already the statistical result of large amount of data, so there no need to conduct multiple experiments to plot the errorhar. For the data in Figure we took a total of 5 data before and after each data point for smoothing operations to improve chart readability and eliminate extreme values over short periods of time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
22

---

Justification: We present the information on the computer resources in Section 4.1 That is, we the inference-only tasks on a server with NVIDIA A6000 48G GPUs. We test the fine-tuning-only tasks and the unified fine-tuning and inference tasks on servers with NVIDIA H800 80G GPUs. Each test process has least 128G of host memory available.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: wwwpeeewiisc..ppuuupeeeepe http:///.cc.........t.cccciiiiia
Answer: [Yes]
Justification: anonymity.
We conformed the NeurlPS Code of Ethics and make sure to preserve
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed. Our work proposes a framework for unified LLM fine-tuning and serving multiple LoRA models, which does not lead to any negative socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., degloomment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
23

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper presents a framework for unified LLM fine-tuning and serving with an SMLM kernel for unified tasks, unified computation flow management, and an implementation of Virtualized Module. This framework does not include any data or models, and therefore this work does not pose any associated risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in the paper are listed here. Alpaca: Creative Commons
ctt.taaccc..o/aaaaaaeeeeeeaaaaae www.o//aaattt/taaoooooooooeeo ttatttttttttttttte lab/ alpaca GSM8K: MIT License, htttp::/huugiiggaaaat
Attribution Non Commercial 4.0,
co http//aaaeettss//ooooo hptp:s///pppaaaiaaaeaeaaeaa gsm8k Llama3-8B: Llama Community License Agreement, httpp::/bugging
co.
Llama - 8B ShareGPT Vicuna: Apache License Version 2.0, https
/hhggingfacce.uummmm co/ datasets/ anon82314891222
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our released code is well organized in the repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
25

---

Answer: [NA]
The entire approach development in this research does not involve LLMs as any important, original, non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not non Please refer to our LLM policy fattps://neuus wwweeurrpssccoooofeeeeeeeeeeeeee www.onnerrenccs//200000. www.eennneeee2///111/ for what should or should not be described.
involve LLMs as any important, original, non ntn-standard components.
26


Reviews:
[{"review_id": "u6PEolWAKj", "reviewer": "Reviewer_fRyY", "summary": "The authors propose Loquetier, a framework for serving and fine-tuning multiple LoRA models simultaneously. The authors describe the workings of their framework and compare its performance with other setups for training and evaluating LoRA fine-tuned models.", "questions": "- I would be grateful if the authors could address my comments regarding comparisons with other methods (adding more baselines in each of the serving and fine-tuning settings).\n\n- I would also like to know if they were able to bypass the issues presented during their previous iterations with FlexLLM, by building the tool from scratch.\n\nImproving upon both of these points would be great for the paper, and I would consider increasing my score.", "limitations": "The authors have adequately addressed the limitations and impact of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written and easy to follow. I didn’t have any issues with following the ideas presented or the arguments made by the authors.\n\n- The proposed framework is also quite comprehensive. Being able to accommodate a variety of use cases is an important ability for a good framework. The fact that Loquetier supports both single and multi LoRA serving and fine-tuning makes it suitable for use in a variety of contexts.\n\n- The structure of the framework itself is solid. The use of the virtualized module as described by the authors serves well to reduce the memory footprint of the model and allow for easier sharing of components across different LoRA instances.\n\nWeaknesses\n\n- My main concern with this paper is the benchmarking of Loquetier. Currently the only meaningful comparison made is to PEFT (as results with FlexLLM are really only presented in the single LoRA serving case, as seen in Table 2). Given that PEFT is very simple to begin with, I am not sure how well Loquetier compares to other frameworks. For the serving case at the very least, the authors should compare to more frameworks, such as VLLM and S-LoRA, to make comparisons more convincing (and ideally more frameworks should be added in the fine-tuning case as well).\n\n- Regarding FlexLLM, I am a bit surprised that results are not presented for the single LoRA fine-tuning case, as this is a mode that is explicitly presented. The authors mention that an unsupported operation prevented them from evaluating this case. In Section 4.1, the authors mention that they used prebuilt docker images to use FlexLLM. I think that the unsupported operation should be avoided if FlexLLM is built from scratch instead. If building the framework from scratch has not been done, I think it should be since that should allow for FlexLLM to be properly included as a comparison framework.\n\n- On a more minor note, I understand the main technical novelty to be the virtualized module. If so, the authors should highlight this a bit more and also directly contrast choices made by other frameworks in Section 3. Doing so would allow for the technical novelty of the paper to be better understood.\n\n**Post Rebuttal Comments**: As noted below, the additional experimental results included in the rebuttal strengthen the paper, and I have thus raised my score.", "was_revised": true, "final_justification": "Despite my initial concerns about this work, the authors have put a lot of effort into addressing both mine and other Reviewers' concerns, especially regarding comparisons with FlexLLM. The additional experimental results support the usefulness of this framework, and thus I think that most of my concerns have been addressed. While there are still only a few comparisons with existing baselines, this can be attributed to the fact that the framework proposed by the authors tackles a unique problem in Multi-LoRA model serving.", "rebuttal": "Thank you for your detailed review of our work. Our responses to the weaknesses and questions you raised are as follows:\n\n# Baselines\n\nWe have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.\n\nFor S-LoRA, FlexLLM have compared with it. S-LoRA is only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM. \n\nAs mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.\n\n# Further Attempts on FlexLLM testing\n\nFor the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.\n\nWe will further attempt to rebuild FlexLLM from scratch based on more commits, which will take some time. We will provide feedback on the results after thorough testing. However, as mentioned in the paper, FlexLLM is missing a critical operation operator (the case for OP_CONV2D in FusedOp::peft_bwd_task), and we have not found any implementation records for this operator in historical commits. Therefore, we do not expect the built version to support LoRA fine-tuning functionality.\n\n# Comparison of Virtualized Module and Selections from Other Works\n\nThe implementation of PEFT directly modifies the base model, thereby rendering it non-reusable once processed. PEFT adopts a high-coupling design for LoRA, incorporating multiple parameters from different LoRAs within the same linear layer module. For different efficient fine-tuning methods, it designs the MixedModel module and related modules, enumerating and storing each supported efficient fine-tuning method for storage and invocation, causing different methods mixed together, resulting in extremely high coupling and poor maintainability.\n\nFor other frameworks, we categorize them into two types. The first type involves frameworks designed from scratch and built from the ground up. Since these frameworks do not need to consider compatibility issues, they generally do not retain the original structure of the base model, as the framework can handle structural modifications. The second category includes systems like S-LoRA and Punica, which merge multiple LoRA models for processing and directly modify the original model structure. This not only renders the original model unusable by other methods but also couples different LoRA models together, thereby reducing operability."}, {"review_id": "25MeX6qWNy", "reviewer": "Reviewer_WdRD", "summary": "The paper presents Loquetier, a virtualized multi-LoRA framework that unifies fine-tuning and serving. It features virtualized modules that isolate model modifications, along with an optimized computation flow and kernel design that merges training and inference paths. The authors conduct an empirical study using three workloads and demonstrate that Loquetier consistently outperforms existing baselines.", "questions": "Please refer to the Weakness section. The paper contains several major issues, and due to their breadth and severity, it's difficult to distill them into a concise set of specific questions.", "limitations": "Yes.\n\nThat said, the listed limitations appear to be superficial and do not address the paper's deeper methodological or conceptual shortcomings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nThe paper introduces the Segmented Multi-LoRA Multiplication (SMLM) kernel and leverages it to unify fine-tuning and inference tasks with multiple LoRA adapters. This architectural direction is promising and relevant for scalable multi-task LLM deployment.\n\n## Weakness\n\n### Writing Quality\n\nThe paper is not well written, and suffers from a lack of clarity and structure. A few specific symptoms include:\n\n**Unclear motivation:**\n\nThe motivation for unifying fine-tuning and inference with multi-LoRA is difficult to follow. On the surface, this work appears to be a natural extension of FlexLLM, which clearly stated its goal: to dynamically reallocate resources for inference workloads under high-traffic conditions, given the strict latency requirements. In contrast, this paper does not articulate why unifying fine-tuning and inference is important, what problem it aims to solve, or how success should be measured.\n\nThe paper claims 3× performance improvement and 46.4× SLO attainment, but fails to explain whether these improvements are addressing a critical bottleneck. Is the 3× performance gain addressing a real, high-priority need? Is the 46.4× SLO improvement sufficient to meet practical latency constraints? Without clearly stated goals, the work risks appearing like it \"shot first and drew the target afterward.\"\n\n**Dangling table:**\n\nTable 1 is presented without any explanation of its purpose or guidance on how to interpret it.\n\n**Insufficient experimental details:**\n\nFor example, as noted in the FlexLLM paper, ShareGPT data lacks critical metadata such as request arrival times. FlexLLM addressed this issue with corrective measures. This paper does not mention whether this limitation was considered, or whether any mitigation strategies were applied.\n\n### Lack of Motivation\n\n(See above.) Without a well-articulated motivation, it’s difficult to assess the importance or relevance of the proposed solution.\n\n### Poor Experimental Design\n\nThe paper avoids any meaningful comparison with FlexLLM, claiming that it \"is trapped in an infinite loop while loading LoRA models under multi-LoRA inference.\" However:\n\nFlexLLM was not designed for multi-LoRA, so it's not fair to disqualify it based on that alone.\n\nThe absence of a direct comparison should not prevent using proxy evaluations, especially for latency and resource efficiency.\n\nFlexLLM may have lower resource efficiency but better latency, which aligns with its design objective. The paper fails to explore or even hypothesize on such trade-offs.\n\n### Poor Baseline Selection\n\nSince FlexLLM is deemed \"infeasible,\" the only baseline considered is PEFT, which is not designed for co-serving fine-tuning and inference tasks. This makes PEFT an overly simplistic and weak baseline.\n\nThe related work section cites several relevant approaches, but none of them appear to have been included as competitive baselines in the experiments.\n\n### Poor Workload Design\n\nAs mentioned earlier, the ShareGPT dataset lacks critical information such as request arrival times, making it difficult to evaluate the realism or representativeness of the simulated workload. Without addressing this, the trustworthiness of the experimental results is significantly weakened.", "was_revised": true, "final_justification": "Thank you for the great effort during the rebuttal phase and the comprehensive clarification. All of my concerns are addressed. I'm raising my score to 4.", "rebuttal": "Thank you for your detailed review of our work. Before responding to the weaknesses and questions you raised. We clarify our research background, scenario definition, and metrics used to measure results.\n\n# Background, Scenario Definition, and Key Experimental Metrics\n\nLoquetier is a framework for unified fine-tuning and serving of multiple LoRAs. As mentioned in the Introduction, the research background is that LoRA and its related efficient fine-tuning techniques have demonstrated excellent performance across various specialized tasks and show significant potential in many tasks, such as personalization and user preference-related tasks, leading to their increasingly widespread adoption and promising future applications. However, the hardware resources available for LLM training, fine-tuning, and serving remain extremely limited. This leads to our application scenario: how to improve the efficiency of multi-LoRA fine-tuning and inference tasks under constrained hardware conditions.\n\nSpecifically, most existing system frameworks require a hard switch when handling different tasks, meaning they must first stop the current task before starting another. Such switches consume significant time and require saving fine-tuning states and migrating inference requests, which occupy substantial bandwidth and storage space. During downtime, the occupied computational resources cannot be utilized, leading to a significant decline in the system's hardware resource utilization efficiency.\n\nUnifying fine-tuning and inference of multiple LoRAs upon one base model can address the aforementioned issues and improve the efficiency of multi-LoRA fine-tuning and inference tasks. Integrating the two distinct tasks is a practical approach, as the only difference between them lies in the gradient calculation and parameter updates during the backpropagation process. Regardless of the task type, sharing the same base model can significantly reduce memory requirements.\n\nAccording to the description in the Experiments section, we measure the efficiency of these two tasks as follows, and list the meanings of all experimental metrics in Appendix B:\n\nThe efficiency metric for the fine-tuning task is the throughput of fine-tuning and evaluation, i.e., the number of tokens fine-tuned and evaluated per unit of time, which is equivalent to the time required to fine-tune with the same data and configuration.\n\nThe efficiency metric for the inference task is inference throughput, which is the number of tokens inferred per unit of time. Additionally, serving metrics targets are used to define the availability of the service system, so the secondary metric is the service level objective attainment (rate). Appendix C clarifies the service metric settings, including the maximum request wait time, maximum decoding latency per token, and average decoding latency. By calculating whether each request meets these strict serving metrics, our experimental results have sufficient practical significance.\n\n# Motivation and Objectives\n\nOur motivation and objectives are mentioned in the paper and the first section \"Background, Scenario Definition, and Key Experimental Metrics\" in response. Our work is not an extension of FlexLLM but rather an optimization of unified fine-tuning and inference work in the rapidly growing and highly promising LoRA domain. We also present experimental metrics for evaluating these tasks and compare our results with other works under the same conditions, demonstrating significant improvements achieved by our approach.\n\nRegarding Table 1 in the paper, we naturally use symbols and footnotes to indicate whether the baseline can complete each scenario. We use $\\checkmark$ to indicate that it can run in the corresponding scenario, $\\triangle$ means it encountered some issues that caused the test to fail but can be achieved by degrading to a more basic method, and $\\times$ refers that it cannot run in that scenario.\n\nRegarding the high-priority requirements you mentioned, according to technical reports, official blogs, and other customer cases from leading internet companies and cloud platforms, such differentiated services are typically provided at the application layer, including real-world scenarios involving the scalable use of LoRA. Our work is a unified underlying framework for fine-tuning and inference tasks, designed to efficiently handle incoming requests to the best of its ability. Loquetier is compatible with various cluster management frameworks and business frameworks, so such requirements can be fulfilled through collaboration with these systems. Based on related work involving multi-LoRA or multi-model parallel inference, including FlexLLM which adopts Orca's research findings, no priority differentiation or other differentiated services have been implemented. As mentioned earlier, this is not the responsibility of the underlying framework.\n\n# Experimental Details Related to FlexLLM\n\nFlexLLM is the first work in the field of unified fine-tuning and inference for multi-PEFT models. We fully acknowledge its pioneering nature and have never denied its contributions in our paper. FlexLLM still has limitations in the specific scenarios it claims to support, which is precisely why we should use it as a critical baseline for comparison. We were also surprised by FlexLLM's performance in multi-LoRA inference scenarios, but our experimental results indicate that it cannot support multi-LoRA inference in a single runtime instance.\n\nPlease allow us to update the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test designed for 5 minutes. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.\n\nWe will conduct additional tests to fully demonstrate FlexLLM's performance in multi-LoRA inference scenarios. We will provide feedback on the experimental results as soon as these tests are completed.\n\nRegarding the trade-off between resource efficiency and latency, we will elaborate further from the following two aspects:\n\nAs mentioned in the first section, we are more concerned with the performance of multi-LoRA tasks when a large number of requests arrive under limited hardware resources. As stated in your review comments, this differs from FlexLLM's objectives. We have outlined the experimental metrics in Appendix B of the paper and in the first section in our response, and these metrics comprehensively reflect whether we have achieved our stated objectives.\n\nAdditionally, we mentioned the issue of differentiated services in the “Motivation and Objectives” section. Since altering the request arrival rate of inference tasks also affects latency, we believe this conclusion also applies to differentiated services.\n\nWe have still managed to conduct inference latency evaluations in both single-LoRA and multi-LoRA scenarios to address your concerns. However, our background and scenario definitions are focused on the fine-tuning and inference of multiple LoRA models, so the comparison of single-LoRA inference scenarios is not our primary focus.\n\nIn a single LoRA scenario, FlexLLM achieves 60 tokens/s (tps), corresponding to a latency of 17ms, while Loquetier reaches 30tps and 33ms. In a multi-LoRA scenario (4 LoRA models with 1 infer request per LoRA), FlexLLM achieves 38 tps and 105ms, while Loquetier reaches 120 tps and 33ms. The Peft baseline lags significantly behind in both scenarios. To better address the scenarios we are discussing, Loquetier's design approach remains the optimal solution at present.\n\n# Baselines\n\nWe have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.\n\nFor S-LoRA, FlexLLM have compared with it. S-LoRA only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM. \n\nAs mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.\n\n# Workload Design\n\nAs for the metadata you mentioned that FlexLLM created from third-party sources, this was added in the version of the paper submitted in May 2025. When we were working on Loquetier, we referred to the only version available at the time (the first version in Feb 2024). We'd rather compare it to that version, but we are open to further discussion on the points you raised. Please let us know if you would like to discuss this part further."}, {"review_id": "X9qY5vt6if", "reviewer": "Reviewer_tvxa", "summary": "This work presents Loquetier, a virtualized multi-LoRA framework that enables unified LoRA fine-tuning and serving. Specifically, Loquetier consists of two main components: (1) a Virtualized Module that isolates model modifications to support multiple PEFT methods within a shared base model architecture, and (2) an optimized computation flow and kernel design that merges fine-tuning and inference paths in forward propagation, enabling efficient batching and minimizing kernel invocation overhead. Experimental results demonstrate its effectiveness.", "questions": "none", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. This work investigates an important problem: how to unify fine-tuning and inference processes for LoRA-based models?\n\n2. The paper is well written and clearly structured. Also, the proposed method is effective compared to baselines.\n\n3. The authors provide code for reproduction.\n\nWeakness:\n\nThis is **not** my area of expertise (I will also give **low confidence** for my rating), so I may have missed some potential weaknesses. One potential concern is that the evaluation only includes FlexLLM as a baseline. However, the \"Related Work\" section mentions several other methods targeting efficient multi-LoRA inference. Including comparisons with them may be necessary.\n\nBesides, the experiments are limited to Llama3-8B. Whether the findings can be generalized to larger models remains unverified.\n\nFinally, it would be better to evaluate the method on more challenging tasks beyond alpaca and GSM8K.", "was_revised": true, "final_justification": "The author's response has successfully resolved some of my concerns. I tend to keep my score.", "rebuttal": "Thank you for your detailed review of our work. Our responses to the weaknesses you raised are as follows:\n\n# Baselines\n\nWe have received valuable suggestions from the reviewers regarding the evaluation. We would like to provide additional explanations and the derived results to address the issues raised.\n\nFor S-LoRA, FlexLLM have compared with it. S-LoRA is only optimized for inference tasks and performs worse than FlexLLM in fine-tuning and inference tasks; FlexLLM's experiments include the Peft baseline for fine-tuning tasks, and our results in similar experimental settings are better than the equivalent fine-tuning performance of Peft in the S-LoRA+Peft baseline in FlexLLM's experiments, while the performance of the Peft is better than that of FlexLLM in several experimental environments (FlexLLM has not tested a direct comparison with Peft when only performing fine-tuning tasks; based on the result in their paper and the number of GPUs used, we can infer that Peft's fine-tuning efficiency is higher than that of FlexLLM under the same hardware resources). Therefore, based on the proxy evaluation method, we can also confirm that our performance is better than that of FlexLLM and S-LoRA. Additionally, S-LoRA significantly outperforms vLLM in its multi-LoRA inference experiments, and its experimental environment is comparable to ours. Furthermore, S-LoRA does not support newer models, including Llama 3. For these reasons, we did not design experiments comparing S-LoRA and vLLM.\n\nAs mentioned in the Introduction, there are very few training-inference integrated frameworks that support multi-LoRA. FlexLLM may be the only proper baseline. The other methods mentioned in the Related work are mostly general optimizations for base models or further improvements to the LoRA method itself, and they are not suitable for evaluation as test baselines; our work is also compatible with these methods.\n\nFor the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon.\n\n# Models\n\nSince we are not an enterprise-level organization and are constrained by hardware resources, it is challenging to conduct experiments with larger models. We are willing to analyze the expected results of larger models on multiple GPUs from a theoretical perspective.\n\nThe LoRA method itself does not conflict with common parallel strategies, such as pipeline parallelism or tensor parallelism. Therefore, from a technical standpoint, the fine-tuning and inference of LoRA models can utilize these parallel strategies to load larger models on multiple GPUs. In terms of pipeline parallelism, data only needs to be transferred once between the two layers on different devices, so our work fully supports this parallel strategy. We also fully support parallel strategies like tensor parallelism that require data sharing between GPUs:\n\nThe forward and backward calculations of LoRA can be expressed by the following formulas:\n\n$$V = X @ A,  Y = V @ B$$\n\n$$B' = V^T @ Y',  V' = Y' @ B^T, A' = X^T @ V', X' = V' @ A^T$$\n\nData parallelism is typically split along the hidden_states dimension, so the weight matrix needs to be split and stored across multiple GPUs, with the requirement of an all-reduce or all-gather performed after each of the $V, Y, V',$ and $X'$ computations are complete. Loquetier's computation process can be split using all-reduce / all-gather, so we also support these parallelization strategies.\n\nThis means we can run larger models in a multi-GPU environment without incurring any additional overhead compared to existing methods.\n\nOne point we would like to highlight is that, based on the relevant research we have collected, there has been little targeted optimization for such parallel strategies when using LoRA, especially in fine-tuning tasks, as the LoRA computation itself already has no further room for parallel optimization. Furthermore, for strategies like tensor parallelism, as shown in the above formula, the communication overhead required for each LoRA linear layer to synchronize is  $2 \\times s \\times (r + h) \\times n$, where $n$ is the number of GPUs involved in parallelism. Considering the number of LoRA layers and the dimension of hidden_size in large models, the communication overhead is extremely high compared to the benefits, so these parallel strategies is typically not used.\n\n# Datasets\n\nWe did not propose improvements to the effectiveness for fine-tuning and inference methods, but rather optimizations to the computational processes for multi-LoRA fine-tuning and inference tasks. So there is no need for validating the effectiveness of fine-tuning on vast downstream tasks with F-1, rouge-L, etc. The current dataset is sufficient to evaluate all the experimental metrics mentioned earlier.\n\nIn this paper, we made improvements at the kernel level. The industry typically evaluates computational quality through the following two parts of validation, and based on our validation results, Loquetier does not affect the quality of computational results.\n\n  1. Verifying whether kernel computational outputs align with standard results: We performed 10,000 kernel operations using random inputs, and the results indicate that no errors were generated in kernel computational results at float16 precision.\n\n  2. Verifying whether the loss converges normally during fine-tuning: Based on data generated during the experiment, the loss converges consistently with the PEFT baseline method."}, {"review_id": "qKhUt16Uwd", "reviewer": "Reviewer_VwoD", "summary": "The paper presents Loquetier, a unified framework that integrates low-rank adaptation (LoRA) fine-tuning and inference for large language models, addressing the scalability challenges in parameter‐efficient fine-tuning techniques. The authors motivate their work by highlighting the increasing computational costs of full-model training and the need for dynamic, runtime-adaptable systems that support multiple fine-tuning configurations concurrently. The proposed solution leverages a Virtualized Module to isolate model modifications and an optimized computation flow with the Segmented Multi-LoRA Multiplication (SMLM) kernel to merge fine-tuning and inference processes. Experiments show that Loquetier outperforms competing baselines in both performance and flexibility under various task settings.", "questions": "See weaknesses.", "limitations": "This work has discussed its limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The framework introduces a SMLM kernel that unifies fine-tuning and inference, enabling mixed-task execution with a shared forward propagation path. The kernel design minimizes kernel invocation overhead and streamlines memory management. The approach effectively balances the computational needs of dynamic workloads in large language model serving.\n\n2. The Virtualized Module mechanism in the framework provides clean isolation of LoRA modifications, allowing for independent and concurrent execution of multiple adapter configurations. The design allows for instance-level migration without incurring memory duplication or requiring kernel restarts. The modular abstraction enhances flexibility and supports a wide variety of parameter-efficient fine-tuning methods.\n\n3. Experiment results across inference-only, fine-tuning-only, and unified tasks show the performance gains of Loquetier. The evaluation demonstrates significant improvements in decoding throughput and service level objective attainment compared to established baselines. \n\n4. The framework is compatible with existing LLM inference optimizations, demonstrating interoperability with kernels such as FlashInfer and strategies like dynamic caching. The ability to leverage state-of-the-art inference techniques enhances its practical applicability. The solution shows versatility by supporting various PEFT methods without requiring extensive changes to the base model architecture.\n\n\nWeaknesses:\n\n1. The slight slowdown observed during model loading due to the creation of Virtualized Modules adds additional latency and may affect real-time system scalability. The trade-off between isolation and initialization efficiency is not fully explored, leaving some uncertainty about performance in latency-critical environments.\n\n2. The framework is primarily tested on LoRA-based PEFT techniques and may not generalize seamlessly to other fine-tuning methods or modifications outside of the LoRA family. The experimental evaluation does not include a variety of alternative PEFT methods, which limits insights into broader applicability.\n\n3. The experimental setup only includes the LLaMA-8B model and the shareGPT, alpaca and GSM8k datasets. However, many real-world datasets that require finetuning and personalization should be considered, which is the core motivation of providing large-scale finetuning services.\n\n4. The writing should be largely improved. Considering that authors propose a new setting, there should be a clear introduction of what are inputs and outputs. How do users use this system and what their dataflow? While the motivation of providing finetuning and serving of PEFT methods is clear, the motivation of the framework design is unclear.\n\n5. While this work provides a new kernel optimization and a Virtualized Module, from the system perspective, there still lacks a clear optimization of how the memory and computation are optimized. This is partly because that authors do not clearly describe the scenario. It is recommended to add some sections to clearly describe the scenarios, using some timeline figures to convey challenges and opportunities of system improvement. Besides, only the code implementation is a weak contribution from the system perspective.", "was_revised": true, "final_justification": "Thanks for the rebuttal. Some of my concerns are addressed. I'd like to increase my score.", "rebuttal": "Thank you for your detailed review of our work. Before responding to the weaknesses and questions you raised, please allow us to once again clarify our research background, scenario definition, and the experimental metrics used to measure results.\n\n# Background, Scenario Definition, and Key Experimental Metrics\n\nFirst, please allow us to reiterate our research background, scenario definition, and the experimental metrics used to measure results.\n\nLoquetier is a framework for unified fine-tuning and serving of multiple LoRAs. As mentioned in the Introduction, the research background is that LoRA and its related efficient fine-tuning techniques have demonstrated excellent performance across various specialized tasks and show significant potential in many tasks, such as personalization and user preference-related tasks, leading to their increasingly widespread adoption and promising future applications. However, the hardware resources available for LLM training, fine-tuning, and serving remain extremely limited. This leads to our application scenario: how to improve the efficiency of multi-LoRA fine-tuning and inference tasks under constrained hardware conditions.\n\nSpecifically, most existing system frameworks require a hard switch when handling different tasks, meaning they must first stop the current task before starting another. Such switches consume significant time and require saving fine-tuning states and migrating inference requests, which occupy substantial bandwidth and storage space. During downtime, the occupied computational resources cannot be utilized, leading to a significant decline in the system's hardware resource utilization efficiency.\n\nUnifying fine-tuning and inference of multiple LoRAs upon one base model can address the aforementioned issues and improve the efficiency of multi-LoRA fine-tuning and inference tasks. Integrating the two distinct tasks is a practical approach, as the only difference between them lies in the gradient calculation and parameter updates during the backpropagation process. Regardless of the task type, sharing the same base model can significantly reduce memory requirements.\n\nAccording to the description in the Experiments section, we measure the efficiency of these two tasks as follows, and list the meanings of all experimental metrics in Appendix B:\n\n - The efficiency metric for the fine-tuning task is the throughput of fine-tuning and evaluation, i.e., the number of tokens fine-tuned and evaluated per unit of time, which is equivalent to the time required to fine-tune with the same data and configuration.\n\n - The efficiency metric for the inference task is inference throughput, which is the number of tokens inferred per unit of time. Additionally, serving metrics targets are used to define the availability of the service system, so the secondary metric is the service level objective attainment (rate). Appendix C clarifies the service metric settings, including the maximum request wait time, maximum decoding latency per token, and average decoding latency. By calculating whether each request meets these strict serving metrics, our experimental results have sufficient practical significance.\n\n# Latency Impact Caused by Virtualized Module\n\nDue to the objective limitations on the number of LoRAs that can be loaded on each GPU, in practical use, we can pre-create a sufficient number of virtual base models for the maximum number of LoRAs that can be accommodated, along with an equivalent number of empty LoRAs, and perform data copying when actually loading LoRAs. Even if these virtual modules need to be dynamically created, one or two pre-prepared virtual models can always be reserved. In this case, the latency introduced by Virtualized Module can be resolved during the framework startup phase, so this latency does not affect performance and efficiency in latency-critical environments.\n\n# Extension to other PEFT methods\n\nAs stated in the paper and the first section \"Background, Scenario Definition, and Key Experimental Metrics\" in our response, Loquetier is a framework for unified fine-tuning and inference task design in multi-LoRA scenarios, rather than overall optimization of various PEFT methods. We use a Virtualized Module mechanism to support the loading of other PEFT models. We also support optimization work for other PEFT methods. For example, if a study has designed an efficient operator for a certain method, we only need to introduce this operator into Loquetier to achieve optimization of this method.\n\n# Models and Datasets\n\nWe did not propose improvements to the effectiveness for fine-tuning and inference methods, but rather optimizations to the computational processes for multi-LoRA fine-tuning and inference tasks. So there is no need for validating the effectiveness of fine-tuning on vast downstream tasks with F-1, rouge-L, etc. The current dataset is sufficient to evaluate all the experimental metrics mentioned earlier.\n\nIn this paper, we made improvements at the kernel level. The industry typically evaluates computational quality through the following two parts of validation, and based on our validation results, Loquetier does not affect the quality of computational results.\n\n  1. Verifying whether kernel computational outputs align with standard results: We performed 10,000 kernel operations using random inputs, and the results indicate that no errors were generated in kernel computational results at float16 precision.\n  2. Verifying whether the loss converges normally during fine-tuning: Based on data generated during the experiment, the loss converges consistently with the PEFT baseline method.\n\nRegarding the dataset used for inference tasks, we applied the same data input fairly to all test baselines. The data used reflects the highest throughput of our framework, and this result is not influenced by the dataset, verifying that the inference task has achieved its objectives. Additionally, we designed a unified task for fine-tuning and inference with dynamic input scenarios within hardware constraints (Section 4.2 - Unified Fine-tuning and Inference). Experimental results show that Loquetier can handle any dynamic input changes under hardware limitations.\n\n# Framework Design and Motivation\n\nAs mentioned in the paper and the first section \"Background, Scenario Definition, and Key Experimental Metrics\" in the response, there is a lack of work on multi-LoRA fine-tuning and inference in the context we describe. Therefore, we designed a unified framework for fine-tuning and inference to significantly improve the execution efficiency of these tasks and hardware utilization in scenarios with limited hardware resources, while maintaining good compatibility with other PEFT methods and related optimization work.\n\nLoquetier does not modify user input or output, which is consistent with other related work. Specifically, each request only needs to provide its text content or token sequence, along with the specified LoRA (LoRA ID). The framework will return the generated text content or token sequence sequentially. For fine-tuning requests, we save the obtained model in the specified file path, and immediate fine-tuning results can be obtained by accessing the originally created LoRA model. The data flow is illustrated in Figure 1 (architecture diagram) of the paper. Due to NeurIPS restrictions on images and external links in replies, we cannot reproduce the image here; please refer to the submitted paper.\n\n# Scenario and Contribution\n\nAs noted in the paper and the first section \"Background, Scenario Definition, and Key Experimental Metrics\" in our response, our scenario involves fine-tuning and inference tasks using multiple LoRA models under limited hardware resource constraints, with the goal of improving the efficiency of these tasks in such scenarios.\n\nOur memory optimization is achieved through Virtualized Module, a mechanism that enables different LoRA or PEFT models to share the same base model. By loading only one base model, we significantly reduce GPU memory usage in multi-LoRA scenarios. The specific virtualized model architecture is detailed in Figure 1 (architecture diagram) of the paper. Our computational optimization synchronizes computations by loading different data into a single kernel call for input data and various LoRA weights, thereby efficiently utilizing hardware resources to accelerate the computational process.\n\nOur work includes over 1,000 lines of CUDA kernel and its related code, nearly 4,000 (3,893) lines of code supporting kernel design, fine-tuning, inference workflows, and model adaptation, as well as over 1,000 lines of code supporting PyTorch native training. We also made modifications to the kernel calls in FlashInfer to support Llama 3's RoPE and other settings.\n\n# Additional Supplement\n\nPlease allow us to update the performance of FlexLLM in multiple LoRA inference scenarios. We conducted a smaller-scale test, and the results showed that FlexLLM ran for several hours while repeatedly loading LoRA in a loop for a test that should not have lasted more than 5 minutes—it indeed did not get stuck in an infinite loop. We have not studied its underlying implementation in detail, so we will not speculate on the cause of this phenomenon."}]

Author rebuttal:

Meta-review: The paper presents Loquetier, a virtualized multi-LoRA framework that unifies fine-tuning and serving. It features virtualized modules that isolate model modifications, along with an optimized computation flow and kernel design that merges training and inference paths. The authors conduct an empirical study using three workloads and demonstrate that Loquetier consistently outperforms existing baselines.

During the rebuttal, the authors provide a comprehensive response, including new experiments, to address the reviewers' concerns. The reviewers also acknowledged the response. The AC believes this paper makes a valuable contribution to the field, so recommends acceptance.
Decision: Accept (poster)