On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
Authors: Rongguang Ye
Ming Tang
Edith C. H. Ngai
Venue: ICLR 2026 Poster (2026)
Keywords: Configuration-aware optimization
Pareto-base configuration search
Quantization
Fine-tuning
Primary area: foundation or frontier models, including LLMs

Abstract: As increasingly large pre-trained models are released, deploying them on edge devices for privacy-preserving applications requires effective compression. Recent works combine quantization with the fine-tuning of high-precision LoRA adapters, which can substantially reduce model size while mitigating the accuracy loss from quantization. However, edge devices have inherently heterogeneous capabilities, while performing configuration-wise fine-tuning for every quantization setting is computationally prohibitive. In this paper, we propose CoA-LoRA, a method that dynamically adjusts the LoRA adapter to arbitrary quantization configurations (i.e., the per-layer bit-width choices of a pre-trained model) without requiring repeated fine-tuning. This is accomplished via a configuration-aware model that maps each configuration to its low-rank adjustments. The effectiveness of this model critically depends on the training configuration set, a collection of configurations chosen to cover different total bit-width budgets. However, constructing a high-quality configuration set is non-trivial. We therefore design a Pareto-based configuration search that iteratively optimizes the training configuration set, yielding more precise low-rank adjustments. Our experiments demonstrate that, unlike the state-of-the-art methods that require fine-tuning a separate LoRA adapter for each configuration, CoA-LoRA incurs no additional time cost while achieving comparable or even superior performance to those methods.

Published as a conferencce paper at ICLR 2026.
ON-THE-FLY ADAPTATION TO QUANTIZATION: CONFIGURATION- AWARE LORA FOR EFFICIENT FINE- TUNING OF QUANTIZED LLMS
Rongguang Ye! Ming Tang Edith C. H. Ngai
'Southern University of Science and Technology
The University of Hong Kong
ABSTRACT
As increasingly large pre-trained models are released, deploying them on edge devices for privacy-preserving applications requires effective compression. Re- cent works combine quantization with the fine-tuning of high-precision LoRA adapters, which can substantially reduce model size while mitigating the accuracy loss from quantization. However, edge devices have inherently heterogeneous pabilities, while performing configuration wise fine-tuning for every quantization setting is computationally prohibitive. In this paper, we propose CoA-LoRA, method that dynamically adjusts the LoRA adapter to arbitrary quantization con- figurations (i.e., the per-layer bit-width choices of pre-trained model) without requiring repeated fine-tuning. This is accomplished via a configuration-awe model that maps each configuration its low-rank adjustments. The effective- ness of this model critically depends on the training configuration set, collection configurations chosen to cover different total bit-width budgets. However, con- structing high-quality configuration set is non-trivial. We therefore design a Pareto-based configuration search that iteratively optimizes the training configu- ration set, yielding more precise low-rank adjustments. Our experiments demon- strate that, unlike the state-of-the-art methods that require fine-tuning a separate LoRA adapter for each configuration, CoA-LoRA incurs additional time cost while achieving comparable or even superior performance to those methods.
INTRODUCTION
With the rapid growth of parameter scale, large language models (LLMs) have demonstrated in- creasingly strong capabilities across a wide range of applications (Zhang et al., 2022; Touvron et al., 2023; Liu et al., 2024a; Workshop et al., 2022; Ye & Tang, 2025), Nevertheless, their model size makes deployment on edge devices impractical. A common solution is to first quantize the pre- trained model and then fine-tune it using Low-Rank Adaptation (LoRA) (Hu et al., 2022), allowing model compression while maintaining performance (Dettmers et al., 2023; Guo et al., 2024; Xu al., 2024). A key factor in this process is the quantization configuration, which specifies the bit-widths of layers in the model and thereby determines the overall compression level. However, existing methods are typically designed for single fixed configuration and thus cannot general- ize across diverse configuration settings. In practice, this limitation becomes critical because edge devices, ranging from smartphones to laptops, require support for diverse compression levels. Con- sequently, a single LoRA adapter is insufficient to deliver consistent performance across all possible configurations Although fine-tuning a separate LoRA adapter for each configuration is possible, this can be extremely time-consuming.
In this work, we aim to address the challenge of adjusting LoRA adapters to arbitrary quantiza- tion configurations in an efficient manner. This is achieved through configuration-wware model that learns to associate each configuration with its corresponding adapter adjustments. This goal raises two key challenges. First, directly mapping from each configuration to the full set of LoRA parameters is intractable, as the output space of the mapping is prohibitively large. Second, the
"Corresooniing author.

---

Published as a conferencce paper at ICLR 2026.
effectiveness of this model hinges the quality of the training configuration set. Yet, constructing such a configuration set is nontrivial: uniform bit-width assignments typically fail to account for the heterogeneous sensitivity of different layers, resulting in suboptimal results. To address these chal- lenges, we propose CoA-LoRA, a configuration-aware model that integrates two key techniques: (i) lightweight mapping from each layer's quantization configuration to compact low-rank ad- justment, which effectively reduces the dimensionality of the mapping output and enables parallel adjustment across layers; and (ii) a configuration
search based on Pareto-based Gaussian pro- cess (Williams & Rasmussen, 1995), which refines the training configuration set and guides more accurate LoRA adjustment. By integrating these two techniques, CoA-LoRA produces high-quality LoRA adapters that generalize well across heterogeneous devices, eliminating the need for repeated fine-tuning when encountering new configurations.
In summary, we make the following contributions:
We introduce CoA-LoRA, a configuration-aware method that generates lightweight low- rank adjustments to LoRA adapters based on layer information and quantization settings, enabling LoRA adapters to be adjusted to any configuration without separate fine-tuning.
We propose a quantization configuration search technique that identifies high-performing configurations across a wide range of bit-widths and leverages them to guide the optimiza- tion of the configuration-awwr model.
Empirical experiments show that CoA-LoRA efficiently serves all quantization configura- tions with a single trained configuration-awe aware model, avoids the need for separate fine- tuning of each configuration, and achieves superior performance most datasets with accuracy gains ranging from 1.74% to 8.89% over state-of-the-arr methods.
2 RELATED WORK
2.1 LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Low-Rank Adaptation (LoRA) (Hu al., 2022) is a widely used parameter-efficient fine-tuning method. It operates by approximating weight updates with low-rank matrices during fine-tuning, which are then merged back into the pretrained weights for inference. Owing its low inference latency and strong adaptation capability, LoRA has become a widely adopted approach for fine- tuning L.MSs (Kopiczko al., 2023; Zhang al., 2024; Qin et al., 2024). Several variants of LoRA have been proposed to improve its flcxibility and performance. For instance, AdaLoRA (Zhang et al., 2023) incorporates an importance- aware rank allocation strategy assign ranks according to the significance of each layer, while DoRA (Liu et al., 2024b) further decomposes the low-rank matrices into magnitude and direction components to approximate palameeerrttr fine-tuning. Beyond these variants, LoRA has also been explored in federated learning to reduce communication overhead between clients and servers (Singhal et al., 2025; Busacca et al., 2024), and more recently applied to restore the performance of quantized LLMs (Qin et al., 2024; Dettmers et al., 2023). Despite these successes, LoRA adapters remain sensitive in practical deployment, especially when models are quantized. This challenge highlights the need for approaches that can retain LoRA's effectiveness across diverse quantization settings.
2.2 WEIGHT QUANTIZATION OF LARGE LANGUAGE MODELS
Weight quantization, which quantizes the model weights while keeping activations in full precision, is a widely used and practical approach for deploying LLMs under memory and compute constraints. Standard cound-to-ncaress (RTN) quantization (Yao et al., 2021) is one of the most widely used tech- niques, where weights w are quantized as w R s x clamp 2e 22 10-1,20- 2 1) with scaling max(u)
factor s = 2 T_T and bit-width b. NormalFloat (NF) (Dettmers et al., 2023) is a 4-bit floating- point format that employs non-uniform quantization via lookup table to approximate the original floating-point values. Despite their advantages, RTN may suffer noticeable performance drop at extremely low bit-widths (Shao et al., 2024); NF initially employed uniform granularity, assigning the same bitwidth to every layer, which restricted fine- mee-graieed layer-wise adjustments and limited potential performance gains (Zhou et al., 2025). To alleviate this, several advanced low-bit quantiza- tion methods have been proposed. For example, GPTQ (Frantar et al., 2023) leverages second-order
2

---

Published as a conferencce paper at ICLR 2026.
information of the quantization error for precise 3- 4-bit weight quantization, while AWQ (Lin et al.,, 2024) and OWQ (Lee et al., 2024) allocate quantization precision based on weight importance. Nevertheless, even with these techniques, quantized LLMs typically experience performance drops compared to pretrained L.Mss. QLoRA (Dettmers et al., 2023) addresses this issue by applying LoRA technique fine-tune quantized weights, partially restoring performance. Building on this idea, LQ-LoRA (Guo et al., 2024) further restores performance by initializing the LoRA adapter using iterative singular value decomposition (SVD), which provides a better starting point for fine- tuning. However, in realistic scenarios involving large-scun scale and heterogeneous devices, performing LoRA fine-tuning for every quantization configuration becomes computationally infeasible. This limitation motivates approaches that can adjust single LoRA adapter across multiple quantization settings, enabling efficient deployment quantized LLMS without repeated fine-tuning.
2.3 LORA ADAPTER GENERATION
Early works such P-diff families (Wang al., 2024) and G.pt (Peebles et al., 2022) pioneered the use of diffusion models to generate network parameters. More recently, CONDP-DIFF (Jin et al., 2024) applies diffusion models to generate LoRA parameters in multi-task learning scenarios. While effective for coarse- gaarse-graied tasks, it struggles with fine-grained tasks. To address this limitation, ICM-LoRA (Shao et al., 2025) leverages in-context learning, and LoRA-Gen (Xiao et al., 2025) improves both efficiency and performance using Mixture-of Mixture-of-Experts (MoE) (Lepikhin et al., 2021). However, these methods generally rely on expert LoRA and well-trained LoRA parameters, which are costly to obtain for quantized LLMs. In contrast, our work aims to adjust LoRA adapters in lightweight and effective manner, restoring model performance under different quantization settings.
3 MOTIVATION
Before introducing our method, we first highlight two key challenges in fine-tuning quantized LLMs: (i) individually training LoRA adapters across multiple bit-widths is highly timecconu,nngg and (ii) a single shared LoRA adapter cannot achieve consistently optimal performance across
different quantization configurations.
To illustrate these challenges, Fig.
Cumulative Time (bours)
Lour
compares QLoRA and Shared-LoRA on the SST-2 task (Wang et al., 2019),
Accuracy (%)
Q-LoRRA
?
using RoBERTa-Large (Liu al., 2019). QLoRA fine-tunes dedi- cated adapter for each quantization setting (2.5-4 bits), yielding strong accuracy but requiring fine-tuning ef- fort that grows linearly with the num- ber of quantized models. Shared- LoRA eliminates this cost by training a shared adapter across all settings, but the gain in efficiency comes with a large drop in accuracy.
85-
Shared-LoRRA Q-LoRA
80
Gap
Average Bit
Average Bit
Figure 1: Accuracy gap (left) and performance compari- son of cumulative fine-tuning time (right) the SST-2 task from the GLUE benchmark using RoBERTa-Larg model.
These observations naturally motivate the following question: can we design method that effi- ciently adjust the LoRA adapter across different quantization settings without repeated fine-tuning?
4 COA-LORA: EFFICIENT FINE-TUNING OF QUANTIZED LLMS
4.1 OVERVIEW OF COA-LORA
As shown in Fig. 2, CoA-LoRA consists of two complementary techniques: (I) Configuration- Aware LoRA Adjustment, which trains model 0 to generate configuration-specific adjustments to LoRA matrices. A key challenge is that learning mapping into the entire LoRA adapter space for each configuration significantly increases both the learning burden and the size of 0. To address this, we generate LoRA adjustments for each layer in parallel, which substantially reduces both model size and training effort. (II) Quantization Configuration Search and Filtering, which identifies
3

---

Published as a conferencce paper at ICLR 2026.
Configuration-Awrre LoRA Adjustment
Configutation
Adjustment Space
Evaluate
Objective Space
(I) Configuration-Aware LoRA
Embedding Space
Adjustment
Accuracy (1)
n)
(
L1,4 E RR
Q (4)
(9)
60%
Q2
T+Ua(Q.
E R
Bits (.)
2,5
5
III) Quantization Configuration Search and Filtering
Le R
n3)
Figure 3: Illustration of configuration-aware LoRA adapters with parallel ad- justment. The configuration- aware model O generates ad- justment matrices I+ I+U (C,)) from the quantization config- uration C in parallel, where I denotes the identity matrix.
(4)
Q
65%
Bits (
Figure 2: CoA-LoRA workflow: optimizing both the quantization
configurations and the configuration-awware model to achieve max-
imum accuracy any given bit-width.
high-quality and diverse configurations used for training O. The main challenge lies in evaluating the quality of configuration and optimizing over the high- high-dimensional discrete confi configuration space. We tackle this via a Pareto-based Gaussian process combined with finite-difference gradient approximation to efficiently optimize the training configuration set.
4.2 CONFIGURATION AWARE LORA ADJUSTMENT
Given a pre-trained weight matrix W E _d and quantization configuration C, quantization produces quantized matrix We E Rdxx, To restore the quantization error, LoRA introduces two low-rank matrices, L1 E Rd and Lg E Rr where < minfd, n). Given a fine-tuning dataset D and a task-fpec loss L, the optimization problem can be expressed as
arg L(W (We + L,L L,L,
(1)
Quantization Configuration Representation. We adopt the NormalFloat (NF) quantization scheme (Dettmers al., 2023) the quantization method to determine quantization configuration.
We choose NormalFloat (NF) primarily because its non- uniform quantization allows better preservation of small- magnitude weights compared to uniform integer-based quantization at the same bit-width, which critical for main- laining performance in low-bit quantization.
Table 1: Layer-level configuration
space of quantization parameters.
Parameter Configuration Space
by (2,3,4,8)
For each layer of LLM, the quantization configuration of NF contains five key parameters: (1) the bit size for the initial matrix by, (2) the first-level bucket size Bu, (3) the bit size for quantizing block-wise absmax values b,, (4) the second-level bucket size Bi, and (5) the final bit size for casting the absmax vector by. A layer-level quantiza- tion configuration is thus given by the parameter set e;
b1 (2,3,4,8)
b2
{bf16, 6,, fp16, Ep32] {16, (16,32,0 64)
Bo
Bi
{16, 64, 256)
[bo.i, b1, b2,5, Bu.s, Br.s] for layer i. Concatenating the configurations of all quantized layers yields a model-level quantization configuration (ci...., CN), where is the number of layers to which
LoRA is applied.
Under the corresponding layer-level configuration c, layer contains a pair low-rank matrices Ly L2 that need be adjusted. According the layer-level configuration space of parameters
4

---

Published as a conferencce paper at ICLR 2026.
listed in Table I, the overall search space is 3--33-3337 3-3),, which grows exponentially with N. To make this space tractable for learning, we embed each layer-level configuration e; into learned vector 2;- In addition, we further embed the layer name (c.g., fully connected layers attention projection layers) and the block index into vectors m and b, respectively. 'The final embedding for layer under the j-th model-level configuration is given by Q, [z] m 6,"1. Based on these layer embeddings, the j-th model-level configuration continuous embedding is defined as their concatenation Q = =Q) .....
(1)) In the following, we denote the corresponding discrete
quantization configuration CU),
Confiuuration-Awre Model. Mapping a configuration embedding to the full set of LoRA pa- rameters L HEEN] would be prohibitively high-dimensional. To overcome this problem, leverage the observation that most of the adaptation signal is concentrated in L2, (Zhu et al., 2024; Hao et al., 2024), finding that also observed under quantized fine-tuning (see Appendix C.I).
Motivated by this observation, we introduce configuration model 0 R maps a layer-level configuration vector Q; to a lightweight adjustment matrix Uo Qq;) E R"x". As shown in Fig. 3, each layer's low-rank matrix L2,: is reparameterized
Rrxr, which
(I + U (2)
where I is the identity matrix. Given a dataset DD let Wc denote the quantized pre-trained model weights under configuration C. We define the adjusted model weights using configuration-aware adjustment function:
WL InsertLoRA Wc. (L ( (C) )i-))
where InsertL.ORA()) inserts each layer's LoRA adjustment into the corresponding layer. During the generation of the LoRA adjustment, algorithm computes the residual between pretrained weights and the quantized weights under C and applies an SVD to derive (C) and !!
The configuration model is then optimized by minimizing the expected task-specific loss L over a set of configurations
[c
(WWleRA :D]
f arg min Ecec
(3)
For practical implementation, we initialize the configuration set C with 50 configurations uniformly sampled between 2.25 and 7.25 bits (see Appendix A.2 for details).
Instead of predicting all LoRA parameters at once, which is prohibitively high- high-dimensional (EN de x r + r x nz), the configuration-awar model generates an x adjustment for each layer in parallel, significantly reducing the output dimensionality and enabling efficient learning.
4.3 PARETO-BASE QUANTIZATION CONFIGURATION SEARCH AND FILTERING
Pareto-Based Surrogate Modeling. To improve the quality of the initial- ized configuration set Q for training the configuration-wware model, we it- eratively update it through Pareto- based search. Each candidate con- figuration is evaluated according to two inherently conflicting objectives: task-specific performance fi and erage bit-width f2, as higher perfor- mance typically requires higher pre- cision. We therefore formulate a bi- objective optimization problem for each configuration C:
Bit
Bit
Segment
C
fz
fz
Segment 2
(
(4)
Segment 3
(
f1
Loss
f1 Loss
Figure 4: Illustration of the Hypervolume Improvement (left) and the Segmented Pareto Front (right). In the right figure, red points indicate the Pareto-optimal configurations, blue points are the configurations preserved in the final set after segmentation, and yellow points discarded subop- timal configurations.
min f(C) = (f) (4) C
The computation fa is provided in the Appendix (Eq. 12). By identifying the Pareto Pareto-optimal optimal con- figurations-the for which other configuration improves one objective without degrading the
5

---

Published as a conferencce paper at ICLR 2026.
other-wa obtain set of configurations that is both nigh-perforning and diverse, forming the se- lected training configuration set C. Projecting these configurations onto the objective space (f1, f2) defines the Pareto front. To evaluate the quality of set of trade-off configurations, pervolume (HV) metric (Zitzler & Thiele, 1999). Given reference point r, the hypervolume Hr measures the area dominated by the Pareto front. In Fig. 4 (left), the combined green and yellow areas indicate the hypervolume formed by the five trade-off configurations with respect to T, where larger HV value indicates a better Pareto front.
the hy-
Direct gradient-basee based optimization of the configuration set C' is infeasible because computing f2 of a model-level quantization configuration in Eq. (4) involves non-differentiable quantization operations and computationally expensive forward passes. To address this challenge, we employ Bayesian optimization, which treats the model's task-specific performance fi (C) as a black-box function and uses Gaussian process (GP) to guide the search. In particular, we model fi (C) using GP:
(C) (mC,(C),(CCCC)))
(5)
where ml (C) is the mean function, k(C, C') is kernel (e.g., RBF), and C, e.
To select a configuration that maximally contributes to the hypervolume of the current configuration set, the Expected Hypervolume Improvement (EHVI):
arg max OEHVI(C) = E (OC)- [H),(f(C),C))
C
where HVI(f(C).C) H. (f(C))) - (C) measures the potential hypervolume increase contributed by C. For example, in Fig. 4 (left), the yellow area indicates the HVI of C(3),
Finite-Differenee Guided Optimization. Problem (6) is challenging to solve because the EHVI function does not admit an analytical gradient the high-dimensional discrete configuration space. Therefore, we approximate the gradient using finite differences
dOEHVI aC;
a(C+ + 5e,) a((C - 5c;) VI
(7)
28
where C, denotes the embedding of the i-th layer, e; is the unit vector along the i-th layer embed- ding, and 8 is the step size (set to in practice). Given the approximate gradients, we optimize each model-level configuration CO) E C through an iterative coordinate search. At iteration t, for the j-th configuration we select the layer-level coordinate i* (here i depends on j) with the largest gradient magnitude: 8" arg max; depnvi
The model-level configuration is then updated along
SCP
this coordinate:
δOEHVI aCk2
C(L2 CU) sign
C
where sign(-) denotes the sign function. This procedure is repeated for T' steps for each config- uration CU) E C. Although each step modifies only one coordinate, multiple coordinates can be updated across steps. After completing T steps for all configurations, the updated configu configurations are collected into new set C' We then evaluate C' obtain f(C') and merge it with the original configuration set C, forming C C'C.
Prerervt-rr Pareto Filtering. The guided search described above expands the candi- date set to C U C'. However, this merged set inevitably contains suboptimal configurations, such as those with identical bit-widths but strietly worse accuracy than others, shown as yellow points in Fig. 4 (right). Training directly on such low-quality configurations can hinder the learning of the configuration-aware model. To address this, we introduce a iiversity-preserving filtering step, which filters the configurations to maintain both Pareto optimality and wide range of bit-widths. Specifically, we divide C U C' into U consecutive segments, denoted by Cp. Within each segment u, we define the u-Pareto front
Pareto (C C. |f(C') y f(C) for all C' E C.,, C' CC),
where f(C') / f(C) means that is not dominated by C', i.c. there exists at least one objective for which (CC) is not worse than ((C').

---

Published as a conferencce paper at ICLR 2026.
Table 2: Comparison of HV, accuracy gap, and training time across four tasks. Training Times are reported in minutes (m). The best HV and accuracy gap per task are highlighted in bold. The average accuracy gap is computed relative to QLoRA, and the dash (*-") indicates the baseline itself.
QNLI
MNLI
SST-2
QQP
Method
Solution
HV
Gap
Time
HV
Gap
Time
HV
Gap
Time HV
Gap
Time
QLoRA
6
0.58
119m 0.54
208m 0.63
97m 0.54
189m
LQ-LoRA
6
0.59
+2.81%
108m 0.57 +8.13% 183m 0.64 +1.54%
91m 0.54 +0.47%
172m
Shared-LoRA
0.60
+2.90%
21m
0.57
+8.11% %
35m 0.61 -5.06%
19m 0.53 -1.18%
32m
CoA-LoRA
00 0.62
+4.34%
57m 0.59 +8.89% 91m 0.67 +1.74%
52m 0.60 +7.87% %
88m
The configuration set is updated by taking the union over all segments, i.e., C +- U ((u)
Pareto"
The overall training process of CoA-LoRA is organized as a cyclic procedure that alternates between two stages. In each epoch, we first train the configuration-awan model 0 on the current set of training configurations C, and then expand and refine C through gradient-guides search and diversity- preserving Pareto filtering.
EMPIRICAL RESULTS
5.1 EXPERIMENTAL SETTING
We evaluate CoA-LoRA in two settings: (I) fine-tuning on the C4 dataset (Dodge et al., 2021), and (II) fine-tuning on several GLUE tasks, namely QNLI, MNLI, SST-2, and QQP (Wang al., 2019). For setting (I), we use 1wwn-2.5-1.50 Qwen-2.5-3B (Qwen et al., 2025), and LLaMA-2-78 (Touvron et al., 2023), whereas for setting (II), we employ RoBgRTa-Large model (Liu al., 2019). Our implementation is available at https: /1 thub. com C
Baselines. We compare CoA-LoRA with several baselines: QLoRA (Dettmers et al., 2023), which quantizes the pretrained model before LoRA fine-tuning; LQ-LoRA (Guo et al., 2024), which quan- tizes the pretrained model and initializes LoRA with SVD; GPTOLoRA, which applies GPTQ quantization followed by LoRA fine-tuning; and Shared-LoRA. which fine-tunes multiple quantized models using a single LoRA adapter. Among them, LQ-LoRA serves as the strongest baseline due to its near-optimal initialization and one-to-one fine-tuning Our goal is for CoA-LoRA achieve performance comparable to LQ-LoRA and QLoRA, while avoiding repeated fine-tuning by efficiently adjusting LoRA adapters new quantization settings.
Evaluation. Following prior work (Guo et al., 2024; Dettmers et al., 2023), we evaluate model performance across GLUE's benchmark tasks using accuracy. For models fine-tuned on C4, we report perplexity on the validation set. For a fair comparison, all baseline methods are quantized in layer-wise manner; given average bit-width across layers, the corresponding layer-wise quanti- zation configurations are automatically determined (see Eq. (11) in the appendix). To demonstrate the effectiveness of CoA-LoRA, we present its performance over 50 configurations spanning 2.5 to 6.25 bits. For QLoRA and LQ-LoRA, due to the high computational cost of fine-tuning each new configuration requires a separate fine-tuning run run-we we only report results for 2.5, 3, 3.5, 4, and bits. We use hypervolume (HV) (Zitzler & Thiele, 1999) to evaluate the performance curves generated by each algorithm for different bit-widths. A larger HV indicates that the algorithm achieves higher performance while covering a broader range of bit-widths. Additionally, we report the total training time of each algorithm, as well as the average accuracy gap relative to QLoRA computed over four bit-widths.
Training Settings. We use a rank of 64 for low-rank adapters and a learning rate of x 10-4, 4, For LLaMA-2-7B, Qwen-2.5-1.5B. and 2.55-B..-3B, sequence length 1024 for epochs. For GLUE, we use a maximum sequence length of 128 and train for 10 epochs. The number of segmented Pareto fronts U in CoA-LoRA is set to 40.
train and evaluate C4 with maximum
Additional experiments are provided in Appendix C, including the impact of the number of segments U and zero-shot accuracy comparisons across downstream tasks, as well as other visualizations.
7

---

Published as a conferencce paper at ICLR 2026.
QNLI
MNLI
SST-2
QQP
90
Accuracy (%)
80
70-
80
70-
70
60
70
60
50
60
Average Bit
Average Bit
Average Bit
Average Bit
Full Fine-Tuning
Shared-LoRA
CoA-LoRA
LQ-LORA
QLoRA
Figure 5: Comparison of accuracy across four tasks under different bit-widths.
Qwen2.5-1.50
Qwen2.5-3B
Llama-2-77
40
20
35
Parplxtity
30
15
30
25-
10
Average Bit
Average Bit
Average Bit
Shared-LoRA
CoA-LoRA
LQ-LoRA
QLoRA
GPTQ-LoRA
Figure Performance comparison under varying bit-widths across different model sizes.
5.2 RESEARCH OBSERVATIONS AND EXPERIMENTS
OBSERVATION 1: COA-LORA ACIIVES STRONG PERFORMANCE WITHOUT REPEATED FINE-TUNING.
Table 2 demonstrates that CoA-LoRA delivers strong accuracy while being more time-efficient than one-to-one fine-tuning methods (QLoRA and LQ-LoRA). This is due to its joint optimization of quantization configurations and configuration-awarr model across multiple settings, which induces mutually reinforcing improvements across configurations rather than treating each configuration independently. Concretely, CoA-LoRA A learns the ability to adjust LoRA adapters to all considered configurations in roughly one hour, whereas QLoRA and LQ-LoRA require 20-40 minutes of fine- tuning per configuration, so their total fine-tuning time increases proportionally with the number of configurations. Shared-LoRA reduces the total fine-tuning time by training one shared set of adapters across multiple quantized settings, but this efficiency comes with degraded accuracy (for example, accuracy gaps of 5.06% on SST-2 and 1.18% on QQP). Fig. 5 shows that CoA-LoRA maintains leading performance across various average bit-widths. Taken together, these results show that CoA-LoRA attains the favorable combination (I) competitive or superior accuracy relative to state-of-the oh-tte-rtt methods, and (II) substantially lower fine-tuning time compared to one-to-one fine-tuning methods, while avoiding the pronounced accuracy instability seen in Shared-LoRA.
OBSERVATION 2: CoA-LORA MAINTAINS ITS EFFECTIVENESS WHEN APPLIED TO LLMS OF VARYING SIZES.
To assess scalability, Fig. 6 presents results three pretrained models ranging from 1.5B to 7B parameters, with = 196, 252, 224 layers, respectively. Across all sizes, we observe consistent trends: CoA-LoRA not only outperforms Shared-LoRA but also achieves accuracy comparable to or surpassing other of art methods. Notably, GPTQLoRA shows curves close to CoA-LoRA on Qwen2.5- 1.5BB and Qwen2.5-3B, but this requires one-to-one fine-tuning at each bit-width. As model size and the number of configurations grow, this cost scales linearly, whereas CoA-LoRA achieves strong results with single training process, demonstrating its scalability to larger LLMss.
8

---

Published as a conferencce paper at ICLR 2026.
QNLI
MNLI
SST-2
QQP
0.88
0.94
Accuricy
0.86
0.80
0.6
0.92
0.84
0.75
0.82
Cod-LAllA (soes)
0.4
0.90
CoA LoRA
0.70
Average Bit
Average Bit
Average Bit
Average Bii
Figure 7: Comparison of CoA-LoRA performance on training and unseen configurations.
QNLI
MNLI
SST-2
QQP
0.8
0.85
0.925
0.80
Accurncy
0.7
0.900
0.75
0.75
Cotfiig seanch
0.875
Frozen contig
0.70
Random extonsion
0.850
0.70
Average Bit
Averae Bit
Average Bit
Average Bit
Figure 8: Effect of configuration search in CoA-LoRA.
OBSERVATION 3: THE LOW-RANK MATRICES ADAFTED BY CoA-LORA EXHIBIT GENERALIZATION ACROSS UNSEEN CONFIGURATIONS.
Although Gaussian process optimization can update the quantization configuration set, CoA-LoRA is only exposed to a limited number of configurations during training. To evaluate its generalization Fig. we plot accuracy against bit-width for both seen and unseen configurations, which allows us to evaluate whether the model generalizes to configurations with similar average bit-widths but heterogeneous distributions. We observe that the two types of curves are highly aligned, particularly on QNLI, MNLI, and QQP, indicating that CoA-LoRA exhibits strong generalization to unseen configurations. On SST-2, while the two curves are not perfectly aligned, the discrepancy remains below 1 %, indicating only a minor deviation. A possible explanation is that SST-2 is a simpler task compared to the others, as it does not involve reasoning across sentence pairs. The model captures less diverse patterns, which results in slightly weaker generalization when evaluated across different configuration distributions.
ability to unseen configurations,
5.3 ABLATION AND SENSITIVITY ANALYSIS
Effect of Configuration Search. Fig. 8 illustrates the impact of applying Gaussian process-based optimization the configuration set. To enable more informative comparison, further introduce a random extension baseline, where each epoch randomly adds ten new quantization confi configurations the set. This comparison leads two key findings.
(I) Configuration search enhances the ability of the configuration-awee aware model to adjust low-rank matrices. Compared with the no-configuration- search setting, we observe substantial gains on QNLI and SST-2, reaching nearly 2%. Improvements on MNLI and QQP are smaller but remain consistent, indicating that all tasks benefit from optimiza- tion, though to varying degrees, (II) Configuration search supplies high-quality candidate config- urations that effectively strengthen the configuration-aware model. In contrast, random extension achieves performance comparable to the no-configuration-see search baseline but remains substantially below that of configuration search. This outcome suggests that increasing the configuration set with arbitrary configurations is ineffective performance gains depend on introducing high-quality candidates obtained via guided optimization (i.e., Pareto-based configuration search).

---

Published as a conferencce paper at ICLR 2026.
Table Hypervolume (HV) and accuracy gap measured ranks 32, 64, and 128 for four tasks. The best- rest-performing value is highlighted in bold. The dash ("-") indicates the baseline itself.
QNL.
QQP
Metric Method
r=32
r=
<= 128
r=
32
r-
r=
128
r=
r=
r= 128
r=
y= 64
r= 128
QLORA
0.573
0.593
0.573
0.545
0.566
0.534
0.634
0.640
0.636
0.516
0.572
0.514
LQ-LoRA
0.591
0,586
0.581
0.565
0,575
0.563
0.644
0.648
0.644
0.523
0.569
0.514
HV
Shared-LoRA
0.453
0.005
0.602
0.574
0,571
0,579
0.611
0.656
0.612
0.527
0.584
0.527
CoA-LoRA
0.619
0.629
0.623
0.590
0.577
0.591
0.669
0.674
0.668
0.596
0.598
0.602
QLoRA
LQ-LoRA
+6.22%
-3.13%
+5.23%
+5.95%
+2.73%
+12.75%
+2.84%
+0.91 %
+2.29%
+2.68%
+0.25%
-2.55%
Gap
Shared-LoRA
115.93%
0.44%
+5.69%
+8.933
-0.62%
+13,80%
66.1%%
-0.57%
4.76%
+1.31%
-0.3%%
+2.64%
CaA-LoRA
+6.99% +1.66% +6.87% +9.60%
00.3%%
+6.87%
+2.36% +0.58% +2.46%
+10.37% +0.67% %
+11.8%%
Table 4: Accuracy comparison under integer mixxed-recisiion quantization (per-layer choices: int2, int3, int4, 8)) Columns correspond to methods, and values within each cell are reported in the order of tasks: QNLI/MNLI/
M SST-2 QQP.
Avg. Bii
QLoRA
LQ-LoRA
Shured-LoRA
CoA-LLRA
3 0.8557
0.8133 / 0.8291 /0.7353 38552/0,7397 0.9208 0.7777 0,7772 / 0.6618/ 0.8199 0.7311 0.8616 0.8616/0.8099 0.8099 / 0.83589 / 0.7621
0.8828
0.8491 / 16 /1/66...88 0.7446 0.8762 / 0.7913 0.9185 70.9952 0.8762 / 0.7632 / 0.8486 / 7626
0.8841 0.88441 00.925/ 0.8223
0,8859 0.8554 00.900 00.7501
0.8761 / 0.8441 0.9369 0.80000 0,8777 / 0.7674 008333 1 7403
0.8863
0.8533 0.9346 0.8160
(0.446600.115 /0.7441 0.8744/0.844/
/.7911
0,8790 0.7704 /0.8440 0.7444
0.8558 0.9323 00.224
Performance under Different Ranks. Table 3 reports the hypervolume (HV) and accuracy gap across different ranks. CoA-LoRA consistently achieves the best HV values, indicating that it scales well with the output dimension of the configuration-awar model. In terms of accuracy gap, Shared- LoRA underperforms QLoRA in 7 out of 12 cases, whereas CoA-LoRA surpasses QLoRA in 11 out of 12 cases, Importantly. QLoRA and LQ-LoRA require separate fine-tuning run for each configuration, while CoA-LoRA reaches comparable or superior results with only a single training process. This advantage comes from both optimizing quantization configurations and training across multiple settings, producing synergistic gains.
Comparison under Integer mixed-precision quantization. CoA-LoRA consistently achieves strong performance across the four tasks, attaining the best results in 10 out of 16 task-bit combinations. Notably, it maintains com- petitive accuracy even at lower bit-widths (e.g., bit). whereas other methods such as QLoRA and LQLoRA tend to degrade under aggressive quantization. Across tasks, CoA-LoRA also exhibits more stable improvements as bit-width increases, while Shared-LoRA and QLoRA show more vari- ability between datasets. These observations suggest that CoA-LoRA's robustness and versatility, demonstrating that its advantages extend beyond floating-point settings to integer-based settings.
Quantization. Table 4 reports results for integer
6 CONCLUSION
In this work, we present CoA-LoRA, a configuration-wwar approach that enables on-the-fly adjust- ment of low-rank adapters for arbitrary quantization configurations. Experiments show that CoA- LoRA consistently outperforms state-of-the-an methods, achieving accuracy gains of 1.74% 8.89% over QLoRA across four GLUE tasks, and HV improvements of 2%-7% compared to LQ-LoRA. Importantly, CoA-LoRA allows real-time adaptation of low-rank adapters to arbitrary configurations without additional fine-tuning, while maintaining stable performance across tasks and strong gener- alization to unseen configurations. These results demonstrate CoA-LoRA's efficiency and robustness for deploying large-scale LLMs under heterogeneous device capabilities.
ACKNOWLEDGMENTS
This work was supported in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A15 15501281911 and the National Natural Science Foundation of China under Grant 62202214, 14, and in part by the UGC General Research Fund no. 17209822 and the Innovation and Technology Commission Fund no. ITS/383/23FP from Hong Kong.
10

---

Published as a conferencce paper at ICLR 2026.
REFERENCES
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Fabio Busacca, Stefano Mangione, Giovanni Neglia, Ilenia Tinnirello, Sergio Palazzo, and Francesco Restuccia. Fedlora: Iot spectrum sensing through fast and energy-efficient federated learning lora networks. 2024 IEEE 21 International Conference on Mobile Ad-Hoc and Smart Systems (MASS), pp. 295-303 IEEE, 2024.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936. 2019.
Toutanova. Boolq:
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv: 1803.05577 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 10088-100115 15, 2023.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, 2021.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations, 2024.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In International Conference on Machine Learning, pp. 77554-17571. PMLR, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Xiaolong Jin, Kai Wang, Dongwen Tang, Wangbo Zhao, Yukun Zhou, Junshu Tang, and Yang You. Conditional lora parameter generation. arXiv preprint arXiv:2408. 01415, 2024.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. Proceed- ings of the AAAI Conference Artificial Intelligence, volume 38, pp. 13355-13364, 2024.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representationss 2021.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awi aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
11

---

Published as a conferencce paper at ICLR 2026.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXfv:2412.19437, 2024a.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: deeohh-deeooppsedd low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1969. 1692, 2019.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, 2020.
William S Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. CoRR, 2022.
Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xian- glong Liu, and Michele Magno. Accurate lora-finetuning quantization of Ilms via information retention. In Forty-first International Conference on Machine Learning, 2024.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https::////. / /arxivv.... org/ abs/ / 2412.111 15115.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Yihua Shao, Minxi Yan, Yang Liu, Siyu Chen, Wenjie Chen, Xinwei Long, Ziyang Yan, Lei Li, Chenyu Zhang. Nicu Sebe, Hao Tang, Yan Wang, Hao Zhao. Mengzhu Wang, and Jingcai Guo. In- context meta lora generation. In Proceedings of the Thirty- Fhurth-Fourth International Joint Conference Artificial Intelligence, IJCAI '25, 2025.
Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1316 1336, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2rXiv:2307.09288 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402. 13144, 2024.
Christopher Williams and Carl Rasmussen. Gaussian processes for regression. Advances in neural information processing systems, 8, 1995.
12

---

Published as a conferencce paper at ICLR 2026.
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-paramete open-access multilingual language model. arXiv preprint arXtv:2211.05100, 2022.
Yicheng Xiao, Lin Song, Rui Yang, Cheng Cheng, Yixiao Ge, Xiu Li, and Ying Shan. Lora-gen: Specializing large language model via online lora generation. Forty- International Con- ference on Machine Learning, 2025.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang. Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian. Qa-lora: Quantization-wware low-rank adaptation of large language models. In The Twelfth International Conference Learning Representations, 2024.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qi- jing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875-118888 1886. PMLR, 2021.
Rongguang Ye and Ming Tang. One-for-all pruning: universal model for customized compression of large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 2591-2604. Association for Computational Linguistics, July 2025, doi: 10.18653/ v 11/2025.indiings 132. URL /nttps://aaiee http///allanthoooogyo.o..o org/2025 findings-ard acl. 132/.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang. and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6915 6919. IEEE, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
Zikai Zhou, Qizheng Zhang, Hermann Kumbong, and Kunle Olukotun. Lowra: Accurate and effi- cient lora fine-tuning of Ilms under bits. In Forty-second International Conference on Machine Learning. 2025.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez De Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzych Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. In Proceedings of the 41st International Conference on Machine Learning, pp.
2024.
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257~ 271, 1999.
13

---

Published as a conferencce paper at ICLR 2026.
THE USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) solely general-purpose assistant for language editing, including grammar correction and sentence polishing. LLMs did not contribute to research ideation, experimental design, analysis, writing of original technical content. All scientific claims, experi- ments, and analyses in this paper are solely the work the authors.
A
ALGORITHM DETAILS
A.1 CONFIGURATION-WWARE MAPPER ARCHITECTURE
The confguraation-wwrr mapper is conditional network that generates layer-specific adjustment matrices for LoRA parameters. The network consists of separate embeddings encoding layer and quantization information, followed by two-layer MLP that produces square adjustment matrix for each layer. The architecture details are summarized below:
(i) Input Embeddings. For each layer, several types of emheddings constructed: Module type: For RoBERTa-Large: query, key, value, output, intermediate, for LLaMA and Qwen2.5: Oprojr Kpraja Uprajs Oproj: Serros-@aeprre gatepraj down-proj UPproj. Quantization parameters: bo. b1, b2 (e.g., fp16, bf16, fp32), Bo. B1. Block position: the block index.
(ii) Embedding Dimension. Each embedding has dimensionality =
(iii) MLP Mapping. The concatenated embeddings form the input to a two-layer MLP with Layer- Norm and SiLU activation. The MLP has input dimension of 7 where the factor 7 corresponds to the concatenation of seven embeddings: one for the module type, five for quantization parameters, and one for the block position. The MLP has hidden dimension of 128 and an output dimension of r², which is reshaped into X r adjustment matrix. The output is reshaped into adjustment matrix.
(iv) Ouipul Adjustment. The adjustment matrix is scaled by a learnable parameter and added to the identity matrix to stabilize the transformation.
The base LoRA adapters constructed by performing SVD the difference between the full- precision and quantized weights. Further details can be found in Appendix A.2.
Adjustment matrices are generated independently for each layer, enabling on-the-fly and parallel computation for all layers. We list the trainable parameters of the configuration-awee aware mapper in Table A.1. The results show that the configuration-awan mapper accounts for less than 0.5% of the total parameters.
Table A.1: Parameter breakdown for different models: total model parameters LoRA parameters
(rank = 128) / configuration-awar mapper parameters.
Model
Parameters
Percentage
RoBgRTa-Large
414.934,577 / 60,845,6177 2,121,265
100% 14% 0.5%
Qwen2.5-1.5B
1,693,553,729/ 149,839,4429 2,121,
100% / 8% / 0.1%
Qwen2.5-3B
3,327,528,513 1,,,9,,,25 2,,121,281
100% / 7% / 0.06%
Llama-2-7B
7,060,352,577 / 321,936,,91 2,,12,,81
100% / 4% / 0.03%
A.2 GENERATION OF THE INITIAL QUANTIZATION CONFIGURATION SET
Let (W(1) HEEN denote the set of LoRA weight matrices to be adjusted. The specific layers that can be adjusted are listed in Table A.2. Inspired by (Guo al., 2024), we introduce binary matrix X E {0, N where denotes the number of quantization configuration variables. In our work, consider five variables (see Table 1).
Our goal to initialize a quantization configuration C/x with an average bit-width b. To this end, first approximate each weight matrix W via low-rank decomposition using Singular Value
14

---

Published as a conferencce paper at ICLR 2026.
Decomposition (SVD):
W Wx UEVT
(10)
Let L1 U,, and L = V,, where U,. E,, and V, correspond the first singular
components of W Wx. We then obtain the binary X by solving the following 0-1 constrained optimization problem:
min X
W (Wx + L + L21x
[2 Wxx x) S budget,
(1I)
Σ Xi = E
where f2( fxx denotes average bit of Wx. can be computed
fa(C) Ele (bo +
b
by Bg
Bo
+
B
(12)
E w
where wi represents the number of parameters layer i and C [bo, by b2, Bo, B1). Here, by
denotes the bitwidth of the precision type. If the type is fp16 or bf16, we set b2 = 16; if it
f2,2, we set b2 = 32. Intuitively, this procedure seeks binary quantization assignment X that balances quantization (via the storage constraint) and reconstruction error (via the low-rank residual), providing well-initialized configuration for subsequent LoRA fine-tuning.
We generate 50 initial quantization configurations with average bit-widths ranging from 2.25 to 7.25 in steps of 0 by solving Problem (11). is worth noting that this integer programming problem can be efficiently solved using standard solvers such Gurobi', taking only 3-5 seconds per instance.
Table A.2: Adjustable layers for different backbone models.
Model
Adjustable Layers
RoBERTa-Large
[query, key, value, output dense, intermediate dense] [q.proj, k-proj, v.proj, o.proj, gate.proj, down.proj, up-proj] [q-proj, k.proj, v-proj, e-proj, gate.proj, down.proj, up-proj] [q-proj, k-proj, v.proj, o.proj, gate.proj, down.proj. up-proj]
Qwen2.5-1.5B
Qwen2.5-3B
Llama-2-7B
A.3 QUANTIZATION CONFIGURATION SELECTION FOR A GIVEN BIT- WIDTH
After CoA-LoRA has been trained, it produces a final configuration set C, which has been iteratively refined using Gaussian process. This set is representative of configurations across wide range of bit-widths. Consequently, for any target bit-width 6, we can select quantization configuration by finding the one that closest some configuration E C, while ensuring that the overall average bit-width equals b.
Formally, this selection problem can be written as
min dist (C,))
(13)
[2(C)=
where dist(C,C) mincree ||C C'Ilp measures the distance to the nearest configuration in C.
Intuitively, problem (13) selects the representativeness of the final configuration set: it ensures that the selected configuration is both feasible for the target bit-width and similar to the training con- figurations, thereby ensuring that the configuration performs well under the specified bit-width. As shown in Fig. A. this procedure ensures that the configuratio- model produces LoRA ad- justments that generalize effectively to new configurations with nearby bit- widths.
hhtpps:/// Phttps://www... http:////wwwgrbbi.///o
15

---

Published as a conferencce paper at ICLR 2026.
QNLI
MNLI
SST-2
QQP
40000 10-0-000
9400-006-00
Actmny (16)
70
50
75
90
Averaage
Average
Average
Average
Figure A.1: Comparison of the final configuration set with arbitrary configurations across four tasks.
Algorithm 1: CoA-LoRA
Input: Initial quantization configurations C, fine-tuning dataset D, number of epochs Ti, number of finite-difference iterations T2, and calibration data Dea
for C E C do
L Evaluate fi(C) Dex) on the calibration data;
Let & = fiC Drn)) C C);
Fit Gaussian process G based on E;
Initialize C' <- 0;
for 11 to Ti do
Sample C C~o C:
# Update the configuration-awwe aware model 0;
arg ming Ecec [c :
10
for C E EC do for 12 = 1 to T22 do # Optimize each configuration;
11
C sign
daguvi 80
e
# Add the optimized configuration to C²;
15
# Obtain segmented Pareto fronts Parcto from C C'C,, for u = I,..., U;
clu) Caareto = {C E C. f(C) Y f(C') for all C' E C., C' + C);
16
# Update training configuration set;
C U C
((M)
18
A.4 TRAINING PROCEDURE
We present the procedure for optimizing the configuration--wwan model in Algorithm 1. Initially, we construct the set of quantization configurations and evaluate their loss on the calibration data. single batch from the training dataset. This gives {fi Foee A Gaussian process is then fitted based on these evaluations. During each training epoch, we jointly -waree aware model and update the training quantization configuration set.
In experiments
optimize the configuration
B EXPERIMENTAL DETAILS
For the computation of Hypervolume (HV) (Zitzler & Thiele, 1999), we first collect the performance metries for each algorithm. The second objective f2 can be either accuracy or perplexity depending on the experiment.
To ensure that smaller values indicate better performance, we normalize fa as follows:
if f2 is accuracy,
f
pnorm
fa T
100
if f2 is perplexity,
(14)
16

---

Published as a conferencce paper at ICLR 2026.
Table C.I: Reference points (used for Hypervolume calculation) and block numbers for four models.
Model
Reference Point
Block Number
RoBeRTa-Large
(1, D)
24
Qwen2.5-1.5B
(1, D)
28
Qwen2.5-3B
(1, 1)
36
Llama-2-7B
(1, 1)
32
Comparison of hypervolume (HV) and average decrease in perplexity (lower is better) relative to QLoRA across three LLMs.
Table C.2:
Qwen2.5-1.5B
Qwen2.5-3B
Llama-2-7B
Method
HV
Gap
HV
Gap
HV
Gap
QLoRA
0,432
0.473
0.593
LQ-LoRA
0.390
-1.87%
0.425
-2.74%
0.571
-3.22%
GPTQ-LoRA
0.427
+ +1.89%
0.427
+0.87%
0.448
-0,80%
Shared-LoRA
0.422
-0.88%
0.464
+0.42%
0.568
+0.30%
CoA-LORA 0.479
+2.97%
0.506
+1.4%%
0.629
+1.25%
Table C.3: Zero-shot accuracy comparison across different methods on eight tasks with an average 3-bit budget. The best result for each task and average is highlighted in bold.
Model
Method
ANLI BoolQ Winogrande RTE PIQA ARC-Easy ARC-Challenge
Average
QLoRA
27.34
66.41
55.47
59.38
70.34
50.78
28.12
51.12
LQ-LoRA
34.38
67.97
57.81
57.81 68.75
52.34
24.22
51.90
Qwen2.5-3B
Shared-LoRA
31.25
64.84
53.91
54.69
71.88
52.34
29.69
51.23
CoA-LoRA
38.28
62.50
59.38
55.47 70.31
55.47
32.03
53.35
QLoRA
39.06
75.00
71.88
60.16
73.44
65.62
42.19
61.05
LQ-LoRA
31.25
76.56
70.31
65.62 75.00
65.52
41.41
60.81
Llama-2-7B
Shared-LoRA
38.28
78.12
68.75
62.50
74.22
67.19
42.97
61.72
CoA-LoRA
38.28
78.91
67.97
62.50
76.56
67.97
41.41
61.94
where fanax denotes the maximum value of f2 across algorithms. After normalization, HV is computed based on the two objectives, with values ranging from 0 to 1, where smaller values indicate better performance both dimensions. The reference points and block numbers for each model are listed in Table C.I.
C ADDITIONAL EXPERIMENTS
C.1 SIMILARITY OF LOW-RANK MATRICES ACROSS DIFFERENT BIT-WIDTH MODELS
We evaluated the correlations between L, and L2 under different ranks and across multiple models. Figures C.1 and C.2 show the block-wise averaged similarity between L and L2 after quantizing RoBERTa-Large to 2, 3, 4, 5, and 6 bits using Eq. 11, followed by low-rank fine-tuning. We ob- serve that the shared components across different configurations are mostly captured L2, while the configuration-specffic knowledge is primarily encoded in L, This observation motivates the -warre
design of the configuration-avve aware model where the model outputs an matrix Up directly transform L2 into UgL2.
C.2 COMPARISON OF ZERO-SHOOT PERFORMANCE ON DOWNSTREAM TASKS
Table C.3 shows that CoA-LoRA achieves strong accuracy across diverse downstream tasks, in- cluding ANLL (Nie et al., 2020), BoolQ (Clark et al., 2019), Winogrande (Sakaguchi et al., 2021), RTE (Wang et al., 2019), PiQA (Bisk et al., 2020), ARC-Easy, and ARC-Challenge (Clark et al., 2018), with average gain of 1.45% the best-performing LQ-LoRA Qwen2.5-3B. Al-
17

---

Published as a conferencce paper at ICLR 2026.
key
valter
denee
inturnediaat denee
0.91
0.91
0.81
6.4
0.01
1.00
0.90
0.89
0.91
1.00
0.91
0.91
0.90
1.00
0.04
3.93
0.81
1.00
0.81
0.82
0.69
1.00
0.90
0.90
0.89
0.50
1.00
0.93
44
0.91
0.91
1.00
0.91
0.92
0.94
1.00
0.94
0.81
0.81
1.00
0.82
-44
0.90
0.00
1.00
0.90
037
11:80
0.80
0.91
01.1
0.91
1.00
0.91
0.91
01554
0.81
1.00
0.89
0.50
0.90
100
6.0
Query
E,
Key
valoe
outpur. dense
Intermediate. dense 12
03001
1.00
03.02
0.01
0.45
1.00
0.34
0.11
0.012
1.00
0.009
00.06
0.00
1.00
01.15
0.32
0.00
1.00
0003
0.01
1.00
0.07
0007
0142
0.34
3.00
0.199
00011
01.00
3.00
0:12
13:03
0.11
1.00
0.18
0.001
0.03
1000
0107
0.01
0.07
1.00
0.12
0.02
0.13
0.19
1.00
0.01
0.08
0.12
1.00
0.02
0.12
0.18
1.00
0:00
0.03
0.07
1.00
0.03
0.07
0.32
1.00
4.0
Query Ea,
kee
vollue
outpor.
intermsdiott
0.78
0.51
906
0.82
0.53 1.00
0.79
0.81
0.82
0.52
3.00
0.51
0.82
0.87
0.88
1.00
0.58
0.88
0.54
0.65
1.00 0.64 0.65
0.54
-44
0.78
0.7%
0.79
1.00 U31
0.50
0.51
0.81
3.00 0.51
0.877
0.850
0.88
3.00
0,09
0.63
0.63 0.64
1.00
0.60
0.80
1300
0.85
3.50
0.50
0.83 0.31
1.00
0.51
0.51
0.82
0.51
3:50
0.87
0.58
0.88
0.59
1.00
0.64
0.64 0.65 0.66
1.00
0.33
0.84
0.54
0.55
1.00
quary--.
output.
Internee
0.05
3.00 0.04
0.02
0.07
0:05
1.00
1.00
9:00
0:12
0.06
12
0.10
0.00
0.09
3.00
0.00
8.12
1.00
0.0
0.02
0.07 1.00
84
64
6.4
0.06
0.09
0.00
1.00 0.16
0.53
0.07
0.08
1.00 0.12
0.04
0.10
0.11
3.00
0.18
0.01
0.04
0.05
1.00
0.08
0.02
0.06
0:07
1.00
0.13
0:04
0:09
0.11 0.16 1.00
0.02
0.06 0.00 0.12 1:00
0.14
1.10
6:12 0:14 1.00
0:01
0.05
1101
0.08
1.00
0.02
0.07 0:00
0.17
quers E,
1.8
Keey
L
valoc
1,
sarput.dcose
intermedinte dease 1,
1.00
0.91
1.00
3,00
00.90
0.91
0.90
6.8
4.
4.6
-..
05
0.91
3.00
69
0.81
1.00
01.11
4.6
00:00
0.58
1.00
0..88
0.93
01:00
3,00
0.90
0.92
0.93
1.00
0.95
0.00
0.81
1.00
01
1.90
0.90
1.00
0.90
0.88
0.50
0.88
1.00
0.90
0.90
0.50
1.00
0.92
0.93
0.98
1.00
0.80
0.81
0.91
1.00
0.90
0.000
0.90
1.00
quaryy-1.
key
1,
value
-1,
iutpui.dnnee
Intermallait...
1.00
0:08
0.02
0..02
100
0.01
0.01
0.01
1.00
0.00
00
0.01
1.00
0.01
0.00
(0.00
1.00
6.01
0.01
0.01
ca
0.6
E.5
0.03
1.50
66
0.06
0.00
4.6
0.01
4.6
1.00
0.06
64
1.50
-A4
-5.00
0..05
-EA
1.00
0.02
3.00
0.16
1.00
0.01
0.05
3.12
1.00
0.01
0.09
0.18
1.50
0.00
0.02
0.06
1.00
0.01
0.66
0.32
1.00
Figure C.1: Correlations of two types of low-rank matrices after LoRA fine-tuning on RoBERTa- Large for models quantized to 2, 3, 4, 5, and 6 bits, measured on the QNLI dataset. The three panels correspond to rank 32 (top), rank 64 (middle), and rank 128 (middle). Each panel shows two rows representing the block-wise averaged similarity for L, and Lz
though fine-tuned on the C4 dataset, CoA-LoRA also maintains robust zero-shot accuracy, showing greater stability than Shared-LoRA.
C.3 IMPACT OF THE NUMBER OF SEGMENTS
Fig. C.3 compares the results under different values of U, where U = O corresponds to the case
without segment Pareto selection. We observe that applying segment Pareto selection (i.e., U 20
18

---

Published as a conferencce paper at ICLR 2026.
8:59
1.00
6.M
0.03
(041
1,00
6.72
1.24
6:7
6.71
6.00
8.20
4000
0.5
0009
9.79
6:73
6.90
0.51
6.79
6.5
1.00
0072
3:00
6:07
1.00
6.10
0.15
1.00
800
6:67
0.33
100
0.00
0:11
1.00
0.42
1000
00.00
1.00
6.10
0:23
6:32
1.00
0415
6:18
0.32
0.11
0:52
1.00
0:20
1.00
0000
0.16
0.20
0.16
1.00
0.45
8:51
0.55
1.00
1.55
1:30
6:M
6:11
9.73
0:11
8.79
8:08
0.31
1030
8.00
1.00
1.57
0444
3:00
0.71
0.59
1.00
1.00
0.70
0.57
6.25
0.10
1.73
033
044
(005
1.14
E.28
8.71
8.00
0.59
0.57
1.00
0.74
9.79
0.00
1.00
0.75
6.70
9.14
1.50
0.51
8.77
0.00
1.00
8.54
000
1.48
1.00
0.04
0.70
Apc
Fau Ag
kg
6:00
040
6.20
1.00
6:41
0.59
0.55
0.21
0.4
8.42
8.47
0.15
0.34
0000
6.50
Avcr-
Aue 1g
044
0344
0664
6.72
1.50
1.06
0447
(457
0.72
0942
0.54
0.00
0.75
0:55
9.77
1.80
6.70
1.00
0.00
0.54
8.79
1.00
8.79
8.55
049
0.64
0.55
6.72
8.00
(.4)
1p 5g
ww 1g
6:31
5000
0.39
0:21
1.00
0:43
0:41
10.00
0.42
Figure C.2: Correlations of two types of low-rank matrices after LoRA fine-tuning for models quan- tized to 2, 3, 4, 5, and 6 bits, measured the C4 dataset. The three panels correspond to Qwen-1.5B (top). Qwen-3B (middle), and Llama-2-7B (bottom). Each panel shows two rows representing the block-wise averaged similarity for L1 and Lg.
QNLI
MNLI
SST-2
QQP
95.0
87.5
80
92.5
Aceuracy
85.0
80
60
90.0
82.5
K
80.0
40
K
40
77.5
6 Avemge Bit
Average
Averge Bit
Average Bit
Figure C.3: Comparison of performance with different segment numbers K across four tasks.
or U = 40) generally outperforms the baseline of U 0, particularly at higher bit-widths. This indicates that segment Pareto selection helps maintain strong performance across a broad range of quantization levels. Moreover, we find that on QNLI and QQP, U = 40 achieves better results than U = 20, whereas on SST-2, U = 40 performs worse. A plausible explanation is that SST-2, being a relatively simple task that does not involve centence-pair reasoning, may not benefit from too many segments, while more challenging tasks tend to require finer segmentation to capture diverse decision boundaries.
19

---

Published as a conferencce paper at ICLR 2026.
QNLI
SST-2
QQP
90
Abcurcy (86)
80
Rank 32
70-
60
50
60
50-
50
50
Avrerage Bit
Avergg Bit
Avrenge Biz
Average Bit
Ascuracy (5g
Rank 64
75
70
60
Average Bit
Average
Avemge Bir
Average
90
Accunicy (%))
70
Rank 128
60
50
50
40
4
4
Average Bit
Average Bit
Avenge Bir
Average Bit
Fall Fine-Tuming
Shared-LoRA
CaA-LoRA
----
LQ-LaRA
-B-
QLaRA
Figure C.4: Comparison of results with ranks 32, 64, and 128 across four tasks.
Table C.4: Perplexity comparison under different quantization bits (2.5, 3, 3.5, 4) on LLaMA-2-13B with rank 64. ". denotes that GPTQ does officially support 2.5- and 3.5-bit quantization.
Avg. Bit
QLoRA
LQ-LoRA
GPTQ-LoRA
Shared-LoRA
CoA-LoRA
2.5
9.11
9.23
8.87
8.65
7.48
8.00
7.95
8.18
7.58
3.5
7.09
7.52
7.59
7.15
4
6.89
7.32
7.15
7.28
6.99
C.4 VISUALIZATION COMPARISON OF METHODS UNDER VARYING RANKS ON ROBERTA-LARGE
Fig. C.4 visualizes the Pareto fronts CoA-LoRA and four baselines, showing that CoA-LoRA achieves accuracy comparable to, and often surpassing, QLoRA and LQ-LoRA, which rely to-one fine-tuning after quantization.
C.5 COMPARISON UNDER LLAMA-2-13B
We evaluated CoA-LoRA LLaMA-2-13B across multiple quantization bits. Table C.4 reports the perplexity for different methods. CoA-LoRA achieves performance comparable to one-to-one full fine-tuning across most bit settings, including bow-bit configurations (2.5-3.5 bits). These results indicate that CoA-LoRA can scale to larger models while maintaining stable and efficient adaptation different quantization levels.
20

---

Published as a conferencce paper at ICLR 2026.
Table C.5: Performance of LoRA quantization methods on LLaMA-2-7B across different ranks (32, 64, 128) and bit-widths (2.5-4) 4). quantization.
denotes that GPTQ does not officially support b.5-bit and 3.5-bit
Rank
Avg.
Bit
QLoRA
LQ-LoRA
GPTQ-LoRA
Shared-LoRA
CoA-LoRA
2.5
9.48
9.94
33.26
8.85
3
8.28
8.63
8.63
8.43
8.76
32
3.5
7.92
7.59
8.01
7.56
4
7.70
7.37
7.73
7.91
7.49
2.5
11.02
21.29
8.87
8.48
3
7.67
8.28
8.62
8.01
7.91
64
3.5
7.27
8.26
7.52
7.52
4
7.06
8.09
7.72
7.27
7.23
2.5
9.46
9.79
24.93
9.21
7.64
7.86
8.66
8.78
7.77
128
3.5
7.91
7.51
7.67
7.51
7.71
7.33
7.72
7.54
7.39
QNLI
MNLI
SST-2
QQP
Configuration Set Size
49
48
47
22
46
31
45
2
5
6
9
10
2 3
6 Epoch
7
10
2
3
4
7
10
3 4
7 8 10 Epoch
Epoch
Epoch
Figure C.5: Configuration set sizes of CoA-LoRA across four tasks.
C.6 COMPARISON OF LLAMA-2-7B UNDER DIFFERENT RANKS
Table C.5 reports the performance of different methods across various ranks and bit-widths. CoA- LoRA consistently exhibits stable results across all bit settings, At rank 32, its performance remains steady, while Shared-LoRA shows extreme fluctuations bit 2.5. Similar patterns are observed at ranks 64 and 128, where CoA-LoRA avoids the large variations exhibited by LQ-LoRA and Shared- LoRA under low bit conditions. These results indicate that CoA-LoRA preserves performance under aggressive quantization and delivers reliable behavior across different Ranks.
21


Reviews:
[{"review_id": "Ig08Mp2ihc", "reviewer": "Reviewer_TGfV", "summary": "This paper tackles the issue that LLM models should be quantized multiple times if they are deployed on various devices.\nIt's because optimal quantization configurations differ according to target hardware.\nTo resolve this problem, the paper proposes CoA-LoRA that can find optimized LoRA adapters for quantization on-the-fly.\n\nThe key idea is to yield adjustment matrices that are used to reparameterize a LoRA adapter.\nBy generating an adjustment matrix to a target quantization configuration, an optimized LoRA adapter can be achieved.\n\nAdjustment matrices are generated with Configuration aware model that is trained for yielding an adjustment matrix for each layer of LLM.\nThis model gets a quantization configuration as an input and returns adjustment matrices optimized for the configuration.\n\nIn order for this method to work well, Configuration aware model should be trained well with representative configuration settings.\nThis paper searches configurations with a Pareto-based Gaussian process and finite-difference gradient approximation for training Configuration aware model.\nStarting from an initial configuration set, Pareto improved solutions against bit or loss are replaced or added to the configuration set.\n\nWith these proposals, the paper designs optimized LoRA adapters for various configuration settings.", "questions": "- In Figure 5, CoA-LoRA shows better results than full fine-tuning on QQP. The reviewer think that full fine-tuning is more likely to show better results compared to a quantized one.\nCan the authors provide additional explanation about that?\n\n- The proposed method facilitates on-the-fly adjustment of LoRA adapters. However, the previous works, such as LQ-LoRA adapts LoRA adapters to a single target setting directly.\nThus, the reviewer think that those direct adaptations are more likely to show better results on a specific target setting.\nBut CoA-LoRA outperforms other previous works even in their target settings. Can the authors provide your opinions on how this came out?\n\n- As far as the reviewer understand, adjustment matrices are multiplied by a shared LoRA adapter. If so, how could the LoRA adapter, before adjustment can be achieved?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper aims to run an LLM on various quantization configurations caused by different hardware properties with minimal performance degradation.\nDesigning LoRA adapters per configuration is too costly, while using a single LoRA adapter on various bit settings leads to performance degradation.\nThis paper accurately recognizes these problems and proposes a solution.\n\n- It is novel that distributes LoRA adapters on-the-fly by yielding adjustment matrices tailored for quantization configuration.\nThe proposed configuration search also helps train Configuration aware model to achieve better solutions.\n\n- The paper shows the effectiveness of the proposal by executing various experiments with several datasets, model sizes, and LoRA ranks, in addition to an ablation study about configuration search.", "weaknesses": "- It is not clear that the experimental settings of other methods are exactly the same per layer, or only the average bits are set to the same.\nSensitivities of layers to performance can differ, and the performance degradation can be caused by it.\nIf the experiments are executed with exactly the same configurations, please clarify it.\n\n- Figure 5 is in the main manuscript, but it is not referenced.\n\n- Q-LoRA and LQ-LoRA are costly to execute with the whole bit settings, but the reviewer think that settings over 4-bits should be executed too, to compare the performances on higher bit settings.\n\n- Recently, there have been several works that generate LoRA adapters according to conditions [1, 2]. However, the paper doesn't mention any of them. Discussing them in the final version of the paper would strengthen the paper.\n\n[1] Shao, Yihua, et al. \"In-Context Meta LoRA Generation.\" arXiv preprint arXiv:2501.17635 (2025).\n\n[2] Xiao, Yicheng, et al. \"LoRA-Gen: Specializing Large Language Model via Online LoRA Generation\" International Conference on Machine Learning. PMLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWLsiXFeTc", "reviewer": "Reviewer_2MdX", "summary": "The authors in this work propose CoA‑LoRA, a configuration‑aware approach that lets a single fine‑tuned adapter serve many quantization settings without re‑finetuning. Their approach conditions a lightweight adapter transformations on the quantization profile and integrates an Pareto configuration search to train over a diverse yet optimal configuration set. Experiments on GLUE and reasoning benchmarks show that CoA-LoRA maintains accuracy comparable to or better than per-configuration baselines while greatly reducing finetuning time and generalizing to unseen quantization settings. The paper’s claims are overall well-supported by results. The methodology is clearly described, and is easy to follow. The experimental setup tested on both GLUE and C4 and the ablations hints towards generalization of their claims. Reporting of training budgets is transparent and helps in comparing fairly. The authors could benefit from describing more detail on the mapper’s architecture and inference latency. This would help assess their proposed implementation's real-world feasibility. The paper is well‑structured and not too difficult to follow, with pseudocode in the appendix. Prior work is discussed appropriately. As an improvement, authors can consider providing more insights into mapper architecture’s detail for better reproducibility. The authors solve a timely and practical problem which if generalizable can help the broader community in efficiently finetuning quantized models. Through experiments, their design choice is motivated by an insightful observation that shared knowledge across configurations is mostly captured in L2​, while configuration-specific information resides in L1​. This is a new insight and  justifies adapting L2 through the lightweight mapping model.", "questions": "1. The L1 and L2 separation is shown only on RoBERTa. Does this pattern hold for decoder-only models like LLaMA or Qwen? Also, does this characteristic change when the rank of the LoRA adapter is increased?\n\n2. How does CoA-LoRA perform with integer quantization (INT4) instead of NormalFloat?\n\n3. Can the method scale to 14B+ models, and what computational limits arise in training or search?\n\n4. Does the configuration-aware mapping support or extend naturally to mixed-precision quantization, or is it limited to homogenous quantization of the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors address an important practical problem of fine tuning of quantized models where per-configuration is not feasible. Their proposed method learns a mapping model which eliminates repeated adapter training while reducing the performance loss due to quantization. \n\n2. Their proposition is based on an important observation of L2 capturing configuration specific information. This is a new insight and justifies adapting L2 through a lightweight transformation. If generalized well, this could be very insightful for the community for future quantized finetuning works.\n\n3. The EHVI‑driven, finite‑difference coordinate search coupled with Pareto filtering is well‑motivated for the accuracy–bit trade‑off.\nThe authors also demonstrate competitive performance across GLUE and commonsense reasoning tasks.", "weaknesses": "1. The paper has limited discussions on the architecture and parameter count of the configuration‑aware mapper ​ and the overhead of generating mapping for a new configuration. Including these discussions would clarify deployability and provide deeper insights to their method for larger audience adaptability. \n\n2. The method is evaluated solely on the NormalFloat (NF) quantization space and does not explore standard integer (INT) quantization schemes such as INT8 or INT4. Demonstrating that CoA-LoRA can generalize to widely used integer quantizers would strengthen the claim of broad applicability.\n\n3. As highlighted in prior work such as QLoRA, the performance gap between pre- and post-fine-tuning typically widens as model size increases. Since CoA-LoRA is evaluated only up to 7B parameters, it remains unclear how the method scales to larger models. Including ablations on 14B-scale or higher architectures would provide valuable insight into whether the proposed configuration-aware adaptation maintains stability and efficiency at greater capacity.\n4. The observation that L2​ captures shared knowledge across quantization configurations, while L1​ encodes configuration-specific information, is derived only from experiments on RoBERTa. It remains unclear whether this property generalizes to larger autoregressive or decoder-only architectures such as LLaMA or Qwen. A cross-model analysis of the ​ decomposition would help confirm whether the same structural separation holds beyond encoder-style models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6VHlGeNnA3", "reviewer": "Reviewer_XFF9", "summary": "The authors propose a way to directly predict LoRA-weights suitable for a given quantization configuration of the base-model. \nThe method solves a bi-level optimization problem. A mapping from the space of possible base-model quantization mappings to LoRA-weights is learned. In order to learn this mapping, a suitable training-set is tuned at the same time as the mapping. \nThe resulting optimized training set along with the mapping makes it possible to find a quantization configuration setting for a given average bit-width constraint (Appendix A.2) and then immediately infer the LoRA weights that aims to recover the performance drop for that configuration setting. Consequently, it is possible to obtain LoRA-weights for a specific bit-width constraint without any necessary fine-tuning.", "questions": "Figure 7: Does \"unseen\" mean the configuration has never been part of $\\mathcal{C}$ or does it mean it's not part of the final configuration set? \n\nWhat exactly is the configuration-aware model - and does it matter? You mention \"light-weight\", but how exactly do you compose [z, m, b] and U?\n\nI find a conceptual break between the fact that your search-space lies in the continuous C, whereas valid configurations are discrete and enumerable. Do you make sure that your update (8) remains within valid bounds? How precisely do you evaluate f_1(C) for a given C (EQ 4) - or equivalently: what exactly is $\\tilde{W}_C$ if C lies somewhere between the embeddings of the discrete quantization configurations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper presents an original idea, each step of the solution is well-motivated. The quality of writing is high, the paper is information-dense yet good to follow.", "weaknesses": "The main weakness I see with the paper is in its problem motivation. When does anyone require the ability to zero-shot LoRA weights for a new quantization configuration? From a research perspective, this problem fascinates me. From a practitioner's perspective, I don't know when I would ever need this. In deploying a model on edge devices, I can reasonably enumerate the set of accelerators I aim to target for a given model (or model-class) and therefore target the corresponding bit-width. On the other hand, a practitioner might care more about latency for a given bit-width of a specific layer - or have specific configuration constraints as far as hardware-level support for quantization configurations is concerned. I understand you choose NF, but I assume your method should generalize to other - practically relevant - formats. I see this as an opportunity to discuss and motivate your method in a wider context: We might not care only about storage, but also latency, under additional hardware constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IH4226wfYP", "reviewer": "Reviewer_Daov", "summary": "This paper proposes CoA-LoRA, a fast (on-the-fly) adaptation of LoRA for various backbone model quantization configurations. Compared to the baselines, which train separate LoRA adapters for each quantization configuration, the proposed method trains a single configuration-aware model $\\theta$ that generates adjustment matrices for the LoRA adapters to adapt to a given configuration. To train the configuration-aware model, the paper proposes a Pareto-based quantization configuration search, which makes the training set (configurations) Pareto-optimal in the configuration space. The experimental results show that the proposed method achieves comparable or better accuracy with a single trained model $\\theta$.", "questions": "* How is the average bitwidth measured? Does the model use the same configurations across layers?\n* How does the model performance in Figure 5 exceed full fine-tuning?\n* How is the accuracy gap measured? Since the experimental results are obtained from multiple quantization configurations, is the accuracy gap averaged?\n* In the ablation study (Fig. 8), how is the performance measured without the Gaussian process? Is it replaced with random sampling and freezing? If so, this reviewer believes random sampling should be repeated during training to ensure the training set size matches that used with the Gaussian process.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The motivation is strong and practical. The paper proposes a method that does not require fine-tuning when changing the quantization configuration. This would save significant GPU-hours.\n* The experimental results show comparable or even better accuracy compared to the baselines.\n* The idea is novel and insightful. The paper proposes a meta-learning approach with a Pareto-based search algorithm for improved quantization accuracy.", "weaknesses": "* Most experiments were conducted on the RoBERTa-Large model (0.3B), which is hard to consider an LLM. In the six experimental results presented in the evaluation section, only a single experiment was conducted on the Qwen and LLaMA models, reporting their perplexity. Therefore, as the title suggests that the method targets LLMs, there should be more focus on actual LLMs.\n* The paper is not fully reproducible. There is a lack of crucial methodological details, and no code is provided. Below is a list of information that should be included.\n\n  * The architecture of the configuration-aware model $\\theta$ is not described; the only information given is that it receives a quantization configuration and outputs an r×r matrix.\n  * Is the configuration-aware model trained layer-wise or model-wise?\n  * The NormalFloat datatype was originally introduced as a 4-bit datatype (NF4). How does the paper handle the k-bit NF type?\n  * How are layer-wise bitwidth configurations chosen? The proposed method uses layer-wise configurations, but the paper does not explain how they are selected for evaluation.\n  * A complete algorithm for using $\\theta$ during quantization and training is missing. How is the backbone model trained and quantized? If the model is trained and then quantized, it consumes more memory during fine-tuning, which is precisely the issue the baseline methods aim to solve. If the model is quantized and then trained, how is the initial quantization configuration chosen, and which configuration is used for training the backbone models and LoRA adapters on the target task?\n* This reviewer believes that some content in the paper should be revised.\n\n  * line 98: The citation for NormalFloat is incorrect (Dettmers et al., 2022 → 2023).\n  * line 99: NormalFloat does not use reduced exponent and fractional bits but employs non-uniform quantization based on a lookup table.\n  * line 101: The cited papers do not contain information about NF suffering from accuracy degradation at very low bit-widths.\n  * line 187: “NF is particularly suitable in our setting because it supports block-wise quantization at a finer granularity”: this reviewer believes NF is not the only datatype supporting block-wise quantization, as any datatype can do so. Therefore, there should be another justification for choosing NF.\n  * Across the paper: QLoRA is a more accurate name than “Q-LoRA” used in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)