CAR-LoRA: Training Compression-Aware and Robust LoRA Adapters for Evolving LLMs
Authors: Rana Shahroz
Zhen Tan
Ruichen Zhang
Hua Wei
Tianlong Chen
Charles Fleming
Venue: ICLR 2026 Poster (2026)
Keywords: Low Rank Adaptation
Edge Devices
Quantization
Compression
Efficient Fine-tuning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: The deployment of large language models (LLMs) for specialized tasks on resource-constrained edge devices like smartphones and sensors presents a significant scalability problem. To run on such hardware, these massive models must be compressed using techniques like \emph{quantization or pruning} to reduce their memory and computational footprint. Concurrently, foundational LLMs are periodically updated by their developers with new data, making their $\textit{internal parameters shift over time}$. While parameter-efficient methods like Low-Rank Adaptation (LoRA) streamline personalization by fine-tuning only a small fraction of parameters, the resulting adapters are $\textbf{brittle}$; a LoRA trained for one specific compression scheme is incompatible with another, and an adapter trained on an older base model performs poorly on an updated one. This forces a costly cycle of retraining for each unique device and every new model release. To address this, we introduce a novel framework that creates a single, universally portable adapter that is both $\textbf{\textit{(i)} compression-aware and \textit{(ii)} temporally robust}$. We achieve this by augmenting the training process with a variety of simulated compression techniques during a single run, utilizing a quantized forward pass to build resilience while maintaining a full-precision backward pass for stable gradient optimization. $\textit{This method yields a unified adapter robust to diverse compression artifacts and the subtle parameter shifts from model evolution}$. Extensive experiments on models such as $\texttt{Llama-2, Llama-3.1, Gemma-2}$, and $\texttt{Mistral}$ across reasoning benchmarks like $\textit{SQA, MATH, and GSM8K}$ demonstrate that our single adapter achieves performance comparable to specialized adapters ($\textit{e.g.}$, QLoRA) that are individually retrained for each compression scheme. Furthermore, we show this single adapter maintains its high performance when applied to future, evolved versions of the base model, eliminating the need for periodic retraining. Our work pioneers an efficient paradigm for edge AI, creating portable model patches that bridge the gap between cloud-based personalization, the diverse hardware ecosystem, and the lifecycle of evolving LLMs.

Published as a conference paper at ICLR 2026
CAR-LORA: TRAINING COMPRESSION-AWARY AND ROBUST LORA ADAPTERS FOR VOOVIIGG LLMs
Rana Muhammad Shahroz Khan'l, Zhen Tan?, Ruichen Zhang', Hua Wei", Tianlong Chen'
Charles Fleming
Unniversity of North Carolina at Chapel Hill,2Arizon State University,
CCicoo
ABSTRACT
The deployment of large language models (LLMs) for specialized tasks on cesource-constrained edge devices like smartphones and sensors presents a signif- icant scalability problem. To run on such hardware, these massive models must be compressed using techniques like quantization or pruning reduce their mem- ory and computational footprint. Concurrently, foundational LLMs are periodi- cally updated by their developers with new data, making their internal parameters shift over time. While crrameter-efficienn methods like Low-Rank Adaptation (LoRA) streamline personalization by fine- fine-tuning only a small fraction of param- eters, the resulting adapters are brittle; a LoRA trained for one specific compres- sion scheme incompatible with another, and an adapter trained on an older base model performs poorly on an updated one. This forces costly cycle of retraining for each unique device and every new model release. To address this, we introduce novel framework that creates single, universally portable adapter that is both (i) compression-wware and (ii) temporally robust. We achieve this by augmenting the training process with variety of simulated compression techniques during a single run, utilizing a quantized forward pass to build resilience while maintain- ing a full-precision backward pass for stable gradient optimization. This method yields a unified adapter robust to diverse compression artifacts and the subtle pa- rameter shifts from model evolution. Extensive experiments on models such as ama-3 1, Gemma-2, and M. st ra 1 across reasoning bench- I. ama 1.1
marks like SQA, MATH, and GSM8K demonstrate that our single adapter achieves performance comparable to specialized adapters (e.g., QLoRA) that are individ- ually retrained for each compression scheme. Furthermore, we show this single adapter maintains its high performance when applied to future, evolved versions of the base model, eliminating the need for periodic retraining. Our work pioneers efficient paradigm for edge AI, creating portable model patches that bridge the gap between cloud-based personalization, the diverse hardware ecosystem, and the lifecycle of evolving LLMs.
INTRODUCTION
Large Language Models (LLMs) such as the Llama series (Touvron et al., 2023a;b; Grattafiori et al., 2024) have achieved transformative progress in reasoning and generation (Wei et al., 2021; Min et al., 2021). While originally deployed in cloud data centers, there is growing interest in running them on edge devices (e.g., smartphones, cars. IoT sensors) to support low-latency interactions, preserve privacy, and enable personalized offline operation.
To unlock their potential, LLMs must be specialized for downstream tasks. Paramettr-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA) (Hu et al., 2022), provide this capability by inserting small, trainable adapters (Houlsby et al., 2019). LoRA enables efficient personalization in the cloud, but deployment on-device exposes two key challenges (Fig. 1):
Hardware heterogeneity. Edge devices differ widely in compute and memory capacity. Compres- sion techniques such as quantization (Dettmers et al., 2022; Dettmers & Zettlemoyer, 2023; Frantar
Work done during an internship Cisco.

---

Published as a conference paper at ICLR 2026
al., 2022; Xiao et al., 2023) and pruning (Sun et al., 2023) essential to fit models on-device, but they alter weight distributions. A LoRA adapter trained on a full- preccisinn model often misaligns under compression, forcing separate retraining for each hardware variant (e.g., INT8. FP4, pruned). This one-adapter-per-deviee paradigm negates LoRA's efficiency and creates unsustainable training and maintenance costs.
Continued Pretraihing Pretrahing
Foundation models are continu- ously updated with new pretraining improve capa- bility and safety, As shown in PortLLM (Khan et al., 2024), this "temporal drift" causes adapters fine- tuned on earlier checkpoints to degrade on newer versions, even without task changes. Maintain- ing performance thus requires repeated retraining, adding further overhead.
Model evolution.
Pretrained LLM
Lora Fine-tuning
Lora
Our
LLM
Updated LLM
Depley
Fine-turing?
Multiple Edge Devices
Cloud
ABC
Our approach. We propose CAR-LORA, a unified training framework that produces a single adapter that is both compression-aware and temporally ro- bust. During training, we stochastically apply com- pression operators (quantization, pruning, layer skipping) in the forward pass, while using a full- precision backward pass for stable gradients. This exposes the adapter to diverse perturbations, regularizing it to generalize across hardware constraints and parameter shifts. The result is single adapter that can be trained once and deployed broadly across devices and future model versions without retraining.
INT8
FP4
METH
CHQA
Pruning
Figure We address two challenges for LoRA deployment for LLMs: (1) evolviig LLMs and (2) hardware heterogeneity.
In Summary, our work makes the following contributions:
A unified training framework that integrates compression simulation, producing a single adapter robust across heterogeneous devices and evolving models.
Extensive evaluations on Llama-3 3. 1. Mistral, and Gemma-2 across reasoning bench- marks (SQA, MATH, GSM8K, ANLI, CSQA, ARC), showing parity with specialized QLo- RAs retrained per configuration.
A practical paradigm for universal adapters, reducing storage, training, and maintenance costs while enabling scalable edge AI deployment.
RELATED WORKS
A full discussion of related works is provided in Appendix B.
Compression in LLMs. Deploying LLMs on edge devices requires model compression via quan- tization (Dettmers al., 2022; Dettmers & Zettlemoyer, 2023) or pruning (Frantar al., 2022; Sun et al., 2023; Frantar & Alistarh, 2023). Works such as QLoRA (Dettmers et al., 2023), Ga- Lore (Zhao et al., 2024), and Q-GaLore (Zhang et al., 2024) demonstrate the feasibility of training adapters on compressed models, while WeLore (Jaiswal et al., 2024) explores non-uniform low-rank structures. However, these approaches follow Strain-for-the-te the-teeet paradigm, requiring retraining for each compression format. Our work differs by embedding the training loop, producing a universal adapter robust across compressions.
cuapression-wwarenee directly into
Parameter Efficient Fine-tuning (PEFT). PEFT methods reduce full-parameter tuning costs by inserting adapter modules (Houlsby et al., 2019; Pfeiffer al., 2020), optimizing prompts (Lester et al., 2021; He et al., 2022; Li & Liang, 2021), low-rank adaptation via LoRA (Hu et al., 2022). While LoRA enables efficient personalization, adapters degrade as base models evolve. PortLLM (Khan et al., 2024) mitigates this drift but does not address hardware heterogeneity. Our method fills this gap by unifying compression- and temporally-robus training.
3 METHODOLOGY
In this section, first establish the foundational concepts necessary to understand our approach, including the Low-Rank Adaptation (LoRA), and common compression techniques. For additional Preliminaries, please refer to Appendix C. We then present motivating toy problem that highlights
2

---

Published as a conference paper at ICLR 2026
Training Stage
Inference Stage
Traditional
Compression
Edge Devices
Orignal
Select
Lore
Quant
Lore Conpersied)
Sunning
Tasks
Figure 2: CAR-LORA framework illustration. The training stage on the left shows method, which creates single, robust LoRA adapter by iteratively fine-tuning compression techniques like pruning and quantization. In the inference stage, this single universal adapter can be deployed across a diverse ecosystem of edge devices with varying compression requirements, eliminating the need for retraining. This is juxtaposed with the traditional approach on the right, which inefficiently requires unique, separately trained adapter for each specific device and compression scheme.
base LLM subjected randomly selected
the failure of naive post-training compression. Finally, we detail our proposed Compression-Awan and Robust LoRA (CAR-LoRA) training framework, explaining how it achieves both hardware and temporal robustness, and describe the resulting inference pipeline.
3.1 PRELIMINARIES
Low-Rank Adaptation (LoRA). To adapt a pretrained LLM with weights Wn E dx to a new task, full fine-tuning would update the entire matrix. LoRA (Hu et al., 2022) proposes more efficient method by freezing W, and introducing a low-rank decomposition to represent the weight update, AW. This is achieved with two smaller matrices B E Rdxr and A B where the rank r < d, k. The forward pass is then modified as:
Wor + AW* Wor BAr
During training on downstream tasks, the loss is minimized regarding the parameters of A and B, drastically reducing the trainable parameters and making personalization computationally feasible.
QLoRA: Adapting Quantized Models. QLoRA (Dettmers et al., 2023) extended the LoRA paradigm to enable fine-tuning on top of a base model that has already been quantized to a very low bit-width, such 4-bit. This achieves significant memory savings not only during inference but also during the training process itself. The key innovation is to backpropagate gradients through the frozen, quantized base model into the full-precision LoRA adapters. Although the base model's weights are stored a compressed format (e.g.. 4-bit NormalFloat or NF4), they are dequantized on- the-fly to higher precision (e.g.. BF16) just before the forward and backward computations. This ensures that the gradient calculations are stable and accurate, allowing the LoRA adapter to learn effectively while the memory footprint of the base model remains minimal. QLoRA represents the late-off-hhe-nt for training specialized adapter for specific compressed model format.
LLM Compression Techniques. To deploy large models on resource-constraiind devices, com- pression is essential. Key techniques include:
This reduces the numerical precision of the model's weights. A weight ten- sor W is mapped to a lower-bit representation using quantization function Cquans (W, b), where b is the target bit-width. A common approach is uniform quantization, where weights are scaled and rounded: W, = reund(clip(( tor and is a zero-point.
Quantization:
- 2b-1)) where s a scaling fac-
2 Pruning: This involves removing redundant weights from the model. This can be for- malized by applying a binary mask M E {0, 1]dx to the weight matrix: W = Cprune (W, M) W @ M. Structured pruning removes entire rows columns, mak- ing it more hardware-friendll, while unstructured pruning removes individual weights.
0 Layer Skipping: During inference, some Transformer layers are dynamically skipped to reduce latency, effectively creating a shallower network for given input.
3

---

Published as a conference paper at ICLR 2026
3.2 A TOy PROBLEM: THE BRITTLENESS OF STANDARD LORA
To motivate approach, we first illustrate the fundamental problem with the standard thFT-then- compress pipeline. Consider a standard LoRA adapter, with parameters AO = BA, trained by minimizing task-specific loss Ltask on a full-precision (BF16) base model with parameters fo:
A arg min (0 + Ad)
(2)
This adapter, A0", represents a high-precision adjustments finely tuned to the exact weight dis- tribution of 0g. Now, consider deployment scenario where this base model must be quantized to INT4. A naive approach would be to apply the compression operator Cquant to both the base model and the trained adapter. The performance of this deployed model would be evalu- ated on Lrask (C (uann((0)) + Cquane (pp)a)(A"))) This invariably leads to a significant drop in perfor- mance. The quantization introduces error and shifts the underlying weight distribution, such that 80 & Cquane (0o). The adapter 20", trained in the full-precision space, is no longer aligned with this new, compressed weight space. This theoretically shows that LoRA adapter not inherently portable across different compression schemes. To achieve optimal performance, one would need to retrain new adapter for each target compression format.
Empirical Verification. We validate this brit- tleness empirically. We first train stan- dard LoRA (BF16) adapter on the Llama-3.1- 8B (Grattafiori et al., 2024) model for 3 reason-
LORA (BF16)
QLORA [FP4]
Naive [INT4]
aLORA (INT8)
Naive (INT8)
80
68
68.
70-
39.0
860
38.9
38.5
550
38
this single trained adapter and evaluate its per- formance under two conditions: (i) Specialized Training (QLoRA): We train new adapters from scratch for INT8 and FP4 quantized mod- els, representing the performance upper-bound. (2) Naive Compression: We take the original BF16-trained LoRA adapter and apply it to an INT8 and 1NT4 quantized base model. The re- sults are summarized in Figure 3. The Naive 1nt8 and Int4 adapters show catastrophic performance collapse compared to both the original LoRA and the specialized QLoRA versions.
ing tasks: SQA, MATH, GSM8K. We then take
840
30
16.5
36.5
36
10.1
6
10
MATH
GSMBK
Dataset
Figure 3: Performance of a single BF16 LoRA adapter naively applicd to quantized models ("Naive"") versus specialized QLoRA adapters. Naive compression leads severe degradation in performance.
3.3 THE CAR-LORA FRAMEWORK
Our proposed framework, CAR-LORA (Compression-Aware and Robust LoRA), is designed to train a single, universal adapter that robust to various compression schemes and temporal model drifts. As demonstrated in PortLLM (Khan al., 2024), robustness to temporal model drift is emer- gent property of LoRA adapters, meaning no special training is required to handle it. Achieving robustness to compression, however, requires a dedicated approach. Therefore, CAR-LoRA focuses on inducing cumpression-awarenees through a novel bi-level optimization process that exposes the adapter to a wide range of model perturbations during single training run.
-onpresioon-ware Bi-Level Optimization. The core of our method is a bi-level optimization loop that trains the LoRA adapter generalizable and robust features. Instead of training on a static base model, we augment the training objective. Let quantization bit-widths, pruning masks). We aim to learn a single adapter A0" that minimizes the expected task loss over distribution p(C) of these operators:
distribution of compressed model states, forcing it to learn
{C1, C2...... ..... Ck be of distinct compression operators (e.g., different
~ arg min EEcxp() (Cras A0]]
(3)
20
We structure this as a bi-level loop:
Outer Loop (Compression Sampling): In each iteration of the outer loop, we randomly sample a compression function Cý E C, for instance, from a uniform distribution over the available compression techniques.
Inner Loop (Adapter Fine-tuning): For selected operator Cj. we perform several steps of standard fine-tun tuning for the LoRA adapter AO the task data. The base model weights du are first compressed to o = ((b) and are then frozen. The task loss is backpropagated only to update the adapter parameters A and B.
4

---

Published as a conference paper at ICLR 2026
A crucial element for stable training is handling the non-differentiable nature of compression oper- ators like quantization. We use a cumpression-forwarre full precision- backward approach. The forward pass uses the perturbed (compressed) weights 8i to compute the loss, which forces the adapter to learn robustly. During the hackward pass, however, we compute the gradients for up- dating the adapter matrices A and B using straight-through estimator (STE). The gradient of the compression operator is treated as the identity, i.e.. (C,(0o) = I. This provides a stable and in- 80g
formative learning signal, preventing vanishing gradients while still ensuring the adapter learns to compensate for compression artifacts.
Handing Structured Pruning with LORA. When the sampled compression is structured prun- ing, which removes entire rows columns from weight matrices, must ensure the LoRA adapter remains dimensionally compatible. We focus exclusively on structured pruning as it is more hardware-friie wwwearrrffiellyyy and common in practice. If a row column is pruned from base weight matrix Wo E Rddk the corresponding dimensions in the LoRA matrices B E Rdxr and A E R must also be addressed.
Let Mrow E 1] and Meat E (0, 1 1 be the diagonal masking matrices representing row and column pruning, respectively. The compressed forward pass becomes:
h (Mrow Won (Mrow B B )x:
During the forward pass, we apply the same masks to the LoRA matrices, effectively zeroing out the corresponding rows in B and columns in A. This ensures the adapter learns to operate within the pruned subspace and maintains structural alignment with the compressed base model.
Handling Layer Skipping. Layer skipping is another compression technique aimed at reducing inference latency by dynamically bypassing certain Transformer layers. When layer skipping is sampled as the compression operator Cj in our training loop, we simulate this process by randomly deactivating subset of the LoRA-adapted layers during the forward pass. To ensure compatibility and avoid shape mismatches, we only consider skipping layers where LoRA adapters are present. If layer is selected to be skipped, its corresponding LoRA adapter is also bypassed for that training step. This forces the remaining active adapters to learn to compensate for the skipped layers, build- ing resilience and ensuring that the final universal adapter can function effectively in environments where dynamic layer skipping is deployed for efficiency.
Inference Pipeline. The inference process with a trained CAR-LoRA adapter is simple and highly efficient. Once the single, universal adapter A0" has been trained, it can be deployed in training- free manner across a multitude of target models and conditions. This "universal" pipeline eliminates the need for maintaining large library of specialized adapters, streamlining deployment and dras- tically reducing maintenance overhead.
The adapter can be directly merged with any version of the base model that has been compressed using one of the techniques from the training set (and beyond), e.g., C2 (fo) (A.)).
Hardware Diversity:
Temporal Diversity: The same adapter can be applied to future, evolved versions of the base model, 0g = 0 + 6,, even if those versions are also compressed for deployment. The final model parameters are simply 0, + A0".
3.4 THEORETICAL ANALYSIS: ERROR BOUNDS FOR PORTABILITY
We provide a theoretical justification for the portability of our CAR-LORA adapter across both compression and temporal evolution. We aim to bound the performance difference between our universally trained adapter and a hypothetical oracle adapter that retrained specifically for each target configuration.
Theorem (Informal): Let A be the CAR-LORA adapter trained on distribution of compres- sions p(C) over the initial model Ou and now adapted by C& for the specific case. Let 0, Gu + d, be an evolved model at time t, and let C₁ be a specific compression operator. Let A6, be an oracle adapter retrained specifically for the compressed, evolved model C2 (9.). The difference in task loss bounded:
(O.) A0") (O.) +00ix) Edrift + Ecomp Egen
(5)
5

---

Published as a conference paper at ICLR 2026
where Cdrift is the enror from temporal drift, <comp is the error from compression mismatch, and Egen is generalization error term that is minimized by our compression-awaln training.
Proof Sketch. The full proof is provided in Appendix D. We begin by decomposing the total error using the triangle inequality and the Lipschitz continuity of the loss function. The total error can be expressed as function of the difference between the applied adapter and the oracle adapter, ||00" A This difference can be further broken down:
(6) The first term, || - represents the generalization gap of our adapter specific com- pression C2 at t = 0. Our training objective, Ec,~p(c) (6.) + C,(A))), explicitly minimizes this expected generalization error (Egen) adcross the distribution of compressions. The second term, | - x represents the shift in the optimal adapter due to both the model evolution from Oo to O2 (Earift) and the change in the compression's 's effect on the evolved model (Ccomp). As ar- gued in PortLLM (Khan et al., 2024), for small temporal drifts à, this shift is also small. Because our training regularizes the adapter to be robust to a wide range of parametric perturbations (from the compression sampling), it inherently finds smoother, more generalizable solution that is less sensitive to the small perturbations from compression and temporal drift, thus all errors are bounded.
| - 207 11 - || - =
Il + 1
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
To validate the effectiveness of our CAR-LoRA framework, we conduct a series of experiments designed to rigorously test its performance across the dual challenges of hardware heterogeneity and temporal model evolution.
Models and Datasets. Our experiments are conducted on a diverse set of modern, open-source LLMs. including
(Grattafiori et al., 2024), Gemma-2-92 (Team al., 2024b), and Mistral-7B (Jiang al., 2023). To evaluate their reasoning capabilities, we use comprehensive suite of downstream benchmarks: SQ4 (lyyer et al., 2017), MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), ANIJ (Nie et al., 2020), CSQA (Talmor et al., 2019) and ARC (Clark al., 2018). For simulating the temporal drift, we continued pretrained the base model using the following pretraining datasets, one for each time step: OpenOrca (Lian et al., 2023a), SlimOrca (Lian et al., 2023b), OpenPlatypus (Lee et al., 2023), and AlpacaGPT4 (Peng et al., 2023). More information about individual downstream tasks can be found in Appendix F.
Training Details. For temporal shift we do continued pretraining on the base models using LoRA (Hu et al., 2022) adapter with rank r = 64 and a = 128 with a learning rate of 0.0001 and 4 epochs each. For all the downstream tasks, we use LoRA adapters of rank of 8 and = 16. For each baseline, we train for 5 epochs on all tasks, and 20 epochs for CAR-LoRA. To simulate a diverse edge hardware ecosystem, we incorporate set of five distinct compression oper- ators, C, into our training and evaluation pipeline: (1) Spurious Quantization: We include multiple bit-widths: 8-bit integer (INT8), 4-bit floating-pont point (FP4), and 4-bit NormalFloat (NF4). (2) Strue- ture Pruning: We apply masks to remove entire rows and columns of weight matrices. (3) Layer Skipping (LS): We dynamically skip a random subset of Transformer layers during inference.
Baselines and Evaluation. We compare CAR-LoRA adapter against a comprehensive set of basclines to contextualize its performance: (1) Zero-Shot: The performance of the base model (either original evolved) without any adapter. (2) Standard LoRA (BF16): An adapter trained on the -rreiiiin base model. (3) Specialized QLoRA: Individual adapters trained from scratch for each specific quantization scheme (INT8, FP4, NF4), representing the performance upper bound.
4.2 MAIN RESULTS ACROSS HARDWARE HETEROGENEITY
Table presents the performance of CAR-LoRA across six reasoning benchmarks under diverse hardware-oriented compression settings, compared against standard LoRA and specialized QLoRA baselines. Several key insights emerge:
CAR-LoRA matches specialized adapters under common quantization. Across INT8, FP4, and NF4 quantization, CAR-LoRA consistently achieves performance on par with, and in some cases slightly exceeding, specialized QLoRA adapters. For instance, on MATH, CAR-LoRA [FP4] reaches 16.7%, outperforming both LoRA [BF16] (16.5%) and QLoRA [FP4] (16.1%). Similarly,

---

Published as a conference paper at ICLR 2026
SQA and GSM8K, CAR-LoRA maintains accuracy within margin (< 0.5%) of the strongest baselines. This demonstrates that a single compression-awe aware adapter can subsume the functionality of multiple retrained QLoRA variants.
CAR-LoRA preserves robustness across reasoning tasks. Unlike naive compression, which typically introduces large accuracy drops, CAR-LoRA stabilizes performance across tasks of vary- ing difficulty. On ANLI and CSQA, which demand higher-order reasoning, CAR-LoRA variants yield accuracies indistinguishable from full-precision LoRA. This indicates that our compression- aware training strategy prevents degradation in more semantically demanding benchmarks.
@ Structured pruning and layer skipping expose the limits of ro- bustness. While CAR-LoRA re- mains competitive under structured pruning. experiences more pro- nounced degradation under layer skipping (LS). Accuracy drops are especially visible on MATH and GSM8K (from 38.9% to 31.1%). These results highlight that while CAR-LoRA generalizes well across bit-width quantization, architectural perturbations that disrupt full infer- ence depth introduce sharper challenges, pointing an avenue for future optimization.
Table I: Performance comparison
reasoning bench-
marks (SQA, MATH, GSM8K, ANLI, CSQA, and ARC)
for various adaptation strategies applied to Llama-3.1-8B.
Model
SQA
MATH
GSM8K
ANLI
CSQA
ARC
Zero Shot
57.6
9.3
19.6
33.8
43.1
LoRA [BF16]
68.7
16.5
38,9
39.9
65.4
60.4
gLoRA [[NT8]
68.8
16.5
38.5
39.5
65.1
60.5
gLoRA [FP4]
68.7
16.1
38.5
39.9
6044
gLoRA [NF4]
68.0
16.4
38.6
39.5
64.9
60.4
CAR-LoRA
68.8
40.0
60.4
CAR LAR-LoRA [[NTS]
68.4
16.1
38.4
39.7
65.4
60.5
CAR-LoRA [FP4]
CAR-LoRA [NF4]
68.5
16.4
38.1
39.4
65.
60.3
CAR-LaRA [LS]
64.4
13.0
31.1
33.6
58.3
CAR-LoRA (SP)
67.6
16.0
37,5
39.5
65.1
60.5
The results confirm that CAR-LORA delivers "universal" capability across heterogencous compression schemes. With only minor trade-off's in extreme cases such LS, our universal adapter maintains parity with individually retrained QLoRAs across di- verse hardware constraints, thereby eliminating the retraining bottleneck that has historically fractured the adapter deployment pipeline
Overall Takeaway:
4.3 ROBUSTNESS ACROSS MODEL ARCHITECTURES.
To evaluate whether the benefits of CAR-LoRA generalize beyond single foundation model, we benchmarked it Mistral-7B and Gemma-2-9B across four reasoning datasets (SQA, MATH, GSM8K, ARC). Table 2 summarizes the results, revealing three key findings:
CAR-LORA consistently outperforms baseline LoRA and QLoRA across both architec- tures. For Mistral-7B, CAR-LORA [BF16] achieves 72.1% on SQA and 17.1% on MATH, sur- passing LoRA [BF16] (70.4%, 17.0%) and all QLoRA variants. Similarly, for Gemma-2-9B. CAR- LoRA [BF16] attains 74.5% on SQA and 18.5% on MATH, outperforming LoRA [BF16] (72.8%, 17.3%) and specialized QLoRA adapters. These gains, while modest absolute value, validate that compression-awwre training yields systematic improvements across distinct model families.
CAR-LoRA preserves high performance under aggressive quantization. Even when con- strained to INT8, FP4, or NF4, CAR-LORA closely matches or exceeds specialized QLORA. On GSMSK with Mistral-7B, CAR-LORA [INT8] scores 39.6%, compared QLoRA [INT8] at 38.1%.
Table 2:
Comparison of Mistral-7B - and Gemma-2-4 2-9B across four reasoning benchmarks (SQA,
MATH, GSMSK, ARC). Best results per column are bolded.
Mistral-7B
Gemma-2-9B
Model Method
SQA
MATH
GSMSK
ARC
SQA
MATH
GSMSK
ARC
Zero-Shot
55.0
10.5
20.2
51.0
58.5
11.2
23.0
53.5
LoRA [BF16]
70.4
17.0
39.6
69.5
72.8
17.3
43.0
72.4
qLoRA [INT8]
70.2
15.8
38.1
69.0
17.0
40.7
71.6
qLoRA [FP4]
70.1
37.9
18.2
40.5
71.5
qLoRA [NF4]
69.9
15.7
38.0
68.9
72.1
16.9
40.6
71.3
CAR-LORA [BF16]
72.1
17.1
39.4
69.6
74.5
18.5
42.8
72.7
CAR-LORA [IN8S]
71.5
16.6
39.6
69.5
74.0
18.0
42.5
73.0
CAR-LoRA [FP4]
71.4
16.8
39.2
69.2
73.8
18.1
42.2
72.9
CAR-LORA INF4]
71.6
16.7
39.1
69.0
73.9
18.0
42.0
72.7
CAR-LoRA (Structured Pruning]
70.8
16.2
38.7
69.6
73.2
17.6
41.5
72.1

---

Published as a conference paper at ICLR 2026
On Gemma-2-9B, CAR-LORA [FP4] reaches 42.2% GSM8K, outperforning all QLoRA base- lines. These results highlight that CAR-LoRA's single adapter can absorb diverse quantization arti- facts while maintaining competitive reasoning performance.
Structured pruning shows controlled degradation without collapse. Under structured prun- ing, CAR-LoRA retains robustness across both models. For example, Gemma-2-9B with structured pruning maintains 41,5% on GSM8K and 72.1 % on ARC, staying within 1-2% of its full-precision counterpart. This resilience contrasts with the brittle behavior of naive LoRA compression, demon- strating that CAR-LORA adapts effectively to lopology-altering perturbations.
Overall Takeaway: These findings underscore that CAR-LoRA's compression-ave training paradigm is not tied a single foundation model. Whether applied Mistral-7B Gemma-2- 9B, the universal adapter provides cross-architectury robustness, enabling near state-of-the- art performance across reasoning tasks without retraining specialized adapters for each family.
4.4 GENERALIZATION TO UNSEEN COMPRESSION.
crucial test of CAR-LoRA's universality whether a single adapter trained on a distri- bution of compression techniques can gen- eralize to unseen schemes not included dur- ing training. To evaluate this, we adopt the following experimental setting: CAR-LoRA trained with exposure to all but one com- pression operator (e.g., trained with INT8, NF4, and pruning, but not FP4). At evalu- ation time, we then test its performance on the withheld operator (FP4 or NF4), thereby simulating deployment on hardware configu- rations unseen during training. Table 3 summarizes the results across three benchmarks: SQA,
Table 3: Generalization Test for quantization methods.
Model
SQA
MATH
GSMSK
LoRA [BF16]
68.74
16.52
38.86
qLoRA [INT8]
68.80
16,46
38.45
qLoRA [FP4]
68.67
16.07
38.49
gLoRA [NF4]
68.03
16.38
38.56
Ours [Unseen FP4]
67.41
15.93
37.42
Ours [Unseen NF4]
67.63
16.07
37.58
MATH, and GSMSK.
O CAR-LORA exhibits graceful generalization to unseen compression operators. When evalu- ated on unseen FP4. CAR-LoRA achieves 67.41% on SQA and 15.93% on MATH, closely tracking the specialized QLoRA [FP4] results (68.67%. 16.07% respectively). A similar pattern emerges on unseen NF4, where CAR-LoRA attains 67.63% (SQA) and 16.07% (MATH), nearly matching LoRA INF4] (68.03%, 16.38%). Importantly, performance GSM8K remains within ~ 1 point of specialized adapters, confirming that robustness extends to reasoning heavy tasks.
@ The observed performance gap is marginal compared to specialized retraining. Although CAR-LoRA under unseen compression lags by ~ 0.5-1% absolute accuracy across datasets, the adapter still preserves the majority of its effectiveness. For instance, GSM8K, unseen FP4 (37.42%) trails QLoRA [FP4] (38.49%) by only ~ point, despite never having encountered FP4 during training. This demonstrates that compression-wware training imbues the adapter with form of "structural prior" that transfers across quantization families.
? Universality mitigates the retraining bottleneck in deployment pipelines. The ability to maintain competitive performance under unseen operators is a significant practical advantage. In real-world scenarios where hardware ecosystems evolve rapidly, requiring support for emerging compression schemes, CAR-LoRA can operate out-of-the-bo without specialized retraining. This contrasts with QLoRA, which necessitates fresh adapter for every new compression target.
4.5 TEMPORAL ROBUSTNESS
In addition to hardware heterogeneity and architectural variation, adapters deployed in practice must remain robust models evolve over time. To test this, evaluate CAR-LoRA under temporal shift setting: adapters are trained on an earlier model checkpoint (e.g.. Llama-3.1 base) and then directly applied to later, updated versions of the model without retraining. Figure 3 summarizes the results across five temporal checkpoints for INT8 quantization, reporting accuracy trends on six reasoning benchmarks.
8

---

Published as a conference paper at ICLR 2026
+
SQA
MATH
GSMBK
ANLI
CSQA
ARC
100
(%)
80
50
30
t=0
t=]
t=3
1=4 t=0
[=1
t=2
t=3
t=4
1=0
[=1
1=2
t=3
t=4
Temporal Shift
Temporal Shift
Temporal Shift
Figure 4: Accuracy across six reasoning benchmarks. (a) Left: INT8 Quantization, (b) Middle: FP4, and (c)
Right: BF16.
CAR-LoRA preserves stable performance across temporal drift. While standard LoRA shows gradual degradation as checkpoints evolve, particularly on MATH and GSM8K, CAR-LoRA maintains nearly flat performance curves. For example, on SQA, CAR-LORA hovers between 08.8-68.245 across all five checkpoints, in stark contrast to LoRA's steeper decline. This indicates that CAR-LoRA's compression- aware training not only adapts hardware perturbations but also carries resilience against temporal parameter drift in evolving L.Mss.
4.6 AMORTIZED SPACE AND TIME ANALYSIS
To fully assess deployment efficiency, we com- pare CAR-LoRA against LoRA and QLoRA in terms of trainable parameters, peak GPU mem- ory, and GPU hours (Table 4). Since LoRA and QLoRA require separate adapters per com- pression operator, their amortized cost for sup- porting five heterogeneous devices is obtained by multiplying the single-device values by five.
Table 4: Efficiency comparison between CAR-LoRA, LoRA and QLoRA on SQA with Mistral-78 as the model architecture. Metric
CAR-LoRA
LoRA
QLoRA
Params
20MM
100M
100M
Max GPU Mem (GB)
350
350
GPU Hours
170
220
In contrast, CAR-LoRA trains a single universal adapter that generalizes across all devices, so its cost remains constant.
CAR-LoRA reduces trainable parameter overhead by 5x. While LoRA and QLoRA each re- quire 20M parameters per device (100M amortized), CAR-LoRA needs only 20M parameters total, regardless of the number of devices. This substantial reduction reflects the efficiency of training a single adapter rather than replicating device-specific ones.
CAR-LORA slashes training time through amortization. Supporting five devices requires 220 GPU hours for LoRA and 211 GPU hours for QLoRA. CAR-LoRA, however, completes training in just 170 GPU hours total, offering a substantial reduction compared to QLoRA and an even larger margin over LoRA.
Total Cost of Ownership (TCO) and Break-even Analysis. We acknowledge that CAR-LORA requires higher upfront training duration per run (20 epochs) compared to standard baselines (5 epochs). However, we argue that the Total Cost of Ownership (TCO) is the relevant metric for organizations deploying edge ecosystems. In realistic scenario where organization supports device tiers and updates models periodically (T), the cost scaling shifts dramatically: Traditionally the training cost scales as O(N T' x epochs), while our's scales as O 20 epochs).
CONCLUSION
This paper addresses the dual challenges of hardware heterogeneity and temporal model evolution that make deploying personalized LLMs on edge devices impractical. We propose a novel training framework that overcomes the costly retraining for each configuration. Our solution is a single, universal adapter that is both compression-aware and temporally robust. We achieve this by aug- menting the training process with diverse set of simulated compression techniques, making the adapter inherently resilient to parameter shifts from both hardware modifications and the natural evolution of LLMs. Our "universal" adapter is validated through extensive experiments. Results show that our single adapter achieves performance on par with a suite of specialized adapters in- dividually retrained for each target hardware. Furthermore, it maintains high performance when applicd to future, evolved versions of the base model. By holistically addressing these critical chal- lenges, work pioneers scalable and efficient paradigm for deploying personalized AI,
9

---

Published as a conference paper at ICLR 2026
bridging the gap between cloud-based personalization and the dynamic realities of the edge ecosys- tem. Beyond its immediate benefits, our framework opens the door to more sustainable deployment pipelines for mndustry-scale applications, where frequent model updates and diverse hardware are the norm. We believe this approach lays the foundation for next-generation adaptive Al systems, enabling seamless personalization at scale and fostering broader accessibility of advanced lan- guage technologies across heterogeneous and conotreeconstrrined envir environments.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is generously supported by the Cisco Faculty Award
ETHICS STATEMENT
We adhere the ICLR Code of Ethics. No private, sensitive, or personally identifiable data are in- volved. Our work does not raise foresecable ethical concerns or produce harmful socictal outcomes.
REPRODUCIBILITY STATEMENT
Reproducibility is central our work. All datasets used our experiments are standard benchmarks that are publicly available. We provide full details of the training setup, model architectures, and evaluation metries in the main paper and appendix. Upon acceptance, we will release our codebase, including scripts for preprocessing, training, and evaluation, along with configuration files and doc- umentation to facilitate exact reproduction of our results. Random seeds and hyperparameters will also be included to further ensure reproducibility.
REFERENCES
Rishi Bommasani. On the opportunities and risks of foundation models. vrXiv:2/08.07258. 2021.
arXiv preprint
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
arX1v:1803.05457vl 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference Machine Learning, pp. 7750-7774. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances neural information processing systems, 35: 00318-30333 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 066:10088-1000 101 2023.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprini arXiv:2407.21788 2024.
Pan He, Yuxi Chen, Yan Wang, and Yanru Zhang. Protum: A new method for prompt tuning based on "[mask]". arXiy preprint vrXiv:2201.12/00 2022.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
11

---

Published as a conference paper at ICLR 2026
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Mohit lyyer, Wen-tau Yih, and Ming-Wa Chang. Search-based neural structured learning for se- quential question answering. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1821-1831, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18653/1//P17-1167 URL
Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407. 1239, 2024.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, URL http:::/app..emantiiccooo
abs/2310.06822. 2023. 263830494.
Rana Muhammad Shahroz Khan, Pingzhi Li, Sukwon Yun, Zhenyu Wang, Shahriar Nirjon, Chau- Wai Wong, and Tianlong Chen. Portllm: Personalizing evolving large language models with training-free free and portable model patches. arXiv preprint arXiv:2410. 2024.
Ariel N. Lee, Cole Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint
2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 2021.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. hitts://httpss huggingface.. col datasets wwwaasessseeennneeeeeeeeeeeeeeee -rwp://pennnneeeeeeeeeeeee Saaee--rraaaaaeeeeee 2023a.
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium".. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verifica- tion, 2023b. URL https: /eatttss://ttpaae http:/httttt//////
coopppn--rraaaalla wtwaaacccccoopeeeeeeeeeeeeeeeeee Oacpenn-rra//ee Saarra//smimnn /mimmrcaa.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:21 15943, 2021.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Ad- versarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.032777 2023.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07770 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1665, 2023.
12

---

Published as a conference paper at ICLR 2026
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18833/11/19-111 1421. URL https: https///aaaaattoooggooee org/N19- 1421.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08200 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé. et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv: 24xx::2408001111 18, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Lc. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0166 01652, 2021.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning. pp. 38087-38099, PMLR, 2023.
Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang- Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients. arXiv preprint NNNv:2407.00299 2024.
Jiweii Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. arXiv preprint arXiv:2403.03577, 2024.
A LLM USAGE
To enhance clarity and readability, we utilized LLMs (specifically OpenAl GPT-4o) exclusively as a language polishing tool. Its role was confined to proofreading, grammatical correction, and stylistic refinementtffnncionns analogous to those provided by traditional grammar checkers and dictionaries. This tool did not contribute to the generation of new scientific content or ideas, and its usage is consistent with standard practices for manuscript preparation.
B EXTENDED RELATED WORKS
Compression in LLMs. The deployment of LLMs on resource-constrained edge devices neces- sitates model compression. The primary strategies for this are quantization, which reduces the numerical precision of weights (Dettmers et al., 2022; Dettmers & Zettlemoyer, 2023), and prun- ing, which removes entire parameters or structures to minimize memory footprint and accelerate inference (Frantar et al., 2022; Sun et al., 2023: Frantar & Alistarh, 2023). The intersection of PEFT and compression an active area of research aimed at creating efficient, deployable models. OLoRA (Dettmers al., 2023), for instance, made a significant breakthrough by demonstrating that
13

---

Published as a conference paper at ICLR 2026
it is possible to fine-tune LoRA adapter top of 4-bit quantized base model, achieving sub- stantial memory savings. More advanced methods like GaLore (Zhao et al., 2024) and its quantized version, Q-GaLore (Zhang et al., 2024), have further pushed the boundaries of memory-efficient training by combining quantization with low-rank gradient projections. Other works like ore have explored the non-uniform low-rank properties of weight matrices inform better compression strategies (Jaiswal et al., 2024). However, all these approaches still operate under a "train-for-the- target" paradigm; an adapter trained via QLoRA for 4-bit model is incompatible with an 8-bit model or pruned model without retraining. The existing literature treats personalization for a specific compressed format as a distinct, isolated task. Our work departs from this established, se- quential approach by integrating single, universal adapter that robust to these post-training modifications from the outset and thus holistically solves the deployment challenge.
cempressiion-wwreneee directly into the training loop, creating
Parameter Efficient Fine-tuning (PEFT). To mitigate the exorbitant costs of full- ulll-parmeter fine- tuning, variety of Parameter-Efficien Fine-Tuning (PEFT) methods have been proposed. These approaches range from inserting small, trainable "adapter" modules between layers (Houlsby et al., 2019; Pfeiffer et al., 2020) to optimizing continuous prompts via prompt or prefix tuning (Lester et al., 2021; He et al., 2022; Li & Liang, 2021). Among the most successful and widely adopted is Low-Rank Adaptation (LoRA) (Hu al., 2022), which freezes the pretrained model weights and injects trainable, low-rank matrices into the Transformer layers. This drastically reduces the number of trainable parameters, making personalization more accessible. However, while LoRA elegantly solves the initial fine-tuning cost, the resulting adapters are often brittle and sensitive to shifts in the base model's parameters. The PortLLM framework (Khan al., 2024) was the first to systematically address this temporal challenge, demonstrating that adapters suffer from performance degradation as base models evolve and proposing training-free free patching mechanism maintain compatibility. While PortLLM provides crucial piece of the puzzle by addressing model evolution, it does not account for the parallel problem deploying a single adapter to a diverse and compressed hardware ecosystem, leaving critical gap in creating truly portable and durable solutions.
Large Language Models (LLMs). The field of natural language processing has been fundamen- tally reshaped by the advent of LLMs. Foundational architectures like the Llama series (Touvron et al., 2023a;b; Grattafiori et al., 2024), Mistral (Jiang et al., 2023), and Gemma (Team et al., 2024a;b) have set standards in performance, demonstrating remarkable capabilities across wide range of tasks. This success builds on the architectural evolution from earlier models like the Transformer (Vaswani al., 2017), scaling up the decoder-only paradigm unprecedented sizes. However, the immense scale of these models makes full fine-tuning computationally prohibitive for most applications, and their generic pretraining often falls short in specialized, comain-specifti con- texts (Bommasani, 2021). The widespread adoption and proven power of these models have created a pressing need for methods that can efficiently adapt them for specific use cases, particularly for de- ployment beyond centralized cloud infrastructure. This necessity directly motivates the exploration of both parameter-efficient adaptation and model compression.
C EXTENDED PRELIMINARIES
Transformer Architecture. Modern LLMs are predominantly based on the decoder-only Trans- former architecture (Vaswani et al., 2017). At its core are two main components: the melti-head self-attention mechanism and the position-wise feed-forwant network (FFN), For given input se- quence embedding X E Ra where n is the sequence length and is the model dimension, the self attention mechanism computes Query Q, Key K, and Value V projections using weight matri- ces Wo. WK, Wy E Rdxd. The output is calculated as:
QKT Von
Attention( K, K) softmax
This allows the model to weigh the importance of different tokens in the sequence. The FFN, a two-layer MI.P, is then applied to each position independently add expressive capacity.
14

---

Published as a conference paper at ICLR 2026
D PROOF OF THEOREM
Assumptions. We introduce the following assumptions, which are standard in the analysis of parameter-effiient fine-tuning under model perturbations:
1. Smoothness of the loss. The task loss L is L-Lipschitz continuous and f-smooth with respect to model parameters:
[C(0)) L(02) & L|10 02ll.
[VL(0)) (
VL(02)) - 3 0211-
Bounded temporal drift. The base model evolves smoothly over time: = Oo + de, with
||6e]| A, where A is bounded constant.
Compression stability. Each compression operator C₃ is non-expansive, i.c.,
[[Cx(01) (x 110 02ll: v0,,02.
This ensures compression does not arbitrarily amplify parameter perturbations.
Lipschitz continuity of the oracle mapping. For fixed compression operator Ck. let
Fi (0) arg min ((C) + A0)
denote the oracle adapter mapping. Assume F& is L Lipischhi:
[11(01)) - (x20)) - Lr|10 - -22l-
Theorem Let A be the CAR-LoRA adapter trained on distribution of compressions p(C) over the initial model Og. Let Δ6ú, be the oracle udapter retrained specifically for (x)))). Then the performance gap is bounded by
((.) + A0") (O.) + 207,) S
(1400 - ~00.. + || - A0
(8)
where A07.- = F& (0o) is the oracle adapter for the initial compressed model C2 (00).
Proof. By definition of A0
((.) A0;,) S ((C)(O,) + A0),
(9)
for any A6. Therefore, the gap is non-negative. Applying L-LLppshitzz continuity of L yields:
Gap = (0,) + A (G.) A07,)
(10)
< L ||A0" - 10,,
(11)
Next, introduce A6i, = (fo) the oracle adapter for the initial compressed model. By the triangle inequality,
||-00 A0,
A00
-0 x
|10%,
A0
(12)
Generalization emorr
Stahility emrrr
Generalization error. The first term measures how far A deviates from the oracle for specific compression C₁ at / = 0. Since Ad" is trained by
A
arg min ECjeM(e) (C)))(0)) +A6]]. A#
minimizes the expected loss across all compressions in p(C). implicitly regularizing for robustness. Denote this deviation as
gen (k) = | A01
Stability error. The second term measures how much the oracle shifts as the base model evolves. Since Ad Fxo) and A0 (6.).. Lipschitz continuity of Fş implies
|
A0 Lrl10 0oll T.r 11..l-
15

---

Published as a conference paper at ICLR 2026
Denote this as Caab R). This captures both temporal drift (Cami)) and compression mismatch (Ccomp) As noted in PorILLM (Khan et al., 2024), for incremental updates typical of model evolution, [|];]] is small, keeping Esah bounded.
Final bound. Substituting back, we obtain
(0,) + A0") ((C)(0)) + A0T,) S L. (Egen(k) + Estab (t, kt,
(13)
This completes the proof.
Discussion.
The assumptions are natural in practice. (i) Smoothness holds for standard loss func- tions (e.g., cross-entropy). (ii) Temporal drift is bounded since model updates are incremental. (iii) Quantization and structured pruning satisfy non-expansiveness under common norms. (iv) Lips- chitz continuity of F& reflects empirical robustness of adapters to small parameter changes. To- gether, these justify why CAR-LoRA achieves bounded performance degradation across compres- sion schemes and evolving models.
E ABLATION STUDIES
To rigorously understand the sources of CAR-LoRA's robustness and define its operational limits, we conduct three targeted ablation studies. We dissect the specific contributions of our compression operators, quantify the exact magnitude of temporal degradation in baselines, and stress-test the adapter on extreme, unseen quantization levels (2-bit and 3-bit).
E. DISSECTING THE SOURCE OF ROBUSTNESS
We performed component-wise ablation study on the SQA dataset (Mistral-7B) to isolate the contributions of our key training mechanisms. Specifically, we analyzed the impact of training with restricted subsets of compression operators compared to the full CAR-LoRA framework.
As shown Table 5, we
compare baseline LoRA (BF16 only), Ablation (Quantization only), and Ablation (Pruning + Layer Skipping only). Key Findings: Specificity of
Table 5: Component wise ablation study on Mistral-7B (SQA Dataset).
Training Configuration
Compression Operators Used
SQA (INT4)
SQA (L.S)
Baseline LoRA
None (BF16 only)
54.3%
51.2%
Ablation
Quantization Only (INT844
53.5%
Ablation B
Pruning LS Only
58.1%
62.8%
CAR-LoRA (Ours)
All (Quant + Prune L.))
68.4%
64.4%
Robustness: Ablation demonstrates that training with quantization alone improves INT4 robust- ness significantly (+13.6%) but fails to generalize to structural changes like Layer Skipping (LS). Conversely, Ablation improves structural robustness (LS +1 +11.6%) but lags significantly in quan- tization robustness. Synergistic Effect: CAR-LoRA achieves the best performance across both metrics. This demonstrates that exposing the adapter diverse compression types during training (bi-level optimization) creates synergistic effect, where the adapter learns a "structural prior" that more generalizable than any single subset of operators.
E.2 QUANTITATIVE ANALYSIS OF TEMPORAL DRIFT
While we qualitatively discussed the brittleness standard LoRA under model evolution, we pro- vide here direct quantitative comparison. We track the performance degradation of standard LoRA adapter versus CAR-LoRA over sequential model updates (checkpoints = 0 to = 4) on the SQA dataset using Alaama-3.1-83. The temporal evolution is simulated by continuing pretraining on the OpenOrca, SlimOrca, OpenPlatypus, and AlpacaGPT datasets sequentially.
As shown in Table 6, the standard LoRA adapter suffers a monotonic degradation, drop- ping by approximately 14% over four update
Table 6: Temporal degradation analysis on SQA
(Llama-3.1 3.1-8BB across 4 simulated model updates.
Checkpoint
Standard LoRA (Ace) CAR-LoRA (Acc.)
Delta
t= 0 (Basc
68.7%
68.8%
+0.1%
65.2%.
68.67
+3.4%
16
58.4%
68.3%
+9.9%
54.9%
68.25
+13.3%

---

Published as a conference paper at ICLR 2026
eycles. In contrast, CAR-LoRA remains re- markably stable, with a variance of less than 1%. This provides direct quantitative proof that conpression-aware training forces the adapter into flatter loss landscape that is inherently ro- bust to the parameter drift caused by continued pretraining.
17

---

Published as a conference paper at ICLR 2026
E.3 GENERALIZATION TO EXTREME QUANTIZATION (UNSHEN)
A critical question for "universal" adapters is the limit of their robustness. To test this, we evaluated CAR-LoRA adapter (trained on INT4/ITT//PP/Pruningg on Llama-3.1 quantized to extreme bit-widths unseen during training: INT3 and INT2 (using GPTQ-style rounding).
Analysis: INT3 Robustness: CAR-LoRA
generalizes surprisingly well to 3-bit quanti- zation, retaining ~90% of its original perfor- mance despite never seeing 3-bit weights dur- ing training. This confirms that the learned robustness is not merely overfitting to specific INT4 patterns. The INT2 Phase Transition:
Table 7: Generalization to unseen extreme quantiza-
levels (INT3 and INT2) SQA.
Conliguration
INT4 (Seen)
INT3 (Unseen)
INT2 (Unseen)
Standard LoRA (Naive)
54.3%
28.15
12.4%
CAR-LoRA (Ouns)
68.4%
61.7%
43.5%
à Accuracy
+14.1%
+33.%
+31.%
At 2-bit quantization, performance drops to 43.5%. While this avoids the total collapse seen in Stan- dard LoRA (12.4%), it represents a significant degradation. We identify this as "phase transition" point where the information loss in the base model is INT2 explicitly in the training distribution) becomes necessary.
that distinct retraining (or including
F DATASET STATISTICS
Table The datasets used our Experimental Setup.
Dataset
Domain
License
Train (Original)
Train (Filtered)
Test
SQA lyyer al. (2017)
Commonsense
MIT
2,061
1.544
229
CSQA Talmor al. (2019)
Commonsense
MIT
9,741
6,478
1.140
ARC Clark et al. (2018)
Commonsense
CC BY-SA 4.0
1,199
1,035
1.172
MATH Hendrycks al. (2021)
Math
MIT
7,500
2,511
5,000
GSM8K Cobbe
(2021)
Math
MIT
7.379
4,293
1,339
ANLI (r3) Nie al. (2020)
NLI
CC BY-NC 4.0
100,459
883
1,200
18


Reviews:
[{"review_id": "2RbB3RUg2V", "reviewer": "Reviewer_M81r", "summary": "This paper introduces CAR-LoRA (Compression-Aware and Robust LoRA) — a framework for training a single, universal LoRA adapter that remains effective across both compressed (e.g., quantized or pruned) and evolving large language models (LLMs).\nTraditional LoRA adapters must be retrained whenever a base model changes or when deploying to devices with different hardware compression formats, which is inefficient and costly. CAR-LoRA solves this by integrating simulated compression operators (quantization, pruning, layer skipping) during training. It uses a quantized forward pass and full-precision backward pass, ensuring the adapter learns to be robust against compression-induced perturbations.", "questions": "How exactly is the distribution p(C) of compression operators chosen during training? Is it uniform across quantization, pruning, and layer skipping, or weighted to reflect real deployment likelihoods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Strong empirical validation.\nExperiments span multiple open-source LLM architectures and reasoning benchmarks, showing broad generalizability. Results show negligible degradation compared to retrained baselines.\n\n2. Solid theoretical grounding.\nThe authors provide a theoretical bound explaining why the adapter remains effective under compression and temporal drift, supported by assumptions like Lipschitz continuity and bounded perturbations.", "weaknesses": "1. Limited exploration of layer-skipping robustness.\nResults show notable performance drops (e.g., MATH accuracy from 38.9% to 31.1%) under layer skipping. The authors mention this but do not deeply analyze why or propose mitigation strategies.\n\n2. Computational cost not negligible for initial training.\nAlthough amortized cost is lower, CAR-LoRA still requires longer single-run training (20 epochs vs. 5 for baselines). Some organizations might find this up-front cost high.\n\n3. Missing ablation studies.\nThe paper lacks a detailed breakdown of which components (e.g., quantized forward pass, structured pruning simulation, STE approximation) contribute most to robustness.\n\n4. Unclear figures and writing.\nThe paper writing needs improvement, with many details remaining unclear. The model architecture in figure 2 is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zlECnGMDRq", "reviewer": "Reviewer_XCK8", "summary": "This paper proposes CAR-LoRA, a training framework for creating a universal robust LoRA adapter. The key idea is to integrate simulated compression techniques (quantization, pruning, etc.) during training. Experiments across Llama-3.1, Mistral, and Gemma-2 on benchmarks like SQA, MATH, GSM8K demonstrate that CAR-LoRA matches or slightly outperforms specialized QLoRA adapters, while requiring only one training run.", "questions": "- Table 4 shows that CAR-LoRA requires 350 GB of GPU memory. Is that during training?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper is well-structured and articulates the deployment challenge (heterogeneous compression and evolving LLMs) clearly.\n- The authors use a toy example to motivate their problem.\n- The idea of training a single LoRA adapter under randomized compression perturbations is simple yet conceptually appealing.", "weaknesses": "- Incremental novelty:\n     - The core idea of injecting random compression perturbations during LoRA training is conceptually simple and largely derived from prior work in quantization-aware and robustness training.\n- Unsubstantiated claims:\n     - The paper claims that standard LoRA exhibits a “steeper decline” under temporal drift, but does not present quantitative evidence or citations to support this claim. Figure 4 appears to plot only CAR-LoRA results, with LoRA’s decline mentioned qualitatively. The absence of explicit LoRA baselines per checkpoint prevents verification of the claimed robustness gap.\n    -  In addition, while CAR-LoRA shows limited robustness to numerical quantization, there is no evidence of robustness to actual hardware diversity. There are no inference metrics, no deployment tests on constrained or heterogeneous devices, and no demonstration across hardware backends. This weakens the core claim of “hardware heterogeneity.”\n- Edge-deployment claims without device-level evidence: \n     - Despite the edge framing, the paper only uses  “Amortized” parameter/GPU-hour accounting (Table 4) to claim efficiency, but that doesn’t substitute for real deployment metrics. For edge devices, metrics such as latency, memory footprint, and energy consumption matter more than the time taken to train a LoRA for that device.\n- Failure to validate robustness for layer skipping:\n    - The method explicitly motivates robustness to diverse compression schemes, including layer skipping, but empirical results show that CAR-LoRA performs poorly when layers are removed.\n- Superficial theoretical analysis:\n    - Theorem 1 offers a generic Lipschitz-based bound decomposing errors into drift, compression, and generalization terms, but it lacks quantitative insight or predictive power.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J700OE66uc", "reviewer": "Reviewer_8M4z", "summary": "The paper introduces CAR-LoRA, a compression-aware, temporally robust LoRA training framework that uses a single universal adapter to work across various deployment settings. By sampling compression operators during training, CAR-LoRA trains a single LoRA adapter to stay effective across diverse compression settings(INT8/FP4/NF4, pruning, layer skipping). Experiments show near-parity or small gains vs. QLoRA, including transfer to unseen compressions and stability under continued-pretraining drift. The proposed approach aims to improve efficiency by avoiding retraining while maintaining competitive task performance.", "questions": "- In section 4.5 you claim CAR-LoRA is resilient to temporal parameter drift, but your test only compares checkpoints trained on the same data, where drift may be small. Suppose we train CAR-LoRA on base model using task A, then continue training base model on a different task B. If we apply the original CAR-LoRA to this B-updated model, does it still perform well on task A? What do you think about tackling this problem?\n- About layer skipping, what factors make LS particularly brittle. Do you consider other methods to close the gap?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Natural and persuasive method\n  - It cleanly combines “compressed-forward + STE-backward” to train a single adapter across heterogeneous deployments, matching real-world needs. The design is simple to implement yet principled.\n- Extensive and robust experiments\n  -  Across multiple backbones and standard reasoning benchmarks, the performance stays competitive with strong baselines, indicating the portability of this method.", "weaknesses": "- Insufficient comparison to naive LoRA.\n  - Figures 3/4 don’t plot naive LoRA performance across evolving checkpoints, so the claimed degradation is unsubstantiated. Please include the LoRA curve or a table with checkpoint-wise metrics.\n- limited cross-operator generalization evidence.\n  - “Unseen compression” tests stay mostly within the quantization family. A stronger test would train only under quantization and evaluate on pruning or layer skipping to demonstrate true cross-operator robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CWWhRQ17AD", "reviewer": "Reviewer_CNEk", "summary": "During the deployment of LLMs, compression is often a necessity. However, customizations through LoRA face a problem in that they are trained to fit one specific compression scenario and must be retrained for other scenarios. In this work, CAR-LoRA is proposed as a unified training framework that produces a single adapter that is both compression-aware and temporally robust. It achieves robustness through a two-loop structure. In the outer loop, it simulates different compression scenarios to make the LoRA adapter aware of potential compressions during deployment. Results show competitive evaluations on benchmarks compared with qLoRA.", "questions": "1. I'm not sure if I understand the notations around Theorem 1 correctly.  $\\|\\|\\Delta \\theta^* - \\Delta \\theta^*_{t,k}\\|\\|$ is a function of the difference between the applied adapter and the oracle adapter (line 272). \n\n (There seems to be some problems formatting LaTeX in OpenReview; the next lines belong to the same question)\n\n$\\Delta \\theta^*$ is the CAR-LoRA adapter (line 262). I think this is not a good notation definition. By notation, Theorem 1 proves that some unknown function of the difference between two LoRA weights has a bound. There are two potential problems. Are you directly subtracting the weights? How do you define the function (currently it says it's a function of the difference, but how to formally understand it)? Notation reads like it's the norm of the weight differences, but weight does not necessarily transfer to accuracy (i.e., the loss defined in Eq. 5). Intuitively, I seem to get what the authors try to say, but please either put it as an intuitive discussion or revise the notation.\n\n2. If possible, please provide the training loss curve for CAR-LoRA vs. qLoRA. \n\n3. Please provide additional experiments for 2-bit and 3-bit (INT2 and INT3) quantization when finetuned with CAR-LoRA that did not simulate those quantization during training (first point of weakness).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea is intuitive and seems easy to apply to current models.\n- CAR-LoRA shows competitive results with existing qLoRA while only needing one training for all compression.\n- It's important and beneficial to see the training cost report, and CAR-LoRA does not seem to be more costly.", "weaknesses": "- I'm not sure if this method makes the adapter generalize to \"all scenarios.\"\n\nCurrent ablations on the generalization of CAR-LoRA stop at 4-bit quantization. Does the conclusion in Sec 4.4 still hold if CAR-LoRA is tested on lower bits (2-bit or 3-bit)? We know that model weights need to be finetuned to a very distinct distribution if they were to be quantized to extremely low bits. I wonder if CAR-LoRA remains robust under those scenarios.\n\n- Needs more discussions for previous one-for-all attempts.\n\nThe authors miss discussions on previous attempts to make models robust to multiple compression scenarios. For example, [1] and [2]. Please search relevant works in this area and add them to the related works.\n\n[1] Xu et al. MultiQuant: Training Once for Multi-bit Quantization of Neural Networks, IJCAI 2022\n\n[2] Yi et al. One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments, ACL 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)