LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Authors: Yanan Li
Fanxu Meng
Muhan Zhang
Shiai Zhu
Shangguang Wang
Mengwei Xu
Venue: NeurIPS 2025 poster ()
Keywords: LLM Upgrades; LoRA Adaptation
Primary area: deep_learning
TL;DR: How can we leverage the existing LoRA weights to adapt to the latest model version with less effort?

Abstract: As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates a critical question: "How can we efficiently leverage existing LoRA weights to adapt to newer model versions?" To address this, we propose LoRASuite, a modular approach tailored specifically to various types of LLM updates. First, we compute a transfer matrix utilizing known parameters from both old and new LLMs. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory consumption by 5.5 GB and computational time by 78.23%.

LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Yanan Litel Fanxu Meng
Muhan Zhang'
Shiai Zhu Shangguang Wang Mengwei Xu
Beijinn University of Posts and Telecommunications Peking University #Unaffiliated (YaNanLi, egwang, wwwp)xbppt.eeddeeeee cn fxmengestu, pku. edu. en, muhan@pku edu. en, shiaizhu2-c@my cityu.edii edu.hi
CSCN
Abstract
As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retrain- ing LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates critical question: "How can we efficiently leverage existing LoRA weights to adapt newer model versions?" To address this, we propose LoRASuite, a modular approach tailored specifically to various types of L.LM updates. First, we compute a transfer matrix utilizing known parame- ters from both old and new L.Ms. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LcRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points math tasks, respectively. Additionally, LoRASuite significantly reduces memory con- sumption by 5.5 GB and computational time by 78.23%.
Introduction
LoRA [I] prominent parameter-efficient fine-tuning technique, has garnered significant attention for efficiently adapting pre-trained large language models (LLMs) to specialized downstream tasks using considerably fewer parameters than traditional full-parameter fine-tuning methods. A practical example is mobile device employing an on-device LLM integrated with multiple app-specific LoRA modules to customize capabilities in mobile agent scenarios [2.3].
However, frequent updates to LLM backbones, such the periodic releases of new versions of Llama 5 and Qwen 6 7) quickly render previously trained LoRA weights obsolete. Conse- quently, developers face problematic scenarios: (1) if the corresponding LoRA modules not promptly updated, app-specific functionalities may degrade or fail; (2) if updated, the prevailing method typically involves retraining LoRA weights from scratch. This process is time-consuming, costly, and even environmentally unsustainable. For instance, experimental results from [8] indicate that fine-tuning minutes. Similarly, research by [9] estimated that fine-tuning a sparse Mixtral model with two million queries on an NVIDIA H100 GPU costs approximately USD 3,460. Additionally, training a BERT model with 1 10 million parameters produces about 1,400 pounds of carbon dioxide equiv- compaaeoe to the emissions from round-trip transcontinental flight the United States
boo-billloon-pramette model on Google Pixel 7 Pro can take several hundred
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for one person [O]. This challenge is expected to escalate as the diversity of large models and the demand for various downstream tasks continue to increase. Thus, a natural question arises:
"How can leverage the existing LoRA weights to adapt to the latest model version with less effort?"
To address this previously unexplored question, we first categorize the upgrades of mainstream LLMs into six explicit limitations: vocabulary size, hidden size, intermediate dimensions of up/down- projection layers. layer depth, attention head count, and attention type. These limitations hinder the direct reuse of prior LoRA weights. Next, we propose LORASuiter ite, a modular approach designed to efficiently resolve each of these limitations, in contrast to the eurrent practice of retraining LoRA weights from scratch. For dimensional mismatches, LcRASuite utilizes known parameters from both the old and new LLM versions to compute a transfer matrix, enabling initial adaptation. To address differences in layer depth, we introduce a layer mapping algorithm that maps the LoRA weights from the i-th layer of the original model the j-th layer the upgraded model. Specifically, apply Centered Kernel Alignment (CKA), a representational similarity method, to measure the similarities between layers, and then use dynamic programming to maximize the total sum of CKA similarities. To handle differences in attention head counts, we represent each attention head using input-independent interaction matrices, denoted as Wox and Wo. We then introduce a head mapping algorithm based on the Hungarian method, maximizing the aggregate sum of cosine similarities. However, the transformed LoRA parameters primarily result from matrix multiplications rather than backpropagation, which may cause numerical instability and reduced performance. To mitigate this, we introduce small-scale, skillful fine-tuning stage to ensure the transformed LoRA parameters adapt effectively to the upgraded model while maintaining comparable performance.
We evaluated LoRASuite across multiple tasks, including commonsense reasoning and mathe- matical benchmarks, using various backbone LLMs. Experimental results show that LoRASuit S consistently outperforms scall-scale vanilla LoRA. Notably, for backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LcRASuite significantly reduces memory usage by 5.5 GB and computational time by 78.23%.
Our key contributions are summarized as follows:
To the best of our knowledge, this work is the first to systematically address the chal- lenge of efficient LoRA weight transfer during LLM upgrades, analyzed from six explicit perspectives.
We propose LoRASuite, a modular approach tailored to various types of LLM upgrades, including mayer-mapping strategy based on centered kernel alignment and an attention-heal mapping approach utilizing the Hungarian algorithm.
LoRASuite consistently outperforms small-scale vanilla LoRA across diverse tasks insks-including commonsense reasoning and mathematical andduatioos-nnd, in certain scenarios, even surpasses full-scale LoRA retraining. Additionally, it significantly reduces both memory and time consumption.
2 Related Work
Interpretability LLM. Mechanistic interpretability in deep neural networks has gained significant
attention in recent years. Prior work has largely focused on analyzing hidden representations through techniques like probing [T], activation patching [[2]. and causal tracing [13]. or on interpreting
specific network weights by mapping model components to vocabulary space [1]. Our approach is
based on the latter, and future work can further explore the integration different methods. For a comprehensive discussion mechanistic interpretability in transformer-based LLMs, see [15]
Efficient Fine-tuning. This work builds on LoRA, a classical and efficient fine-tuning method. Beyond LoRA, several other parameter-efficienn fine-tuning (PEFT) techniques have been proposed. Adapter tuning [16] inserts lightweight trainable modules into the frozen backbone. Prompt tun- ing I7 appends trainable soft tokens to the input sequence, while prefix tuning extends this idea by inserting soft tokens into each layer's hidden representations. Hidden state tuning [18), exemplified
by (IA)3, rescales attention keys and values as well as activations in the feed-forward layers. Bias

---

Table 1: Summary of LLM upgrades with the same architecture (selected mainstream models).
Highlighted cells
indicate differences introduced during upgrades.
Architecture
Model
Vocab, Size
Hidden size
Interm. size
# layers
of heads
Attention
Yi-6B
64k
4096
11008
32
32
GQA
Ll
Yi-1,5-9B
64k
4096
11008
48
32
GQA
pythia-Ib
50k
2048
8192
16
8
MHA
GPTNeelNFoCCuuaaI.
pythia-1.4b
50k
2048
8192
24
16
MHA
Noom--560m
250k
1024
4096
24
16
MITA
BIoomForCausalIM
bloomz- lbl
250k
1536
6144
24
16
MHA
Llama-2-7b
32k
4096
1008
32
32
MHA
LlamaForCausalL
Llama-3-8b
128k
4096
14336
32
32
GQA
Qwen- 1.5-1.8B
152k
2048
5504
24
MHA
Qven2ForCausalIM
Qwen-2.5-3B
152k
2048
11008
36
GQA
MiniCPM-S-1B
73k
1536
3840
52
24
GQA
CiniCPKPPoCauuall.
MiniCPM-2B
123k
2304
5760
40
MHA
tuning [9]
updates only the model's bias terms selected subset. Masked weight learning 20)
applies a fixed sparse mask to the model's 's parameters. Input tuning [2] introduces an adapter at the embedding layer to adjust input representations.
Optimization for LoRA. LoRA [. one of the most effective methods for parameter- efffcceet fine-tuning, has gained significant attention in recent years, leading to numerous variants and opti- mizations. example, AdaLoRA [22 dynamically learns required rank size for each model layer, LoSparse [23] integrates LoRA prevent pruning from removing too many expressive neurons, DoRA [24] adapts LoRA based on magnitude and direction, and PiSSA (25 tunes the essential low-rank components while freezing high-rank. nonessential parts. LoRA+ (26) applies different learning rates to the and modules. Our work is orthogonal to these approaches, and future research can explore more efficient combinations of our method with these variants.
Knowledge Transfer between LLMs. Prior research on knowledge transfer between pre-trained language models has primarily focused on transferring prompt- based modules 27
These methods typically align overlapping vocabularies between models, which is conceptually similar to our vocabulary and hidden-size alignment via linear combination described in Section 3.1 While several studies examine transferability LoRA or PEFT modules, they often rely on modifying the original fine-tuning process introducing additional training steps, rather than enabling direct and efficient transfer. For example, Trans-LoRA (30 improves LoRA transferability through synthetic data generation, Trans-PEFT [31] enhances PEFT transfer via modified fine-tuning procedures, and MUSCLE [32] employs knowledge distillation to transfer adapters across models. Furthermore, means of incremental model updat- as--ooolo to implicit dataset upgrades in our setting where direct transfer proves to be the most effective approach.
28
29
some works 33 34 investigate continual pre-training
LoRASuite
Given a pre-trained weight matrix W, E &Rdoxdo; in the original LLM, LoRA trained on a specific downstream corpus comprises low-rank matrices, A, E Rrxxd, and B, Rd.xx, where << (di do,) Typically, matrix B, is initialized with zeros, and A, is with Kaiming Uniform 35 ensuring B, 0 initially.
Upon upgrading the LLM backbone to newer version, as summarized in Table I six explicit factors-vocabullat size, hidden size, intermediate dimension of (p/doww--projcttin layers, layer depth, attention head numbers, and attention type-prevent direct reuse of existing matrices A, and B, for the new weight matrix W. E R. xda,."
Therefore, this section introduces several methods to adapt existing LoRA weights A, and B, to new matrices A,, and B,, without retraining from scratch when upgrading from W, to W,, the same corpus. We specifically focus on upgrades within L.LMs of identical architecture, as defined their respective config, json files. For instance, Phi-1.5 and Phi-2 share the same PhiForCuusalLM

---

architecture, while Phi-3 has different Phi3ForCausalLM architettrr. Consequently, the activation function remains unchanged in our scenario. Exploring the impact of varying activation functions represents a promising direction for future research. In addition to the six explicit upgrade factors, we recognize that implicit upgrades, such as changes in pre-training datasets and post-training methods (e.g., RLHF), also influence model adaptation. Future research could further investigate the effects of these implicit factors.
3.1 Methodology
Vocabulary Size and Hidden Size. Vocabulary size and hidden dimension directly influence the embedding layer weights of model, with hidden dimension mismatches notably restrict- ing the direct reuse of existing LoRA weights. Based on variations in vocabulary size and hid- den dimension, upgrades can be classified into
Table 2: Classification for the upgrades concerning
vocabulary size (VS) and hidden size (HS).
FROM
To
VS
HS
P.I-1.5
PHI-2 LLAMA-3-88
LLAMA-2-78
MINICPM-IB MINICPM-2B
three scenarios, as detailed in Table 2 In the simplest cecnario-whert only the hidden dimension changes-the the transformation matrix W;, can be directly computed using the embedding weights of both the original and upgraded models. Assuming that all parameters from both models are available, the transfer matrix is specifically computed as Wn- = E, En, where E. and E, denote the embedding weights of the original and updated models, respectively. For scenarios involving simultancous changes in vocabulary size and hidden dimension, an additional intersection step is required to filter shared tokens, after which the transformation matrix is calculated using the filtered embedding weights.
Intermediate Size. When LoRA target mod- ules contain up and down projection layers, mis- matches in intermediate dimensions pose an- other significant challenge. Following a sim- ilar strategy, we leverage known parameters from both models compute the transforma- tion matrix W,. Specifically, W, is calculated as W/ W W,, where W. and W,, rep- resent the weights of the original and updated up/down projection layers, respectively.
Algorithm 1 CKA-based Layer Mapping
Input: CKA similarity matrix S E R Original and upgrade layer depth to and ln. Output: pathii] [b] with the highest total CKA.
/# Store the maximum sum Initialize dp[i][i] < -oo for all i,j /# Store path inf Informaioon Initialize path|]]]b]] <- 0 for all i
/# Limit maximum offset for j = 0 to 14, - 1o do dp[o][]] + S00]
Layer Depth. Inspired by prior studies on rep- resentational similarity, we employ centered ker- nel alignment (CKA) to quantify the similar- ity between corresponding layers of two LLMs. Following [36), we adopt minibatch implemen- tation of CKA to reduce memory usage. Based on the computed CKA similarities, we propose a novel dynamic programming- based layer map- ping algorithm, which sequentially aligns cor- responding layers to maximize the total simi- larity, subject to predefined maximum offset constraint.
end for
for i to 1 do
/# Limit maximum offset
for j = i i ||, 4/1 do
max_v v -00
max_i i -
for k = to j-j do
/# Transfer equation
if dpli 1][k] [|][]]]] S > mar then
max_v <- dpli 1][ + S
CKA (37] provides a robust method for quan- tifying neural network representations by mea- suring the similarity between pairs of activation tensors. Let X m and Y E mx denote the activations of two layers, with d1 and d2 neurons, respectively, across the same set of m examples. The elements of the Gram matrices K = XXT and L = YYT repre- sent the pairwise similarities between examples based on X and Y. Using the centering matrix H = I, 111T_ the centered Gram matrices K' HKH and T'' .T.H remove mean ef- fects. HSIC measures their similarity by flattening the matrices and computing the dot product:
mar_ < k
end if end for dpli +- max_
path f- max
end for
end for

---

HSICo(K, L) vee(K") (ec(L')/ - 1)2. HSIC is invariant to orthogonal transformations and neuron permutations but remains sensitive to scaling. CKA normalizes HSIC to yield similarity index between ( and that is invariant to isotropic scaling.
HSICo (K, L) (H, K K)HSIC,(L,
CKA(K, L)
However, computing CKA naively requires storing activations for the entire dataset, which is impractical for wide and deep LLMs. To reduce memory usage. [36] proposes estimating CKA
by averaging HSIC scores across & minibatches HSIC, K, I.) = Σ i=1 HSIC, [X, Y,Y)).
In place of HSICo, which is a biased estimator of HSIC, the usage of an unbiased estimator of HSIC, [88 makes the value of CKA independent of batch size:
1 nin 33 (r(KL) +
L' 2 (n - 1)(n 2)
HSIC, HK,
n 1"
('KL
(2)
where K and L are obtained by setting the diagonal entries of similarity matrices K and zero. This minibatch-basse HSIC estimation is equivalent to the bagging block HSIC method in [3] and converges to the full-dataset HSIC, as proven in (36]
Using the minibatch CKA defined above, we construct a similarity matrix S, where each element Si represents the CKA similarity between the i-th layer of the original model and the j-th layer of the upgraded model. Inspired by dynamic programming, we propose a layer-mapping algorithm that maximizes the total similarity in S by optimally aligning corresponding layers. To further guide the alignment, we impose an ordered mapping constraint, restricting the layer assignment within a threshold Lager, which accounts for differences in layer depth between the original and upgraded models. The detailed procedure is outlined in Algorithm
Head Number. Recall that the attention projection matrices WQ. WK. and Wy can be split along the column axis H parts, denoted as W Wk and Wi E Rdxd/H for 1 < i S H. Similarly, the output projection matrix Wo is split along the row axis into heads, with W E Rd/HHd For each head, we define two input-independent interaction matrix: Wax W
T Rdd and
Wiv Wi: T Radd Intuitively, Wax captures the attention strength between token pairs, while oio models how attending to specific tokens influences the subsequent hidden state.
To address differences in attention head counts, characterize each head using its interaction matrices and compute similarity matrix via cosine similarity between heads from the old and new models, at the layers specified above. When interaction matrix dimensions differ, align them by transforming to a common hidden size de (typically the new model's hidden size) using the embedding- based transformation W₁, We then determine the optimal head mapping by maximizing the sum of similarities, applying the Hungarian algorithm with the similarity matrix treated as a cost matrix. Further algorithmic details and theoretical analysis are provided in Appendix
Changes in attention mechanisms primarily affect the number of WK and Wy heads. For instance, transitioning from GQA to MHA in MiniCPM-1B MiniCPM-2B increases the number of K and V heads from 8 to 36. in the forward pass, we replicate the K and V heads to match the number of Q heads before applying the head mapping algorithm.
Head Dimension. Following the attention head granularity used in the model's for- ward pass, we also adapt LoRA weights to the attention head level. As shown in Ta- ble the hidden size and number of atten- tion heads jointly determine the dimension of each head. Similar to earlier steps, we leverage known parameters from both the original and new models to adapt the LoRA weights. Specifically, first compute the
Table 3: Classification for the upgrades concerning hidden size (HS), head number (HN), and their ratio head dimension (HD HS / HN).
FROM
To
HS
HN
HD
PYTHIA-IB PHI-1.5
PYTHIA-1.44 PHI-2
/
LLAMA-2-7B
LLAMA-2-13B
PYTHIA-410M
PYTHIA-IB
5

---

Algorithm 2 Pseudocode of LoRASuite in PyTorch-like style.
/# Load the original LoRA weights.
Bo, A, load_peft_weighheeee (ORIGINAL
/# Calculate the hidden size transformation matrix.
W₁ =
NEW_TOKENIZER)
/# CKA-based Layer Mapping, where S. represents the CKA similarity
between the i-th layer of the original model and the j-th layer of the upgraded model.
Ldiet = _aa__aayr__apppnnne for each and j in Ldict do
AW.,; Ao.
/* Compute the Hungarian-based head mapping between the ith projection layer of the original model Wo, and the jth layer of the new model */ Hdiet = hhunaarian wwahheaammme masddd_appiinn..... Wn,)) for each ho and hn, in Haie do
/# Head-level transformation based on Equation
3
AWhh- n.j
W AW W Whh
m.j
end for
Bn.j. An.j mV,, end for
weight update AW, for each projection matrix by multiplying the corresponding B, and A.. then split it along the specified axis to extract updates for individual heads. Let (W and (AW)). 2 denote the original weights and weight updates for the i-th head in the original model. Based on the Hungarian algorithm, the weight update for the j-th head in the new model is computed as:
(AWz). W (AW2). (W Wt (W?)
where, Wn, derived from the embedding weights of both models, is omitted if the hidden size remains unchanged. Incorporating original weights improves numerical stability, allowing the transformed LoRA be directly applied in some cases. Finally, after all head mappings are completed, a single SVD step decomposes (A n into its low-rank B, and An components.
We employ different strategies for layer and head mapping due to their distinct structural roles. For layer mapping, we adopt a dynamic programming approach inspired by studies on CNNs, where different layers capture different levels of abstraction, and higher layers build upon the representations of lower layers [40]. In contrast, attention heads within the same layer operate in parallel and formulate head mapping as bipartite matching problem and solve it using
independently [4 the Hungarian algorithm to obtain the optimal one-to-one correspondence.
3.2 Put All Together: A Final Recipe
Integrating the five components described above, the pseudocode of LoRASuite is presented in Algorithm 2 The CKA-based layer similarity and head similarity between the original and upgraded models can be precomputed offline, eliminating the need for repeated calculation during each adaptation.
Since the transformed LoRA parameters are generated via matrix multiplication rather than back- propagation, they may introduce numerical instability and degrade performance. To mitigate this, we introduce lightweight fine-tuning step help the upgraded model better adapt. Unlike standard Transformer Trainer that uses a lincar learning-rate scheduler with warm-up to stabilize optimiza- tion from random initialization 42) model is initialized with transformed parameters. Therefore, we omit the warm-up phase and increase the learning rate to compensate.
3.3 Complexity Analysis
The complexity of LoRASuite primarily arises from three key processes: layer mapping using the CKA method, attention head mapping utilizing the Hungarian algorithm, and the final SVD. As illustrated in Algorithm 1, the CKA-based layer mapping has time complexity of O 2 layer)

---

Table 4: Performance on math tasks during the LLM upgrade from MiniCPM-S-ID to MiniCPM-2B. LFT denotes additional lightweight fine-tuning using small-scale data. Numbers in parentheses indicate the size of the fine-tuning datasets. represents the performance of the vanilla model. Bold and underlined entries indicate the best and second-best results, respectively.
Base Model
PEFT
AddSub
MultiArith
SingleEq
GSM8K
AQuA
MAWPS
SVAMP
Avg.
23.80
5.17
23.62
7.43
17.72
20.17
16
16.27
MiniCPK-S-- -AACC
LoRA (10k)
29.62
54.67
31.5
12.13
8.27
31.93
18.2
26.62
23.29
27.56
12.28
14.57
22.27
23.85
LoRASuite LFT
23.04
28.15
12.36
22.27
20.7
23.96
MiniCPM-2B
LoRA (100)
20.76
40.33
28.54
11.98
15.35
23.95
18.8
22.82
LoRASuite LFT (100)
54.18
75.5
56.69
18.73
14.17
50.84
36.5
43.80
LoRA (10k)
47.59
84.67
47.44
17.97
19.69
47.48
31.9
42.39
Table 5: Performance on commonsenss tasks when LLM upgrades from MiniCPM-S-IB to MiniCPM- 2B. LFT denotes additional lightweight fine-tuning using small-scan -cale data. The number in parentheses represents the scale of fine-tuning datasets. "." represents the performance of the vanilla model. Bold and underlined entries indicate the best and second-bess results, respectively.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-c
ARC-e
OBQA
Avg.
$1.9
33.68
13.46
12.09
39.78
9.56
12.16
8.8
22.68
MiniCPK-S--D
LoRA (10k)
58.17
27.48
53.74
20.63
59.67
27.39
35.23
40.6
40.36
53.55
22.8
39.66
14.5
52.33
20.14
27.19
31.6
32.72
LoRASuite w/o LFT
53.64
22.69
40.17
14.61
52.09
20.56
26.77
31
32,69
MiniCPM-2B
LoRA (100)
63.18
42.37
15.77
50.99
28.5
35.27
40.8
38.60
LoRASuite LFT (100)
61.28
43,91
48.21
24
51.07
32.42
39.02
38.8
42.34
LoRA (10k)
62.35
26.71
50.61
21.17
60.77
27.73
34.64
42.8
40.85
where Paayer denotes the number of layers in the original model, and Atager represents the differ- ence in the number layers between the original and upgraded models. According to [43], the Hungarian algorithm exhibits time complexity of O where Rhead indicates the smaller number of attention heads present either in the original or upgraded model. Based on on the complexity of the SVD process obtain the B Ang in j-th layer is (nddedde Combining these factors, the total computational complexity becomes O(Mayer (Ataeer nhead n ))dddde)) Since Rhidden >> Rhead, Aageer, the overall complexity is effectively dominated by the SVD and simplifies to (Maager Ti hidden)
Experiments
Implementation Details. Our implementation" builds on the publicly available Huggingface transformers and peft libraries, with modifications support initialization from transformed weights. All experiments were conducted on a Linux server equipped with 80 Intel(R) Xcon(R) Gold 6148 CPU @ 2.40GHz cores, 500 GB RAM, and NVIDIA V100 GPUs.
Experimental Settings. For commonsense reasoning tasks, we evaluate BoolQ PIQA 46). SIQA 47). HellaSwag 488. WinoGrande [49, ARC-e, ARC-c [50]. and OBQA 51. For math tasks,
we evaluate AQuA 52,. GSM8K (53), MAWPS (54 and SVAMP 55
To ensure reproducibility, we use the same training and evaluation datasets as LLM-Adapters [56]. Both the original and upgraded models' LoRA (10k) are trained on the full dataset with 10k samples, whereas LoRA (100) and LoRASuite with LFT (100) are trained on identical, randomly selected subsets of the LoRA (10k) dataset with 100 samples.
Hyperparameter Settings. Unless otherwise specified, we use the default LoRA settings: rank 32, alpha 32, and dropout O. A sensitivity analysis of these parameters is provided in Section 4.2 By default, the target modules include q-proj. k_proj, v_proj, and o_proj, with performance for other modules detailed in the Appendix A.3
https//ggihhbb.co//mmmmLl/L//

---

25
80-
74.3
60
Avg. Accuracy (%
Avg. Accuracy (%)
Aove Accurs currayy (%)
Aung Accursey
20
Narg Accurscy
+0
40.00
40
86
10-
20-0
20
10.12
Pythia Pythia-1.40
Bloom-56000
Aloomo-181 1881
1lama-2-78
Llama-3-58
(b)
Figure 2: Average performance comparison math tasks for different types of L.M upgrades.
4.1 Overall Performance
As shown in Table 1 upgrading from MiniCPM-S-IB IB to MiniCPM-2B involves all six explicit aspects. Thus, we primarily use MiniCPM to demonstrate LoRASUi te's overall performance.
Accuracy. and present performance results for math and commonsense tasks, respectively, "LoRASuite w/o LFT denotes LoRASuite performance using only the transformed MiniCPM- S-IB LoRA without further fine-tuning with small-scale data. "LoRASuite w LFT (100)" indicates LcRASuite with additional fine-tuning at data scale of 100. For MiniCPM-22B, "LoRA (10k)" represents LoRA retraining using the complete dataset, while "LoRA (100)" serves as a baseline for retraining at the same scale. Under the default setting with rank 32 for MiniCPM-22B, the number of trainable parameters for LoRA (100), LcRASuite with LFT (100), and LoRA (10k) is the same-233 2...--..59M.. which is approximately 0.78% of all model parameters. From the tables, we observe: First, LoRASuit te achieves notable average score of 43.80 on math tasks, nearly doubling the performance compared same-scale fine-tuning (22.82) and surpassing full dataset retraining (42.39). Similarly, in commonsense tasks, LoRASuite outperforms both scale LoRA fin-tuning and full dataset retraining.
Additionally, "LoRASuite w/o LFT exhibits performance nearly identical to the vanilla MiniCPM- 2B model on both math and commonsense tasks. This result supports the hypothesis that transforma- tion relying solely on matrix multiplication without backpropagation leads to numerical instability. It further validates the effectiveness of additional small-scale fine-tuning.
Memory and time consumption. Figure
ToRAGube wa ITT
FaRAGule
ToRA Petraining
demonstrates that LoRASuite significantly outperforms retraining a new LoRA model, achieving memory savings of 5.5GB and reduc-
50
Max Memory (GB)
40
36.72
Time (min)
100
30
ing time consumption by 78.23%. Although
20
LoRASuite requires a small-scale fine-tuning stage, it yields a modest memory reduction of 5.5 GB due to three factors. First, data pipeline overhead and auxiliary states (e.g.. DataLoader prefetching, dataset shuffling, and caching) scale with both the number of samples and training steps. Second, preloading the full dataset or pre-tokenized storage directly increases host DRAM usage. Third, for variable-length inputs, larger datasets increase the likelihood of leng-sequence batches, resulting in excessive padding and transient memory spikes from dynamic allocation. For "LoRASuite w/o LFT," the memory reduction is more substantial, reaching 36.36% (15.34GB). This memory efficiency stems from loading only the parameters of the new and old models for matrix operations, thereby eliminating the overhead associated with optimizer states and gradients. The notable decrease time consumption occurs because LoRASuite avoids the need for loading extensive datasets, performing backpropagation, or training for multiple epochs from scratch. Even with small-scale fine-tuning, the additional time is only approximately two minutes.
10
(a)
Figure 1: Memory and time comparison between LoRASuite and LoRA retraining.
Multiple LLM backbones. Figures 2 and present the average performance comparisons across math and commonsense tasks, respectively, for different LLM backbones. The required amount of additional fine-tuning data for LoRASuite varies across different LLM backbones. Henceforth, we mainly use "LoRASui
to refer to the variant with lightweight fine-tuning (LoRASui te w LFT (Small)), which also applies to the comparison method LoRA (Small). Our observations from these figures are as follows: First, LoRASu te consistently achieves performance comparable to or exceeding full-scale LoRA retraining in several scenarios. Second, LoRASuite significantly

---

Aug. Accuracy (%)
Aug. Accuracy (%)
Aur. Accurscy (%)
Jova Accurao (36)
60
SEXL
Jug. Accurscy
60
30
40
40
20
20
20-
Pythie-18
Pythia-1.48
Aloomo-181
Llama-2-78
Llama-3-58
(b)
(6)
Figure 3: Average performance comparison on common tasks for different types of LLM upgrades.
45
45
40
40
40
Accurocy
Avr. Accuracy
Aug. Accuracy
35-
LoNASuie 1 (huil
25
LoftACuuee be LoRA (Small)
-3-
LORA (Small)
30
-30
30
25
25-
LofASuute
IoRA (Smwll
1c.4
30-4
6o-4
9c-4
10-3
100
200
400
800
1000
Rank
Learning Rate (b)
Datascale [c]
(a)
Figure 4: Average performance on math tasks under different settings for the MiniCPM-S-IB to MiniCPM-2B upgrade.
outperforms same-scale LoRA fine-tuning (LoRA (Small)) across all cases, with notably pronounced improvement-nearly threefold-in math tasks when upgrading from Swwnn-1.5-1.. to Qwen- 2.5-3B. Additionally, LoRASuite consistently surpasses "LoRASuite w/o LFT," highlighting lightweight small-scale fintuing's critical role in enhancing numerical stability. Finally, "LoRASuir w/o LFT" occasionally achieves performance similar or slightly better than the vanilla model and "LoRA (Small)," suggesting that numerical stability may not be a significant concern for certain models. Future research could focus on developing more robust algorithms that avoid additional fine-tuning and accommodate diverse model update scenarios.
4.2 Sensitive Analysis
Different rank, aa presents the performance comparison between LoRASuite and baseline methods on math tasks across varying LoRA ranks. LoRASuite consistently outperforms vanilla LoRA in both small-scale and full-scale fine-tuning scenarios. Notably, at a LoRA rank of 4, LcRASuite achieves 93% of the performance obtained from training with 10K samples using only 100 samples. At LoRA rank of 16, LoRASuite e achieves its greatest improvement over full-scale LoRA, with an average increase of approximately 2.67 points.
Different LFT learning rate. The experimental results depicted in Figure 4 (b) indicate that Lo- RASuite is highly sensitive changes the learning rate compared to vanilla LoRA (Small). Specifically, when the learning rate is le-4, LoRASuite marginally outperforms LoRA (Small) by only 0.27 percentage points on average. However, at a learning rate 9e-4, LoRASui te sub- stantially outperforms LoRA (Small) by average of 21.30 points, representing approximately twofold improvement in performance. This sensitivity likely arises because LoRASui te leverages previously trained LoRa weights to quickly identify crucial task-specific features, creating steeper optimization landscape and greater dependence on the chosen learning rate.
Different LFT data scale. When trained on small datasets, LoRASuite maintains stable per- formance and consistently outperforms LoRA (Small). However, its performance declines with increasing dataset size, likely due to LoRASuite reliance on historically trained LoRA weights, making it more susceptible to overfitting.
4.3 Application to Other PEFTs
Table and Table report the average performance of two PEFT methods, AdaLoRA and DoRA, on math tasks when upgrading the LLM from MiniCPM-S-II 1B to MiniCPM-2B, As shown, LoRASui te yields substantial improvement for DoRA adaptation: while small-scale vanilla LoRA achieves
only 23. 23.54% accuracy, LoRASuite improves this by more than 1.8x, reducing the performance

---

Table 6: Performance on math tasks during the LLM upgrade from MiniCPM-S- IB to MiniCPM-2B with Ad.LoRA. Numbers in parentheses indicate the size of the fine-tuning datasets. Bold and underlined entries indicate the best and second-bes best results, respectively.
Base Model
PEFT
AddSub
MultiArith
SingleEq
GSMBK
AQuA
MAWPS
SVAMP
Avg.
23.80
5.17
23.62
7.43
17.72
20.17
16
16.27
MiniCPK-S S-1B
AdaLoRA (10k)
48.61
76.67
58.66
27.29
17.32
52.1
36.2
45,26
23.29
48
27.56
12.28
14.57
22.27
19
23.85
LoRASuite w/o LFT
42.28
67.17
50
25.63
13.78
33.19
34.2
38.04
MiniCPM-28
AdaLoRA (Ik)
52.41
71.67
59.66
28.81
19.29
52.94
35.8
45.80
LoRASuite w LFT (Ik)
39.75
82.67
65.35
31.46
20.47
56.3
41.20
51.03
AdaLoRA (10K)
55.44
82.17
65.55
31.69
14.57
63.03
40.80
50.46
Table Performance on math tasks during the L.LM upgrade from MiniCPM-S- to MiniCPM-2B with DoRA. Numbers in parentheses indicate the size of the fine-tuning datasets. Bold and underlined entries indicate the best and second-best results, respectively.
Base Model
PEFT
AddSub
MultiArith
SingleBq
GSM8K
AQuA
MAWPS
SVAMP
Avg.
23.80
5.17
23.62
7.43
17.72
20.17
16
16.27
MiniCPK-S-1B
DoRA (10k)
31.90
60.67
31.89
12.59
12.60
32.35
20.80
28.97
23.29
27.56
12.28
14.57
22.27
19
23.85
LoRASuite w/o LFT
22.03
29.13
12.51
16.14
23.11
19.30
24.17
MiniCPM-2B
DoRA (100)
22.53
27.76
11.90
15.75
23.53
19.30
23.54
LoRASuite L.FT (100)
55.44
55.91
19.33
37.20
42.88
DoRA (10k)
47.09
48.43
17.66
20.47
47.06
34.60
43.19
gap with full-scale LoRA to less than 1%. In contrast, the improvement offered by LoRASuite over LoRA (Small) is less pronounced for AdaLoRA, although it still surpasses full-scale LoRA retraining. This discrepancy is likely due to rank incompatibilities between corresponding layers of the original and upgraded models.
Conclusion and Discussion
We propose LoRASuite, modular framework designed to handle diverse types of LLM upgrades. To the best of our knowledge, this is the first work to systematically address the challenge of efficient LoRA weights transfer during LLM upgrades across six explicit perspectives. LoRASuite consistently outperforms small-scale vanilla LoRA across diverse tasks and, in certain scenarios, even surpasses full-scale LoRA retraining with both memory and time reduction.
Limitations. Currently, LoRASuite requires an additional small-scale fine-tuning step achieve superior performance. Future research could investigate strategies to eliminate this step without compromising performance, further minimizing memory usage. Additionally, our study primarily addresses explicit upgrade aspects of LLMs; future investigations could extend to implicit upgrades, such variations pre-training datasets and post-training methods, with knowledge editing meth- ods 13 57 58). Lastly, exploring LoRA adaptation methods applicable to different architectures remains an open avenue for future work.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their insightful feedback and Longxi Gao for assistance with complementary experiments. This work was supported by the National Natural Science Foundation of China (No.62425203, No.62032003) and partly by the Supercomputing Platform of Beijing University Posts and Telecommunications Fanxu Meng and Muhan Zhang supported by the National Key R&D Program of China (2022/D016 60300). Mengwei Xu was also supported by the National Natural Science Foundation of China (No.6252222) and the Beijing Natural Science Foundation (No.L253005), Mengwei Xu is the corresponding author.
10

---

References
[1] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," " in International Conference Learning Representations, 2022. [Online]. Available: https:/opparvieeeee/fffuummiimi
YeeFFf9
[2] Google, "Gemini nano with the google ai edge sdk," 2025. [Online]. Available: http:///wwepppeaaaooooooo/c////a
[3] H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, J.J., S. Jiang, Y. Liu, Y. Zhang, and Y. Liu, "Autodroid: Lim-powered task automation in android," in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024, pp. 543-557.
[4] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288. 2023.
[5] Meta, "The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation." 2025. [Online]. Available: https//ai.mmtaa.ombbb//laaaa 4- multimodal multimmodal-melllgennee
[6] A. Yang, B. Yang, B. Hui et al., "Qwen2 technical report," arXiv preprint 2024.
[7] A. Yang, B. Yang, B. Zhang et al., "Qwen2.5 technical report," arXiv preprint arXiv:2412.15115 2024.
[8] M. Xu, D. Cai, Y. Wu, X. Li, and S. Wang. "Fwdllm: Efficient federated finetuning of large language models with perturbed inferences." in 2024 USENIX Annual Technical Conference (USENIX ATC 24), 2024, pp. 579-596.
[9] Y. Xia, Kim, Y. Chen, Ye, Kundu, C. Hao, and Talati, "Understanding the performance and estimating the cost of Ilm fine-tuning," arXiv preprint arXiv:2408. 2024.
[10] E. Strubell, A. Ganesh, and A. McCallum, "Energy and policy considerations for modern deep learning research," in Proceedings of the AAAI conference on artificial intelligence, vol. 34, 09, 2020, pp. 13 693-13 696.
[11] A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, "What you can cram into single vector: Probing sentence embeddings for linguistic properties," arXiv preprint arXiv:
2018
[12] A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-lonso, "Towards automated circuit discovery for mechanistic interpretability." Advances in Neural Information Processing Systems, vol. 36, pp. (6318-16352. 2023.
[13] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, "Locating and editing factual associations in gpt," Advances in Neural Information Processing Systems, vol. 35, pp. 359-17 17 372, 2022.
[14] G. Dar, M. Geva, A. Gupta, and Berant, "Analyzing transformers embedding space," arXiv preprint arXiv:2209.025.3 2022.
[15] D. Rai, Y. Zhou, S. Feng. A. Saparov, and Z. Yao, "A practical review of mechanistic inter- pretability for transformer-based language models," arXiv preprint arXiv:2407.02646 2024.
[16] N. Houlsby, A. Giurgiu. S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo. M. At- tariyan, and S. Gelly, "Paraneter-efficient transfer learning for nlp," in International conference on machine learning. PMLR, 2019, pp. 2790-2799.
[17] B. Lester, R. AI-Rfou, and N. Constant, "The power of scale for parameter-efficient prompt tuning," arXiv preprint arXiv:2104.08691, 2021.
[18] H. Liu, D. Tam, M. Mugeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, "Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning," Advances in Neural Information Processing Systems, vol. 35, pp. 1950-1965, 2022.
11

---

[19] E. B. Zaken, S. Ravfogel, and Y. Goldberg, "Bitfit: Simple parameter-efficient fine-tuning for transformer-bas based masked mogguage-modele arXiv preprint arXiv:2106.10199 2021.
[20] Y.-L. Sung, V. Nair, and C. A. Raffel, "Training neural networks with fixed sparse masks," Advances in Neural Information Processing Systems, vol. 34, pp. 24 193-24 2021.
[21] An, Li, Lin, Q. Liu, B. Chen, Q. Fu, W. Chen, N. Zheng, and J.-G. Lou, "Input-tuning Adapting unfamiliar inputs to frozen pretrained models," arXiv preprint arXiv:2203.03/3. 2022.
[22] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, "Adaptive budget allocation for parameter-efficient fine-tuning," in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https//opennevieee.e//fommddddd Ig62uWRJjiY
[23] Y. Li, Y. Yu, Q. Zhang, C. Liang, He, W. Chen, and T. Zhao, "Losparse: Structured compres- sion of large language models based on low-rank and sparse approximation," in International Conference on Machine Learning. PMLR, 2023, pp. 20 306-300333 350.
[24] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, "Dora: Weight-decomposed low-rank adaptation." arXiv preprint arXiv:2402.09353. 2024.
[25] Meng, 7. Wang, and M. Zhang, "Pissa: Principal singular values and singular vectors adaptation of large language models," arXiv preprint arXiv:2404.02945 2024.
[26] S. Hayou, N. Ghosh, and B. Yu, "Lora+: Efficient low rank adaptation of large models," arXiv preprint arXiv:2402. /23554, 2024.
[27] Su, X. Wang. Y. Qin, C.-M. Chan, Y. Lin, H. Wang, K. Wen, Z. Liu, J. ei al., "Ontrans- ferability of prompt tuning for natural language processing," arXiv preprint arXiv:2111.067/9 2021.
[28] B. Lester, J. Yurtsever, S. Shakeri, and N. Constant, "Reducing retraining by recycling parameter- efficient prompts," arXiv preprint 2xX1v:2008.05577, 2022.
[29] Z. Wu, Y. Wu, and L. Mou, "Zero-shot continuous prompt transfer: Generalizing task semantics across language models," in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available:
[30] R. Wang, S. Ghosh, D. Cox, D. Antognini, A. Oliva, R. Feris, and L. Karlinsky, "Trans-lora" towards data-free transferable parameter efficient finetuning," Advances in Neural Information Processing Systems, vol. 37, pp. 61 217-61 237, 2024.
[31] N. Gu, Fu, Liu, K. Ma, 7.. Lin, and W. Wang, "Adapt once, thrive with updates: Transferable parameter-efficien fine-tuning on evolving base models," arXiv preprint arXiv:2506.06844, 2025.
[32] J. Echterhoff, F. Faghri, R. Vemulapalli, T.-Y. Hu, Ci, O. Tuzel, and H. Pouransari, "Muscle: A model update strategy for compatible Ilm evolution," arXiv preprint MiXiv:2407.09433 2024.
[33] Y. Qin, C. Qian, X. Han, Y. Lin, H. Wang, R. Xie, Z. Liu, M. Sun, and J. Zhou, "Recyclable tuning for continual pre-training," arXiv preprint arXiv:2305.08702, 2023.
[34] R. Shahroz, P. Li, S. Yun, Z. Wang, S. Nirjon, C.-W. Wong, and T. Chen, "PortLLM: Personalizing evolving large language models with training-free and portable model patches," in The Thirleenth International Conference on Learning Representations, 2025. [Online]. Available: http://pweeeeeeee.ee////mmddeeee
[35] K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,"" in Proceedings of the IEFE international conference on computer vision, 2015, pp. 1026-1034.
12

---

[36] T. Nguyen, M. Raghu, and S. Kornblith, "Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth." " in International Conference on Learning Representations, 2021. [Online]. Available: http:://peneiiiieeeeeeo///ttaaaa
[37] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, "Similarity of neural network representations revisited," in International conference machine learning. PMLR, 2019, pp. 35 1519-3529.
[38] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt, "Feature selection via dependence maximization," The Journal of Machine Learning Research, vol. 13, pp. 1393-1434, 2012.
[39] M. Yamada, Umezu, K. Fukumizu, and I. Takeuchi, "Post selection inference with kernels," " in International conference on artificial intelligence and statistics. PMLR, 2018, pp. 152-160.
[40] M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in Euro- pean conference on computer vision. Springer, 2014, pp. 818-833.
[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[42] L.. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and Han, "On the variance of the adaptive learning rate and beyond," arXiv preprint arXiv:/9088 90.990000033... 2019.
[43] J. Munkres, "Algorithms for the assignment and transportation problems,"JJuunall Journal of the society for industrial and applied mathematics, vol. 5, no. 1, pp. 32-38, 1957.
[44] X. Li, S. Wang, and Y. Cai, "Tutorial: Complexity analysis of singular value decomposition and its variants," arXiv preprint arXiv: 1906. 12085, 2019.
[45] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, "Boolq: Exploring the surprising difficulty of natural yes/no questions." arXiv preprint arXiv: 10044, 2019.
[46] Y. Bisk, R. Zellers, J. Gao, Y. Choi et al., "Piqa: Reasoning about physical commonsense in natural language," in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 05, 2020, pp. 7432-7439.
[47] M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi, "Socialiqa: Commonsense reasoning about social interactions," " arXiv preprint arXiv: (00::::99000777 2019.
[48] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, "Hellaswag: Can a machine really finish your sentence?" arXiv preprint arXiv:1905.07830 00700000000000....0 2019.
[49] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, "Winogrande: An adversarial winograd schema challenge at scale," Communications of the ACM, vol. 64, no. 9, pp. 99-106, 2021.
[50] Clark, I. Cowhey, O. Etzioni. T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, "Think you have solved question answering? try arc, the ai2 reasoning challenge," arXiv preprint arXiv: 0803005557. 2018
[51] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, "Can a suit of armor conduct electricity? a new dataset for open book question answering," arXiv preprint arXiv:1809.0278) 2018.
[52] W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, "Program induction by rationale generation: Learning to solve and explain algebraic word problems," arXiv preprint 1705v 04 146, 2017.
[53] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano al., "Training verifiers to solve math word problems," arXiv preprint arXiv:2110.14168 2021.
[54] R. Kedzio-ke S. Roy, A. Amini, N. Kushman, and H. Hajishirzi, "Mawps: A math word problem repository," in Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, 2016, pp. 1152-1157.
13

---

[55] A. Patel, S. Bhattamishra, and N. Goyal, "Are nip models really able to solve simple math word problems?" arXiv preprint arXiv:2003.07191 2021.
[56] Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, Bing, and S. Poria, "LIm-adapters: An adapter family for parameter-efficien fine-tuning of large language models," arXiv preprint arXiv:2304.0193. 2023.
[57] Wang, Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, F. Huang, and H. Chen, "Wise: Rethinking the knowledge memory for lifelong model editing of large language models," Advances in Neural Information Processing Systems, vol. 37, pp. 53 764-53 797, 2024.
[58] Y. Yao, N. Zhang, Z. Xi, M. Wang, Z. Xu, S. Deng, and H. Chen, "Knowledge circuits in pretrained transformers," Advances Neural Information Processing Systems, vol. 37, pp. 118-71-118 602, 2024.
[59] Q. Sun, M. Pickett, A. K. Nain, and I.. Jones, "Transformer layers as painters." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 24, 2025, pp. 25 219-255 227.
[60] A. Morcos, M.. Raghu, and S. Bengio. "Insights on representational similarity in neural networks with canonical correlation," Advances in neural information processing systems, vol. 31, 2018.
[61] L. Yu, B. Yu, H. Yu, Huang, and Y. Li, "Language models are super mario: Absorbing abilities from homologous models as a free lunch," in Forty-firsi International Conference on Machine Learning, 2024.
[62] Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, "Resolving interference when merging models," arXiv preprint arXiv:2306.00 0106600/7000 0778, vol. 2023.
14

---

Algorithm 3 Hungarian Algorithm
Require: Cost matrix C of size
Ensure: Optimal assignment permutation
# Phase I: Matrix Preparation
1: Pad matrix with zeros to make it square (n +- max(m, n))
2: C < resulting
matrix
# Phase 2: Row/Column Reduction
3: for each row do
4: Tmin f min m (C(i,
5: Clè.
Clù, Tmin
6: end for
7: for each column do
8: Cmin f -min(Cll:.]))
9: Cl.. Cl:,J] Cmin
10: end for
Phase 3: Initial Matching
11: Find maximum matching M using zeros
if matching M is perfect then
13: return M
14: end if
# Phase 4: Iterative Adjustment
15: repeat
16: Cover all zeros with minimum lines
17: & + number of covering lines
18: if then
return current matching 20: end if
19:
21: 8 + min min(C[i,.]] uncovered elements]
for all uncovered elements Cli,, do
23:
Cli.j] + C'i,,]l -
24:
end for
25:
for all doubly covered elements Cii,j] do
Cli - +
27:
end for
28:
Update matching M
29:
until perfect matching found
A Appendix
A.1 Details of Hungarian Algorithm
The Hungarian algorithm, also known as the Kuhn-Munkres algorithm, combinatorial optimization technique for solving the assignment problem. guarantees find the optimal one-to-one assignment that minimizes the total cost bipartite graph. As shown in the above algorithm, the procedure mainly consists of four phases: (1) Matrix Preparation: Convert the cost matrix square matrix by zero-padding if necessary. (2) Row/Column Reduction: Subtract the minimum value of each row from its elements, then repeat for columns, creating at least one zero per row and column. (3) Initial Matching: Identify a maximum matching using zero-cost entries. If a perfect match is found, the algorithm terminates. (4) Iterative Adjustment: unmatched, iteratively (4-a) cover zeros with minimal lines, (4-b) compute the smallest uncovered element 8, (4-c) adjust the matrix by subtracting from uncovered elements and adding to doubly-covered elements, and (4-d) update the matching. This loop continues until the number of covering lines equals the matrix dimension, guaranteeing an optimal assignment. The algorithm operates O(n²) time 43 where n the number of nodes in the smaller partition.
15

---

A.2 Hyperparameters
Tables annd detail the hyperparameters of LoRA and LoRASuite for commonsense math tasks, respectively.
Table 8: Hyperparameter configurations of LoRA for models on the commonsense reasoning tasks.
Table Hyperparameter configurations of LoRA for models on the mathematical tasks.
Hyperparameters LoRA
LoRASuite
Hyperparameters
LoRA
LoRASuite
Rank a Dropout Optimizer LR
32
Rank
32
32
Dropout Optimizer
O
A
Adam
3c-4
lc-3
LR
3e-4
le-3
LR Scheduler
Linear
LR Scheduler
Linear
Batch Size
16
Batch Size
16
Warmup Ratio
0.1
Warmup Steps
100
Epochs
Epochs
3
Target Module
Q.K.V.O
Target Module
Q.K.V.O
A.3 Different Target Modules
Figures and present the performance of LoRASuite using q_proj, k_proj, v_proj, o_proj. upoproj. and down_ proj target modules. As shown, the performance improvement on the math dataset becomes more pronounced as the number of target modules increases. likely explanation that the commonsense corpus significantly overlaps with the LLM pre-training dataset; consequently, LoRASuite's additional scall fine-tuning is more impactful for mathematical tasks.
Vanila
LoRASute W/ LFT
LoRA (5mail)
LoRASuite
LoRA (FJII
Aug. Accuracy (99)
20,0
44
Aorg Accurasy
30
Avg. Accuracy
Accuracy
30
20
2127
24
18
20
10-
Blom Nloom-1811
1ame---T11 Lama-1-88
Gwees1 5-1 Oweno 5-18
MINEPN-93
Figure 5: Performance comparison on math tasks for different LLMs with up up_proj and down_ __ojj as target modules.
Vanila
LORA (5mai))
LoRASultc
LoRA (FJl]
40
60
12
Avg. Accuracy (%))
2L22
Avg. Accuracy eary (1%)
Avg. Accuracy (96)
32.11
30
40
80-
L4:a
16
20
Avg.
Blomm 560m Stomm- 101
Liama-2-70 Liama-3-00
Gwen-13
Owen-2.5.11 2-3-0
Mi-XC7M---18 MinCPN-23
Figure 6: Performance comparison on commonsense tasks for different LLMs with up_proj and down_prooj as target modules.
A.4 Heatmap of Layer Similarity
Figures 7 to 10 illustrate the CKA layer similarities of various LLM backbones before and after upgrading. The similarity patterns vary significantly across models, with MiniCPM and Pythia showing the highest variance. Notably, all models exhibit a clear block structure, similar to the three distinct representation spaces "beginnn".ii "middle," and "end" described in [59].
16

---

30
30
0.30
MiniC 28
25
0.90
20
0.
0.97
15
05
15
0.36
10
10
0.95
5
5-
0.34
0
10
20
30
40
50
MiniCPM-S-18
10
20
30
40
Yi-1.5-98
Figure Heatmap CKA layer similarity Figure 8: Heatmap CKA layer similarity be-
between MiniCPM-S-IB and MiniCPM-2B.
tween Yi-6B and Yi-1.5-9B.
20
12.5
pythia-1b
10.0
7.5
10
0.96
2.5
02
0.9
0.0
5
10
15
20
0
0.94
pythla-1.4b
10
20
30
Qwen2.5-3B
Figure Heatmap CKA layer similarity Figure 10: Heatmap of CKA layer similarity be- between pythia-1b 1b and pythia-1.44
tween Qwenl 1.888B and Qwen2.5-3B.
A.5
Detailed Results for Different LLM Backbones
Tables T0
19 provide the raw data underlying the aggregated results in Figures and
Table 10: Performance math tasks when LLM upgrades from Yi-6B to Yi-1.5-9B. The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
AddSub
MultiArith
SingleEq
GSMBK
AQuA
MAWPS
SVAMP
Avg.
48.86
49.02
17.21
13.78
39.08
37.9
38.41
Y1-6B
LoRA (10K)
68.61
72.05
28.51
22.44
61.76
44.9
56.04
78.48
66.67
82.87
47.99
41.34
78.57
65.1
65.86
LoRASuite LFT
90.89
88.98
52.01
38.58
86.55
78
73.07
Yi-1.5-9B
LoRA (100)
82.78
70.5
86.02
54.66
68.9
LoRASuite LFT (100)
86.58
91.33
89.76
66.26
30.31
84.45
75.1
74.83
LoRA (10k)
86.84
95.33
91.14
68.54
32.28
85.29
74.8
76.32
Table 11: Performance commonsense tasks when LLM upgrades from Yi-6B Yi-1.5-9B, The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-c
ARC-e
OBQA
Avg.
64.25
36.51
14.94
17.63
4.66
16.13
16.5
17.6
23.53
Y1-6B
LoRA (10k)
69.33
37.92
64.43
52.64
70.72
41.55
51.01
57.40
55.63
61.13
76.39
63.2
66.13
17.36
58.11
66.54
45.4
56.78
LoRASuite w/o
56.94
73.78
78.41
77.56
89.48
79.20
74,29
Yi-1.5-99
LoRA (100)
75.90
65.05
60.13
73.98
80.05
68.16
LoRASuite LFT (1OD)
69.30
72.09
72.82
48.98
73.09
73.55
86.78
70.70
LoRA (10k)
71.71
81.77
78.20
90.69
82.48
84.04
92.34
83.8
83.13

---

Table 12: Performance on math tasks when LLM upgrades from Pythia- IB to Pythia-I.4B. The number in parentheses represents the scale fine-tuning datasets.
Base Model
PEFT
AddSub
MultiArith
SingleFq
GSM8K
AQuA
MAWPS
SVAMP
Avg.
1.01
2.50
0.98
0.83
21.26
0.84
1.1
4.07
Pythia-1B
LoRA (10k)
12.66
19.50
12.40
1.97
14.57
7.56
8.30
10.99
1.52
2.67
0.39
1.14
20.47
0
0.9
3.87
LoRASuite w/o LFT
0.76
2.33
1.18
0.53
21.65
0
0.8
3.89
Pythia-1.4B
LoRA (IK)
2.03
4.83
3.15
1.52
12.6
0.84
2.1
3.87
LoRASuite w LFT (Ik)
10.63
15
9.65
1.82
15.75
7.56
6.2
9.52
LoRA (1Ok)
27.85
43.67
25.39
3.56
20.87
21.43
14.4
22.45
Table 13: Performance on commonsense tasks when LLM upgrades from Pythia-1B Pythia-
The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-e
ARC-e OBQA
Avg.
54.19
15.56
3.53
8.22
26.05
4.61
4.08
15.41
Pythia-18
LoRA (1OK)
61.65
33.08
34.03
25.06
49.49
23.29
24.62
28.4
34.95
53.82
11.32
14.48
16.22
22.65
7.51
9.93
10.2
18.27
LoRASuite w/o LFT
51.16
11.48
13.82
14.91
14.29
8.79
9.4
Pythia-1.48
LoRA (100)
57.16
45.76
29.48
48.07
21.50
LoRASuite L.FT (100)
56.06
48.64
32.70
25.05
50.43
22.70
23.40
35,43
LoRA (10K)
48.38
45.1
32.45
24.99
51.54
23.12
25.21
29.4
35.02
Table 14: Performance on math tasks when LLM upgrades from Bloom-560m to Bloomz- IB1. The number parentheses represents the scale fine-tuning datasets.
Base Model
PEFT
AddSub
MultiArith
SingleFy
GSM8K
AQuA
MAWPS
SVAMP
Avg.
0.76
2.17
0.59
0.76
5.91
0.42
0.80
1.63
Bloom-560m
LoRA (10k)
8.35
8.17
7.68
2.27
20.47
5.04
2.10
7.73
1.77
4.67
0.79
2.65
11.81
2.1
3
3.83
LoRASuite w/o LFT
1.77
4.67
0.79
2.5
12.2
2.1
2.9
3.85
Bloomz-1B1
LoRA (Ik)
2.03
2.95
1.97
11.81
4.2
1.7
3.81
LoRASuite w LFT (Ik)
6.84
4.72
1.67
17.72
5.88
3
6.26
LoRA (10K)
17.22
16.67
19.09
3.11
19.29
15.13
11.1
14.52
Table 15: Performance on commonsense tasks when LLM upgrades from Bloom-560m Bloomz- IBI. The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-e ARC-e OBQA
Avg.
6.88
7.32
14.43
2.45
5.46
3.91
4.4
6.67
Bloom-560m
LoRA (10k)
54.86
50.76
33.67
25.23
51.38
24.06
25.38
26.8
36.52
38.65
35.36
0.15
23.96
13.89
0.51
0.34
0.60
14.18
LoRASuite
38.65
23.90
13.97
0.34
0.80
14.26
LoRA (100)
56.06
0.71
31.12
22.21
38.75
21.76
26.4
27.02
LoRASuite LFT (100)
62.17
32.91
25.09
50.83
22.70
25.08
27.60
37.01
LoRA (10k)
60.92
51.51
35.21
24.33
51.07
25.43
24.12
22.4
36.87
18

---

Table 16: Performance on math tasks when LLM upgrades from Llama-2-7B to Llama-3-8B. The
number parentheses represents the scale tne-tuning datasets.
Base Model
PFFT
AddSub
MultiArith
SingleFq
GSMSK
AQuA
MAWPS
SVAMP
Avg.
1.77
1.50
1.77
0.99
21.65
0.84
1.50
4.29
Llama-2-4 2-7B
LoRA (10K)
29.11
42.50
28.15
8.11
11.42
28.57
18.60
23.78
18.73
16.67
19.09
4.09
21.26
19.33
13.3
16.07
LoRASuite w/o LFT
18.99
16.33
18.90
3.18
22,05
16.39
12.70
15.51
Llama-3 BB
LoRA (100)
20.25
20.50
21.46
7.05
21.65
15.97
14.90
17.40
LoRASuite w LFT (100)
45.82
70
46.85
17.74
20.47
44.96
35.4
40.18
LoRA (10k)
55.70
85
56.69
23.35
23.62
53.78
48.50
49.52
Table 17: Performance commonsense tasks when LLM upgrades from Llama-2-7B to Llama-3-8B.
The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-c
ARC-e
OBQA
Avg.
56.21
14.68
5.55
7.11
10.60
17.82
Llama-2-7B
LoRA (10k)
63.18
25.46
42.63
31.60
58.72
20.82
27.95
32.40
37.85
51.44
45.81
34.34
23.09
12.47
27.22
29.71
26.60
31.34
LoRASuite w/o LFT
52.26
45.97
33.67
23.07
12.94
27.05
30.13
26
31.39
Llama-3-8B
LoRA (100)
62.29
48.59
42.78
29.25
51.93
35.67
43.86
43.4
44.72
LoRASuite w LFT (100)
62.39
57.67
45.80
26.76
54.54
35.32
45.12
42.40
46.25
LoRA (10k)
72.97
50.22
51.54
37.3
72.14
31.48
34.60
41
48.91
Table 18: Performance math tasks when LLM upgrades from Dwen-11.-1.8D to Qwen-2.5-3B.
The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
AddSub
MultiArith
SingleEq
GSMSK
AQuA
MAWPS
SVAMP
Avg.
32.91
$1.33
$2.56
9.70
11.02
47.06
32.70
Quen-1.5-1.80
LoRA (10k)
52.91
79.67
59.25
12.74
14.17
55.04
33.50
43.90
22.53
18.33
13.39
5.91
28.35
7.98
15.90
16.06
LoRASuite w/o LFF
23.29
21.50
13.98
5.84
25.59
6.30
15.10
15.94
Qwen-2.5-33
LoRA (100)
30.13
25.83
15.55
8.19
26.77
13.45
15.40
19.33
LoRASuite LFF (100)
53.16
82
66.73
44.50
37,40
57.98
56.40
56.88
LoRA (10k)
42.28
88.67
62.40
32.90
24.80
44.96
56.20
50.32
Table 19: Performance commonsense tasks when LLM upgrades from Swen--1.51..0 to Qwen- 2.5-3B. The number in parentheses represents the scale of fine-tuning datasets.
Base Model
PEFT
BoolQ
PIQA
SIQA
HellaSwag
WinoG
ARC-c
ARC-e
OBQA
Avg.
51.16
43.42
18.17
24.13
18.55
20.14
22.01
23.40
27.62
Quen-1.5. 1.5-1.88
LoRA (10k)
52.75
54.52
58.55
32.48
57.14
37.46
49.66
51.80
48.30
65.17
57.89
43.81
36.69
52.72
36.18
40.07
40.60
46.64
LoRASuite w/o
57.40
36.19
51.30
36.35
40.36
40.60
46.42
Quen
LoRA (100)
28.10
56.91
47.29
38.87
50.04
37.45
43.81
39.20
42.71
LoRASuite LFT (100)
63.09
58.27
60.70
33.73
54.46
49.66
61.36
54
54.41
LoRA (10k)
60.31
59.96
$9.21
46.93
63.93
49.66
56.36
54
56.30
19

---

A.6 Ablation Studies
LoRASulte
First
XXX
Last
Prorusiss IFC
PADCA
Accuracy
40
42:80
+1.00
Aug. Accuracy (56)
Accuracy (9)
40
02.30
43.00
41.04
4:00
Avg. Accuracy (%)
20-
20
20
Mach lunks
Commerserttv lanks
Moth Tocks (6]
Commonsense Tacks
Figure 11: Performance comparison with dif- ferent layer mapping algorithms when up- grading the LLM from MiniCPM-S-IIB to MiniCPM-2B.
Figure 12: Performance comparison with
different representational similarity methods when upgrading the LLM from MiniCPM-S- IB to MiniCPM-2B
Aigstr
80
$1:03
Avg Accuracy (56)
60
Avg. Accuracy (%)
7L.11
Accuracy
40
Accuracy
40
14:5
60
40
20
20
-20
Mach Tasiss
Commerssenee Taaks
1.5 Qw5n2-1.8
Uama
2-713
Figure 13: Performance comparison with different attention head mapping algorithms when upgrading the LLM from MiniCPM-S- IB MiniCPM-2B.
Figure 14: Performance math tasks
when LLM upgrades from Qwen-1.5-1..8 to Qwen2-1.5B and from Alama-2--13b-f to Llama-3.1-70B.
CKA-based Layer Mapping. Figure 11 compares the performance of different layer mapping algorithms: "LoRASuite" refers to our CKA- method described in Section 3.1 "First" maps layers sequentially from beginning to end; "Medium" spreads mappings outward from the middle; and "Last" maps from end to beginning. Our CKA-based approach outperforms the others on both math and commonsense tasks, demonstrating the effectiveness of using representational similarity to guide layer mapping.
Figure 12 further evaluates different representational similarity metries. Canonical Correlation Analy- sis (CCA) (37] identifies linear projections that maximize correlation between two sets of activations. Projection Weighted CCA (PWCCA) 600 extends CCA by weighting individual canonical correla- tions based on their importance. Procrustes analysis seeks the optimal orthogonal transformation that minimizes the Frobenius norm between two matrices. Our minibatch CKA-based method achieves superior performance across both task types, demonstrating the advantage of using minibatch CKA for guiding layer mapping.
Attentio-hhead Mapping. Figure [3 compares the performance of different attention head mapping algorithms. "LoRASuite" refers to our Hungarian-basee method described in Section 3.] while "Lo- RASuite w/o Head Mapping" uses the default ordering without enforcing one-to-one correspondence based cosine similarity, and the "Direct Algebra" method calculates the new LoRA weights by AW, = W. W. + AW, + where is just learnable set of weights need to fine-tune. Our approach consistently outperforms the baseline on both math and commonsense tasks. highlighting the effectiveness of using inteeedndeenntntt interaction matrices and similarity- -bsed head mapping. For the direct algebra method, the performance gap, especially on commonsense tasks, suggests that directly aligning to the trained W, + AW, may discard valuable information encoded in the new weight W,, through its pretraining and post-training. As result, even with further lightweight fine-tuning, this method struggles to fully leverage the potential of the stronger target model W
20

---

A.7 Loss Convergence
1.4
LoRA (100)
LoRASuite with (100)
1.2
Training Loss
1.0
0.8
0.6
5
10
15
Steps
Figure 15: Step-wise training loss comparison between "LoRA (100)" and "LoRASuite with LFT (100)".
In Figure 15 visualize the step-wise training loss for "LoRA (100)" and "LoRASuite with LFT (100)" under the settings of batch size = 16 and micro batch size = for MiniCPM-2B. As mentioned in the Section 3.2] the default linear learning-rate scheduler of vanilla LoRA (100) with warm-up phase to stabilize training from random initialization. However, with limited data, a considerable steps is spent in the warm-up stage, which slows convergence and limits performance. In contrast, LoRASuite e starts with transformed parameters already close to a good solution, allowing us to omit warm-up and use a higher learning rate to accelerate convergence. This leads better performance, especially in low-data regimes.
Table 20: Performance comparison for merged LoRA weights.
Base Model
PEFT Module
Avg. Math Accuracy
Avg. Common Accuracy
MiniCPM-S-IB MiniCPM-2B
TIES merged LoRA
17.45 23.90 (1.37x)
11.53 22.57 (1.96x)
TeES-merged Lonnoouieetransfommeed LoRA
A.8 Merged LoRA Adaptation
Prior studies, such as DARE [6]] and TIES [62]. have demonstrated the feasibility and effectiveness of merging task-specific LoRA modules. In this section, we evaluate whether LoRASu te maintains its effectiveness when LoRA weights are merged across different tasks.
merged MiniCPM-S-IBBs math and commonsense LoRA adapters using TIES with equal weights (0.5 each) and applied the same merging procedure to the LoRASuite- transformed adapters of MiniCPM-2B. Compared with the simple average of individual task perfor- mances, the merged LoR Auutte--ransffommee adapters improved math and commonsense scores by 1.37x and 1. 1.96x, respectively, demonstrating that our method remains effective even when merging weights from distinct tasks.
As shown in Table 20
A.9 More LLM Backbones
Figure 14 (a) presents the performance math tasks when upgrading LLM backbones from Qwen- 1.5-1.8B to Qwen2-1.551 covering five major architectural changes: hidden size, intermediate size, layer depth, head number, and attention type. Similarly, Figure [4] reports results for the upgrade from Llama-2-13B to Llama-3.1 1oma--3.1-70B. 1-70B, encompassing all six key architectural changes. In both cases, LoRASuite remains consistently effective.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contribution is outlined as a separate paragraph in $
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are outlined as a separate paragraph in $
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We have provided the time complexity analysis for the algorithms involved in 3.3 and SA..I] respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Consistent with previous studies, our evaluation employs the widely recognized open-source benchmark 566
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use the same open-source dataset as [56 and our code is available in https: http::://gitubb.. comm/\anaaL111/ooAASttee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides/ wwwe://oooooeioooeoooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results, See the NeurIPS code and data submission guidelines (https
//nip.cc. wttp:ccccppcccccceeeeec gwwdguiddeeeee /eodedeee/ooeebbiieeeeee ielbbmmsssionPee ssionpoiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed instructions on how to reproduce the main experimental
in $
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We primarily report average performance metrics rather than error bars in our experiments. This choice aligns with prior studies that also emphasize mean performance. Additionally, due to significant variability across different datasets, reporting deviations comparable in magnitude to the averages would not yield meaningful insights. Neverthe- less, we provide comprehensive data, enabling interested readers to compute error bars independently.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide detailed hardware information in
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and believe that our research conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Justification: In S we discussed real-world examples illustrating how our approach can promisingly mitigate the negative impacts associated with the current status quo, including excessive time consumption, high costs, and environmental harm.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper is intended for LoRa adaptation and does not involve high-risk data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have properly cited the original code, data, and models in $
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [No]
Justification: We will provide detailed documentation for the new assets upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects; therefore, Institutional Review Board (IRB) approval is not applicable.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "aeanrhaCVj", "reviewer": "Reviewer_Y2Ki", "summary": "This paper proposes a novel method to leverage existing LoRA weights for adaptation in the updated models. The core idea assumes that we have access to both old and newer weights of a specified model and therefore we can calculate a transfer matrix which we can then utilize to adapt the parameters. They use the Centered Kernel Alignment (CKA) method for layer mapping and the Hungarian algorithm (aka the Kuhn-Munkres algorithm) for attention head mapping. The CKA sequentially aligns corresponding layers to maximize the total similarity, and the Hungarian algorithm", "questions": "- Why in figure 4-b, the change of learning rate has a significant impact on the performance of LoRASuite? What’s the justification?\n\n- Why in Algorithm 2, the input to the cka_layer_mapping() is S? S has not been introduced properly.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n- The paper is well-written and it is easy to understand. The application is technically sound and potentially useful as it can reduce the training costs.\n- The method is innovative and tries to tackle an interesting problem.\n- The experiments are informative and it can potentially help future research for solving this issue.\n\nWeakness:\n- The LoRASuite is an expensive method (in the order of cubic) which might limit the applicability of this method. \n- It has not been clearly explained why they used different methods for head and layer mapping.\n- As shown in table 4 and 5, the performance of LoRASuite is not consistent across different datasets.\n- Their method doesn’t seem to be applicable to customized architectures.", "was_revised": true, "final_justification": "In general, this paper is trying to solve a novel problem, the proposed method can be potentially useful and the paper is technically sound, and well-written. The authors also shows some level of competence during the rebuttal and that increased my trust to their results. Despite, the limitation of the experiment in the sense that they have strong assumptions about the architecture and availability of the model weights, I would like to increase my score as I still believe this might initiate exciting future research directions.", "rebuttal": "***Q1: The LoRASuite is an expensive method (in the order of cubic) which might limit the applicability of this method.***\n\nA1: Thank you for your valuable comment. While LoRASuite involves algorithms with cubic complexity, the practical cost remains manageable. This is because the number of layers and attention heads in mainstream LLMs typically remains under 100, keeping the actual computation well within the capabilities of modern processors. \n\n---\n\n***Q2: It has not been clearly explained why they used different methods for head and layer mapping.***\n\nA2: Thank you for your thoughtful comment. We use different methods for layer and head mapping based on the distinct structural roles they play. For layer mapping, we adopt a dynamic programming approach inspired by prior studies in image classification with CNNs, where different layers capture different levels of abstraction, and higher layers build upon the representations of lower layers. This motivates a sequential alignment strategy. In contrast, attention heads within the same layer are parallel and independent, so we treat head mapping as a bipartite matching problem and use the Hungarian algorithm to find the optimal one-to-one correspondence. We will clarify this design choice in the revised manuscript.\n\n___\n\n***Q3: As shown in table 4 and 5, the performance of LoRASuite is not consistent across different datasets.***\n\nA3: Thank you for your observation. The performance variation of LoRASuite across different datasets can be attributed not only to the task differences but also to the underlying pretraining data of both the original and upgraded models. If the pretraining corpora differ significantly, the representations learned by the models may diverge, which can affect the compatibility and effectiveness of the transferred LoRA weights. We will clarify this important factor in the revised manuscript.\n\n---\n***Q4: Their method doesn’t seem to be applicable to customized architectures.***\n\nA4: Thank you for your feedback. Our current evaluations focus on mainstream LLMs to ensure broad applicability and reproducibility. However, LoRASuite is also applicable to customized architectures, as long as the original and upgraded models share a similar structural design—for example, when both follow the Transformer paradigm. In such cases, the core idea of layer mapping and attention head mapping remains valid. \n\n---\n\n***Q5:  Why in figure 4-b, the change of learning rate has a significant impact on the performance of LoRASuite? What’s the justification?***\n\nA5: Thank you for your insightful feedback. Unlike LoRA (Small), which employs a linear learning-rate scheduler with a warm-up stage to stabilize optimization from random initialization, LoRASuite starts with transformed parameters that are already close to a good solution. Hence, we omit the warm-up phase. However, this initialization results in a steeper optimization landscape, where a small learning rate combined with limited data can cause the model to underfit or fail to converge effectively. We will clarify this in the revised manuscript.\n\n---\n\n***Q6: Why in Algorithm 2, the input to the cka_layer_mapping() is S? S has not been introduced properly.***\n\nA6: Thank you for your feedback. As described in Line 175, the matrix $S$ is defined such that each element $S_{ij}$ represents the CKA similarity between the $i$-th layer of the original model and the $j$-th layer of the upgraded model. We will clarify the introduction of $S$ in Algorithm 2 in our revised manuscript."}, {"review_id": "RcbGOdRAty", "reviewer": "Reviewer_obTS", "summary": "The paper proposes LoRASuite, a modular approach tailored specifically to various types of LLM updates. It identifies structural differences (e.g., vocabulary size, hidden size, attention heads) between old and new models and adapts existing LoRA weights accordingly through transfer matrices, layer/head mapping (via CKA and the Hungarian algorithm), and a lightweight fine-tuning (LFT) step. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods and full-scale\nLoRA retraining, while LoRASuite significantly reduces memory con-\nsumption by 5.5 GB and computational time by 78.23%.", "questions": "- Why LoRASuite is much more sensitive to the learning rate compared to vanilla LoRA (fig4-b)?\n- How do you set the maximum offset constraint in Algo 1?\n- What happens if the optimal head mapping is not one-to-one, i.e., the information could be distributed across or aggregated from multiple heads)?\n- What happens if the tokenizer or embedding changes significantly?\n- What could be some potential solutions for implicit upgrades, such as changes in pre-training datasets and post-training methods?\n- Does the LoRASuite still work when the LoRA weights are for different tasks?\n- Curious whether there are relatively different sensitivity when matching various factors, e.g., is layer matching more important/sensitive than head mapping?", "limitations": "See my questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper studies an interesting yet practically important and under-explored problem of adapting LoRA weights during LLM upgrades.\n- The paper systematically explores the problem by explicitly identifying the potential changed dimensions (e.g., vocabulary size, hidden size, attention heads)\n- Extensive experiments across multiple model upgrades, task types (math, commonsense) and thorough baselines demonstrates strong empirical results. Extensive ablations on layer and head mapping.\n\nWeaknesses:\nSee my questions.", "was_revised": true, "final_justification": "The research problem is novel and interesting, and the paper presents a systematic exploration with strong empirical results. The rebuttal largely addressed my concerns. The main limitation lies in the assumption for model upgrades—similar architecture that enables the proposed transfer matrix, and that model weights are available. More practical scenarios include implicit upgrade, and finer-grained information mapping across attention heads — making mapping more challenging and expensive (see my comments). These could limit the method to generalize to more other real-world settings. But overall the paper is interesting. So I keep my score of a \"Borderline Accept\".", "rebuttal": "***Q1: Why is LoRASuite much more sensitive to the learning rate compared to vanilla LoRA (fig4-b)?***\n\nA1: Thank you for your insightful feedback. Unlike LoRA (Small), which employs a linear learning-rate scheduler with a warm-up stage to stabilize optimization from random initialization, LoRASuite starts with transformed parameters that are already close to a good solution. Hence, we omit the warm-up phase. However, this initialization results in a steeper optimization landscape, where a small learning rate combined with limited data can cause the model to underfit or fail to converge effectively. We will clarify this in the revised manuscript.\n\n---\n\n***Q2: How do you set the maximum offset constraint in Algo 1?.***\n\nA2: Thank you for your feedback. The maximum offset constraint in Algorithm 1 is set to the absolute difference in layer counts between the original and upgraded models. We will clarify this detail in the revised manuscript.\n\n---\n\n***Q3: What happens if the optimal head mapping is not one-to-one, i.e., the information could be distributed across or aggregated from multiple heads)?***\n\nA3: Thank you for your insightful feedback. Indeed, as you pointed out, information being distributed across or aggregated from multiple heads is a valid scenario. In such cases, one would need to compute similarity scores between all attention heads across every layer of the original and upgraded models, distribute information from each attention head in the original model to multiple heads in the upgraded model, and aggregate these mappings accordingly. However, this comprehensive method would have a time complexity of approximately $O((n_{layer} \\times n_{head} )^3)$ due to the cubic complexity of the Hungarian algorithm. Hence, our current method can be viewed as a practical simplification to mitigate computational overhead.\n\n---\n\n***Q4: What happens if the tokenizer or embedding changes significantly?***\n\nA4: Thank you for this valuable question. As described in Line 130, we manually calculate the transformation matrix using shared tokens between models. However, if the tokenizer or embedding changes significantly, additional alignment techniques—such as supervised or unsupervised embedding alignment methods [1], or entity alignment methods from graph domains [2]—would be required before applying our proposed method. Investigating these alignment strategies is an interesting direction for future research, and we will clarify this in our revised manuscript.\n\n[1] (EMNLP’2018) Gromov-Wasserstein Alignment of Word Embedding Spaces.\n\n[2] (NIPS’24) Entity Alignment with Noisy Annotations from Large Language Models.\n\n---\n\n***Q5: What could be some potential solutions for implicit upgrades, such as changes in pre-training datasets and post-training methods?***\n\nA5: Thank you for your insightful feedback. For implicit upgrades involving changes in pre-training datasets or post-training methods, prior studies on knowledge editing [1, 2, 3, 4] provide promising directions. Integrating knowledge-editing techniques with our proposed methods would be an interesting area for future research. We will clarify this in our revised manuscript.\n\n[1]  (NIPS’22) Locating and editing factual associations in gpt.\n\n[2] (ICLR’23) Massediting memory in a transformer. \n\n[3] (NIPS’24) WISE: Rethinking the knowledge memory for lifelong model editing of large language models.\n\n[4] (NIPS’24) Knowledge circuits in pretrained transformers.\n\n---\n\n***Q6: Does the LoRASuite still work when the LoRA weights are for different tasks?***\n\nA6: Thank you for this insightful question. LoRASuite is designed to adapt LoRA weights across model upgrades and remains effective even when the weights are trained on different tasks. Prior studies such as DARE and TIES have shown that merging task-specific LoRA modules is both feasible and effective. In our case, each task-specific LoRA can first be transformed using LoRASuite, and then merged using established methods like TIES to ensure compatibility with the upgraded model while preserving task-specific knowledge.\n\nFor example, we merged the MiniCPM-S-1B math and commonsense LoRA adapters using TIES with equal weights (0.5 each), and performed the same merging on the LoRASuite-transformed adapters for MiniCPM-2B. Compared to the simple average of individual task performances, the merged LoRASuite-transformed adapters improved math and commonsense scores by 1.37× and 1.96×, respectively. We will include these results in the revised manuscript.\n\n| PEFT Module    | Avg. performance on Math Tasks |  Avg. performance on Commonsense Tasks |\n| :-------- | :-------: |:-------: |\n| TIES-merged  LoRA on MiniCPM-S-1B  | 17.45    | 11.53    |\n| TIES-merged LoRASuite-transformed LoRA on MiniCPM-2B | 23.90 (1.37×)    |22.57 (1.96×)    |\n\n---\n\n***Q7: Curious whether there are relatively different sensitivity when matching various factors, e.g., is layer matching more important/sensitive than head mapping?***\n\nA7: Thank you for your insightful feedback. In Appendix 6.1, we conducted ablation studies on both layer mapping and head mapping. Based on the results shown in Figures 11 and 12, we observed that layer mapping has a significantly greater influence on the average performance compared to head mapping. We will clarify this observation in our revised manuscript."}, {"review_id": "3c7ZhDfXq9", "reviewer": "Reviewer_Mxt5", "summary": "This paper studies the problem of adapting LoRA adapters for large language models (LLMs) when upgrading to new model versions that differ in architecture or parameters. The authors identify six types of incompatibilities that may arise during such upgrades. To address these, they propose LoRASuite, a modular framework that applies a sequence of techniques, including dimensionality transfer, layer and attention head alignment, and a brief fine-tuning step, to transfer LoRA adapters from an older LLM to a newer one. The method is evaluated on several backbone LLM families (like Llama, Yi, MiniCPM, Qwen, etc.) and two types of tasks, reporting improved results on adapter transfer efficiency, compute savings, and downstream performance than standard LoRA retraining.", "questions": "1. Please address the weaknesses mentioned before.\n2. line 292: what does \"full-scale LoRA retraining\" mean? Is it the same as \"LoRA (10k)\" in Table 4/5? Same for \"LoRA (small)\". Please define it in the paper explicitly and keep it consistent throughout the paper.\n3. Why did you choose to upgrade Qwen1.5-1.8B to Qwen2.5-3B, while skipping Qwen2-1.5B? It could be interesting to see a comparative study on the performance of LoRASuite when upgrading to Qwen2-1.5B vs. Qwen2.5-3B, as the former is a more direct upgrade in terms of model size and architecture.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The paper provides a systematic and detailed analysis of six specific incompatibility types, such as changes in vocabulary size, hidden dimensions, and attention structure, that arise when transferring LoRA adapters across large language model upgrades.\n2. It introduces a modular framework, LoRASuite, that integrates linear transfer, layer-wise alignment using CKA and dynamic programming, and attention head matching via the Hungarian algorithm, offering a practical and potentially extensible solution for LoRA transfer.\n3. The proposed approach is empirically validated on a diverse set of LLM backbones and on both math and commonsense QA tasks, with improved performance metrics compared to standard LoRA retraining.\n\n\nWeaknesses:\n1. Although the paper proposes an interesting method for transferring LoRA adapters, it is primarily evaluated on small to medium-scale models (up to 9B parameters), whereas its practical value would be much greater for larger models (e.g., 13B to 70B+), where adapter retraining is significantly more resource-intensive. This leaves the impact on truly large-scale LLMs unverified.\n2. The selected models used for evaluation are relatively dated, for example, Llama-3 to Llama-3.1/3.2 and Qwen1.5 to Qwen2/2.5/3. As a result, it remains unclear whether the proposed method can bring notable improvements when applied to more recent or stronger model baselines.\n3. The main advantage of the proposed method, saving compute and improving performance in low-data regimes, is only demonstrated when the fine-tuning dataset is extremely small (e.g., 100 samples), and the benefit diminishes or disappears as the dataset size increases, according to results in Section 4.2. Furthermore, the paper does not describe how the small fine-tuning datasets are selected, nor does it report results averaged over multiple random draws. This omission is problematic given the potential instability and variance in performance when using such small data subsets.", "was_revised": true, "final_justification": "While the authors have acknowledged the concerns and promised additional experiments and clarifications, the major concerns regarding scalability to large/recent models and the statistical robustness of the small-sample adaptation regime remain unresolved at this stage. I would like to keep my original rating.", "rebuttal": "***Q1& Q2: Although the paper proposes an interesting method for transferring LoRA adapters, it is primarily evaluated on small to medium-scale models (up to 9B parameters), whereas its practical value would be much greater for larger models (e.g., 13B to 70B+), where adapter retraining is significantly more resource-intensive. This leaves the impact on truly large-scale LLMs unverified.\nThe selected models used for evaluation are relatively dated, for example, Llama-3 to Llama-3.1/3.2 and Qwen1.5 to Qwen2/2.5/3. As a result, it remains unclear whether the proposed method can bring notable improvements when applied to more recent or stronger model baselines.***\n\nThank you for your feedback. Our initial motivation was primarily aimed at mobile scenarios, where an on-device foundation model supports multiple application-specific LoRA adapters for different downstream tasks (as detailed in [AICore](https://android-developers.googleblog.com/2023/12/a-new-foundation-for-ai-on-android.html)). Thus, we initially focused on small to medium-scale models. We acknowledge your valuable point that applying LoRASuite to larger-scale models (e.g., 13B to 70B+ parameters) would significantly enhance its practical value, especially given the higher resource costs of retraining adapters at this scale. Due to time constraints, we will include additional experiments with more recent and larger models in the revised manuscript to address this concern and highlight the broader applicability of our approach.\n\n---\n\n***Q3: The main advantage of the proposed method, saving compute and improving performance in low-data regimes, is only demonstrated when the fine-tuning dataset is extremely small (e.g., 100 samples), and the benefit diminishes or disappears as the dataset size increases, according to results in Section 4.2. Furthermore, the paper does not describe how the small fine-tuning datasets are selected, nor does it report results averaged over multiple random draws. This omission is problematic given the potential instability and variance in performance when using such small data subsets.***\n\nThank you for your constructive feedback. Since the converted LoRA parameters already encapsulate the knowledge learned by the original model, the purpose of small-scale fine-tuning is merely to help the parameters adapt to the upgraded model. As the dataset size increases, the additional fine-tuning can easily lead to overfitting, which explains the diminishing benefit. Currently, the small fine-tuning datasets are randomly selected. We will explicitly describe the dataset selection process in the revised manuscript.\n\n---\n\n***Q4: line 292: what does \"full-scale LoRA retraining\" mean? Is it the same as \"LoRA (10k)\" in Table 4/5? Same for \"LoRA (small)\". Please define it in the paper explicitly and keep it consistent throughout the paper.***\n\nThank you for your feedback. \"full-scale LoRA retraining\" refers to the scenario labeled as \"LoRA (10k)\" in Tables 4 and 5. Additionally, as described in Line 293, LoRASuite involves a small-scale fine-tuning step to mitigate potential performance degradation caused by direct matrix multiplication; thus, \"LoRA (small)\" corresponds to the same small-scale fine-tuning labeled as \"LoRA (100)\" in Tables 4 and 5 when upgrading from MiniCPM-S-1B to MiniCPM-2B. We will explicitly define these terms and ensure consistency throughout the revised manuscript.\n\n---\n\n***Q5: Why did you choose to upgrade Qwen1.5-1.8B to Qwen2.5-3B, while skipping Qwen2-1.5B? It could be interesting to see a comparative study on the performance of LoRASuite when upgrading to Qwen2-1.5B vs. Qwen2.5-3B, as the former is a more direct upgrade in terms of model size and architecture.***\n\nThank you for this insightful suggestion. We initially chose the upgrade path from Qwen1.5-1.8B to Qwen2.5-3B to highlight LoRASuite’s effectiveness across intermediate size, layer count, and attention heads. However, we agree that evaluating a more direct upgrade to Qwen2-1.5B offers valuable insights into LoRASuite’s performance under incremental changes. The following table presents the performance on math tasks when upgrading from Qwen1.5-1.8B to Qwen2-1.5B, further confirming the effectiveness of LoRASuite in this setting. We will include this comparative study in the revised manuscript.\n\n| Base Model   | PEFT | AddSub |MultiArith |SingleEq |GSM8K |AQuA |MAWPS |SVAMP |Avg. |\n| :---------------- | :------: | :----: |:----: |:----: |:----: |:----: |:----: |:----: |:----: |\n| Qwen1.5-1.8B       |   -   | 32.91 | 51.33 | 52.56 | 9.7 | 11.02 | 47.06 | 32.7 | 33.90|\n| Qwen1.5-1.8B       |   LoRA (10k)   | 52.91 | 79.67 | 59.25 | 12.74 | 14.17 | 55.04 | 33.5 | 43.90 | \n| Qwen2-1.5B       |   -   | 46.58 |72 |42.32 |19.86 |13.78 |33.61 |25.3 |36.21|\n| Qwen2-1.5B       |   LoRASuite w/o LFT   | 45.06 |73.67 |45.08 |21 |15.35 |32.35 |26.7 |37.03|\n| Qwen2-1.5B       |  LoRA (100)   | 38.23 | 68.5 | 39.76 | 19.03 | 13.39 | 26.47 | 23.1 | 32.64|\n| Qwen2-1.5B       |   LoRASuite w LFT   | 60.51| 78.83| 72.83| 29.42| 16.93| 57.98| 48.2| 52.10|\n| Qwen2-1.5B       |  LoRA (10k)   | 70.38 | 88.83 | 76.57 | 30.33 | 19.29 | 71.85 | 51.4 | 58.38|"}, {"review_id": "j8EffFcP8f", "reviewer": "Reviewer_yDmS", "summary": "This paper introduces a modular framework for reusing LoRA adapters across newer versions of language models. The idea is novel and has practical relevance. Experimental results demonstrate that LoRASuite achieves better performance than conventional LoRA fine-tuning in low-data regimes.", "questions": "- What are the statistics of the training dataset? \n\n- What is the training dataset for lightweight fine-tuning? \n- The\n\n- Could you present the training curve of the lightweigh for LoRA (100), LoRASuite with LFT (100), and LoRA (10k)?\n\n- Could you show the training parameters size for LoRA (100), LoRASuite w LFT (100), and LoRA (10k)?\n- It is a little weird why LoRASuite w/o LFT and base model get almost the same results. For example, since you use the LoRA adapters from MiniCPM-S-1B. For MiniCPM-2B, the LoRASuite w/o LFT get (23.96) compared to the base (23.85). \n\n- \"Although LoRASuite requires small-scale fine-tuning, this\nresults in a modest memory reduction of 5.5GB,\nprimarily due to the smaller sample size\" \nWhy can a smaller sample size reduce memory?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "## strengths\n\n- How to use LoRA weights in a new model version is a very interesting idea. \n- The paper is well-organized and well-written. \n- The experimental results are comprehensive, especially when the authors compare the results with many backbone models.  \n\n## weaknesses\n\n- Actually, training a LoRA for a new version of a model is not expensive. Although LoRASuite has achieved some improvement compared to the base model, the w/o version still needs some additional training datasets to conduct the lightweight fine-tuning, which is not flexible. \n\n- There is no code released. I suspect the reproduction of the results. \n\n- The LoRAsuit is sensitive to the learning rate. In this case, the experimental results should have more training loss analysis. For example, there should be a training loss figure and model size comparison.", "was_revised": true, "final_justification": "I have read the paper again, and I think the idea is very interesting. The authors address most of my concerns and I hope this paper can be accepted.", "rebuttal": "***Q1: What are the statistics of the training dataset?***\n\nA1: Thank you for your question. To ensure reproducibility, we use the same training and evaluation dataset as [LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters?tab=readme-ov-file), which has over 1.2k stars on GitHub and has been widely adopted in published studies. We will include a detailed description of the dataset statistics in the experimental setup section for clarity and completeness.\n\n---\n***Q2: What is the training dataset for lightweight fine-tuning?***\n\nA2: Thank you for your valuable feedback. The lightweight fine-tuning dataset is currently a randomly selected subset of the dataset used for full-scale LoRA training. We will clarify this in the revised manuscript.\n\n---\n\n***Q3: Could you present the training curve of the lightweigh for LoRA (100), LoRASuite with LFT (100), and LoRA (10k)?***\n\nA3: Thank you for your valuable feedback. As we cannot include additional figures during the rebuttal period, we instead provide the step-wise training loss for LoRA (100) and LoRASuite with LFT (100) under the settings of batch size = 16 and micro batch size = 4.\n\n| Model | Step1 | Step2 | Step3 | Step4 | Step5 | Step6 | Step7 | Step8 | Step9 | Step10 | Step11 | Step12 | Step13 | Step14 | Step15 | Step16 | Step17 | Step18 |\n|------|------|------|------|------|------|------|------|------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| LoRA (100) | 1.210 | 1.199 | 1.081| 1.120 |1.067 | 1.108 | 0.996 | 1.124 | 1.027 | 1.113 | 1.018 | 1.150 | 1.150 | 1.374 | 1.004 | 1.078 | 1.120 | 1.086 | \n| LoRASuite with LFT (100) | 1.140 | 1.028 | 0.921 | 0.914 | 0.786 | 0.859 | 0.955 | 0.738 | 0.680 | 0.702 | 0.683 | 0.709 | 0.749 | 0.675 | 0.625 | 0.601 | 0.760 | 0.663 |\n\n---\n***Q4: Could you show the training parameters size for LoRA (100), LoRASuite w LFT (100), and LoRA (10k)?***\n\nA4: Thank you for your valuable feedback. Under the default setting with rank 32 for MiniCPM-2B, the number of trainable parameters for LoRA (100), LoRASuite with LFT (100), and LoRA (10k) is the same—23.59M, which is approximately 0.78% of all model parameters. We will clarify this in the revised manuscript.\n\n---\n\n***Q5: It is a little weird why LoRASuite w/o LFT and base model get almost the same results. For example, since you use the LoRA adapters from MiniCPM-S-1B. For MiniCPM-2B, the LoRASuite w/o LFT get (23.96) compared to the base (23.85).***\n\nA5: Thank you for your valuable feedback. As noted in Line 268, we believe this phenomenon occurs because the transformed LoRA, when relying solely on matrix multiplication without backpropagation, has limited compatibility with the new model. This is why a small-scale fine-tuning step is necessary. The significant performance improvement achieved after this fine-tuning further supports our assumption.\n\n---\n\n***Q6: Although LoRASuite requires small-scale fine-tuning, this results in a modest memory reduction of 5.5GB, primarily due to the smaller sample size\" Why can a smaller sample size reduce memory?***\n\nA6: Thank you for your valuable feedback. While the core memory footprint of the model—such as parameters, gradients, and optimizer states—is independent of the dataset size, several other factors contribute to increased memory usage when the dataset is larger.\n\nFirst, data pipeline overhead and auxiliary states (e.g., DataLoader prefetching, Trainer’s internal buffers, dataset shuffling, and caching) scale with both the number of samples and training steps. Second, if the full dataset is loaded into memory (e.g., using HuggingFace Datasets `load_dataset(...)` or pre-tokenized and stored as a list), the total number of samples directly affects host DRAM usage. Lastly, for variable-length inputs, larger datasets increase the likelihood of batches containing extremely long sequences, which can trigger excessive padding and dynamic memory allocation, resulting in transient memory spikes."}]

Author rebuttal:

Meta-review: It received ratings of 5,5,4,3. The only review that does not suggest accepting it is concerned about scalability of the method to more recent models. However, the the positive reviews point out that the paper presents a novel approach to adapting pre-trained model weights for updates in new versions of the same model architecture. This is an innovative solution that demonstrates promising results across various backbone models. The paper is well-structured, clearly written, and supported by extensive empirical evidence. While there may be limitations in terms of applicability to all real-world scenarios, such as implicit upgrades or different architectures, the work opens up exciting possibilities for future research in this area. Overall, these factors contribute to a solid justification for accepting the paper.
Decision: Accept (poster)