Title: LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Upgrades; LoRA Adaptation
Abstract: As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates a critical question: "How can we efficiently leverage existing LoRA weights to adapt to newer model versions?" To address this, we propose LoRASuite, a modular approach tailored specifically to various types of LLM updates. First, we compute a transfer matrix utilizing known parameters from both old and new LLMs. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory consumption by 5.5 GB and computational time by 78.23%.

LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades

Abstract

As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates a critical question: "How can we efficiently leverage existing LoRA weights to adapt to newer model versions?" To address this, we propose LoRASuite , a modular approach tailored specifically to various types of LLM updates. First, we compute a transfer matrix utilizing known parameters from both old and new LLMs. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory consumption by 5.5 GB and computational time by 78.23%.

1 Introduction

LoRA [1], a prominent parameter-efficient fine-tuning technique, has garnered significant attention for efficiently adapting pre-trained large language models (LLMs) to specialized downstream tasks using considerably fewer parameters than traditional full-parameter fine-tuning methods. A practical example is a mobile device employing an on-device LLM integrated with multiple app-specific LoRA modules to customize capabilities in mobile agent scenarios [2, 3].

However, frequent updates to LLM backbones, such as the periodic releases of new versions of Llama [4, 5] and Qwen [6, 7], quickly render previously trained LoRA weights obsolete. Consequently, developers face two problematic scenarios: (1) if the corresponding LoRA modules are not promptly updated, app-specific functionalities may degrade or fail; (2) if updated, the prevailing method typically involves retraining LoRA weights from scratch. This process is time-consuming, costly, and even environmentally unsustainable. For instance, experimental results from [8] indicate that fine-tuning a sub-billion-parameter model on a Google Pixel 7 Pro can take several hundred minutes. Similarly, research by [9] estimated that fine-tuning a sparse Mixtral model with two million queries on an NVIDIA H100 GPU costs approximately USD 3,460. Additionally, training a BERT model with 110 million parameters produces about 1,400 pounds of carbon dioxide equivalent-comparable to the emissions from a round-trip transcontinental flight in the United States shiaizhu2-c@my.cityu.edu.hk

for one person [10]. This challenge is expected to escalate as the diversity of large models and the demand for various downstream tasks continue to increase. Thus, a natural question arises:

'How can we leverage the existing LoRA weights to adapt to the latest model version with less effort?'

To address this previously unexplored question, we first categorize the upgrades of mainstream LLMs into six explicit limitations: vocabulary size, hidden size, intermediate dimensions of up/downprojection layers, layer depth, attention head count, and attention type. These limitations hinder the direct reuse of prior LoRA weights. Next, we propose LoRASuite , a modular approach designed to efficiently resolve each of these limitations, in contrast to the current practice of retraining LoRA weights from scratch. For dimensional mismatches, LoRASuite utilizes known parameters from both the old and new LLM versions to compute a transfer matrix, enabling initial adaptation. To address differences in layer depth, we introduce a layer mapping algorithm that maps the LoRA weights from the i -th layer of the original model to the j -th layer of the upgraded model. Specifically, we apply Centered Kernel Alignment (CKA), a representational similarity method, to measure the similarities between layers, and then use dynamic programming to maximize the total sum of CKA similarities. To handle differences in attention head counts, we represent each attention head using input-independent interaction matrices, denoted as W i QK and W i V O . We then introduce a head mapping algorithm based on the Hungarian method, maximizing the aggregate sum of cosine similarities. However, the transformed LoRA parameters primarily result from matrix multiplications rather than backpropagation, which may cause numerical instability and reduced performance. To mitigate this, we introduce a small-scale, skillful fine-tuning stage to ensure the transformed LoRA parameters adapt effectively to the upgraded model while maintaining comparable performance.

We evaluated LoRASuite across multiple tasks, including commonsense reasoning and mathematical benchmarks, using various backbone LLMs. Experimental results show that LoRASuite consistently outperforms small-scale vanilla LoRA. Notably, for backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory usage by 5.5 GB and computational time by 78.23%.

Our key contributions are summarized as follows:

To the best of our knowledge, this work is the first to systematically address the challenge of efficient LoRA weight transfer during LLM upgrades, analyzed from six explicit perspectives.

We propose LoRASuite , a modular approach tailored to various types of LLM upgrades, including a layer-mapping strategy based on centered kernel alignment and an attention-head mapping approach utilizing the Hungarian algorithm.

LoRASuite consistently outperforms small-scale vanilla LoRA across diverse tasks-including commonsense reasoning and mathematical evaluations-and, in certain scenarios, even surpasses full-scale LoRA retraining. Additionally, it significantly reduces both memory and time consumption.

2 Related Work

Interpretability of LLM. Mechanistic interpretability in deep neural networks has gained significant attention in recent years. Prior work has largely focused on analyzing hidden representations through techniques like probing [11], activation patching [12], and causal tracing [13], or on interpreting specific network weights by mapping model components to vocabulary space [14]. Our approach is based on the latter, and future work can further explore the integration of different methods. For a comprehensive discussion on mechanistic interpretability in transformer-based LLMs, see [15].

Efficient Fine-tuning. This work builds on LoRA, a classical and efficient fine-tuning method. Beyond LoRA, several other parameter-efficient fine-tuning (PEFT) techniques have been proposed. Adapter tuning [16] inserts lightweight trainable modules into the frozen backbone. Prompt tuning [17] appends trainable soft tokens to the input sequence, while prefix tuning extends this idea by inserting soft tokens into each layer's hidden representations. Hidden state tuning [18], exemplified by ( IA ) 3 , rescales attention keys and values as well as activations in the feed-forward layers. Bias

tuning [19] updates only the model's bias terms or a selected subset. Masked weight learning [20] applies a fixed sparse mask to the model's parameters. Input tuning [21] introduces an adapter at the embedding layer to adjust input representations.

Optimization for LoRA. LoRA [1], one of the most effective methods for parameter-efficient fine-tuning, has gained significant attention in recent years, leading to numerous variants and optimizations. For example, AdaLoRA [22] dynamically learns the required rank size for each model layer, LoSparse [23] integrates LoRA to prevent pruning from removing too many expressive neurons, DoRA [24] adapts LoRA based on magnitude and direction, and PiSSA [25] tunes the essential low-rank components while freezing high-rank, nonessential parts. LoRA+ [26] applies different learning rates to the A and B modules. Our work is orthogonal to these approaches, and future research can explore more efficient combinations of our method with these variants.

Knowledge Transfer between LLMs. Prior research on knowledge transfer between pre-trained language models has primarily focused on transferring prompt-based modules [27, 28, 29]. These methods typically align overlapping vocabularies between models, which is conceptually similar to our vocabulary and hidden-size alignment via linear combination described in Section 3.1. While several studies examine the transferability of LoRA or PEFT modules, they often rely on modifying the original fine-tuning process or introducing additional training steps, rather than enabling direct and efficient transfer. For example, Trans-LoRA [30] improves LoRA transferability through synthetic data generation, Trans-PEFT [31] enhances PEFT transfer via modified fine-tuning procedures, and MUSCLE [32] employs knowledge distillation to transfer adapters across models. Furthermore, some works [33, 34] investigate continual pre-training as a means of incremental model updating-analogous to implicit dataset upgrades in our setting-where direct transfer proves to be the most effective approach.

3 LoRASuite

Given a pre-trained weight matrix W o ∈ R d o × d o t in the original LLM, LoRA trained on a specific downstream corpus comprises two low-rank matrices, A o ∈ R r × d o t and B o ∈ R d o × r , where r ≪ min ( d o , d o t ) . Typically, matrix B o is initialized with zeros, and A o is with Kaiming Uniform [35], ensuring B o A o = 0 initially.

Upon upgrading the LLM backbone to a newer version, as summarized in Table 1, six explicit factors-vocabulary size, hidden size, intermediate dimension of up/down-projection layers, layer depth, attention head numbers, and attention type-prevent direct reuse of existing matrices A o and B o for the new weight matrix W n ∈ R d n × d nt .

Therefore, this section introduces several methods to adapt existing LoRA weights A o and B o to new matrices A n and B n without retraining from scratch when upgrading from W o to W n on the same corpus. We specifically focus on upgrades within LLMs of identical architecture, as defined in their respective config.json files. For instance, Phi-1.5 and Phi-2 share the same PhiForCausalLM

architecture, while Phi-3 has a different Phi3ForCausalLM architecture. Consequently, the activation function remains unchanged in our scenario. Exploring the impact of varying activation functions represents a promising direction for future research. In addition to the six explicit upgrade factors, we recognize that implicit upgrades, such as changes in pre-training datasets and post-training methods (e.g., RLHF), also influence model adaptation. Future research could further investigate the effects of these implicit factors.

3.1 Methodology

Vocabulary Size and Hidden Size. Vocabulary size and hidden dimension directly influence the embedding layer weights of a model, with hidden dimension mismatches notably restricting the direct reuse of existing LoRA weights. Based on variations in vocabulary size and hidden dimension, upgrades can be classified into

three scenarios, as detailed in Table 2. In the simplest scenario-where only the hidden dimension changes-the transformation matrix W h can be directly computed using the embedding weights of both the original and upgraded models. Assuming that all parameters from both models are available, the transfer matrix is specifically computed as W h = E -1 o E n , where E o and E n denote the embedding weights of the original and updated models, respectively. For scenarios involving simultaneous changes in vocabulary size and hidden dimension, an additional intersection step is required to filter shared tokens, after which the transformation matrix is calculated using the filtered embedding weights.

Intermediate Size. When LoRA target modules contain up and down projection layers, mismatches in intermediate dimensions pose another significant challenge. Following a similar strategy, we leverage known parameters from both models to compute the transformation matrix W i . Specifically, W i is calculated as W i = W -1 o W h W n , where W o and W n represent the weights of the original and updated up/down projection layers, respectively.

Layer Depth. Inspired by prior studies on representational similarity, we employ centered kernel alignment (CKA) to quantify the similarity between corresponding layers of two LLMs. Following [36], we adopt a minibatch implementation of CKA to reduce memory usage. Based on the computed CKA similarities, we propose a novel dynamic programming-based layer mapping algorithm, which sequentially aligns corresponding layers to maximize the total similarity, subject to a predefined maximum offset constraint.

CKA [37] provides a robust method for quantifying neural network representations by measuring the similarity between pairs of activation tensors. Let X ∈ R m × d 1 and Y ∈ R m × d 2 denote the activations of two layers, with d 1 and d 2 neurons, respectively, across the same set of m examples. The elements of the Gram matrices K = XX T and L = Y Y T repre-

Algorithm 1 CKA-based Layer Mapping

sent the pairwise similarities between examples based on X and Y . Using the centering matrix H = I n -1 n 11 T , the centered Gram matrices K ′ = HKH and L ′ = HLH remove mean effects. HSIC measures their similarity by flattening the matrices and computing the dot product:

HSIC 0 ( K,L ) = vec ( K ′ ) · vec ( L ′ ) / ( m -1) 2 . HSIC is invariant to orthogonal transformations and neuron permutations but remains sensitive to scaling. CKA normalizes HSIC to yield a similarity index between 0 and 1 that is invariant to isotropic scaling.

However, computing CKA naively requires storing activations for the entire dataset, which is impractical for wide and deep LLMs. To reduce memory usage, [36] proposes estimating CKA by averaging HSIC scores across k minibatches HSIC 0 ( K,L ) ⇒ 1 k ∑ k i =1 HSIC 1 ( X i X T i , Y i Y T i ) . In place of HSIC 0 , which is a biased estimator of HSIC, the usage of an unbiased estimator of HSIC 1 [38] makes the value of CKA independent of batch size:

where ˜ K and ˜ L are obtained by setting the diagonal entries of similarity matrices K and L to zero. This minibatch-based HSIC estimation is equivalent to the bagging block HSIC method in [39] and converges to the full-dataset HSIC, as proven in [36].

Using the minibatch CKA defined above, we construct a similarity matrix S , where each element S i,j represents the CKA similarity between the i -th layer of the original model and the j -th layer of the upgraded model. Inspired by dynamic programming, we propose a layer-mapping algorithm that maximizes the total similarity in S by optimally aligning corresponding layers. To further guide the alignment, we impose an ordered mapping constraint, restricting the layer assignment within a threshold ∆ layer , which accounts for differences in layer depth between the original and upgraded models. The detailed procedure is outlined in Algorithm 1.

Head Number. Recall that the attention projection matrices W Q , W K , and W V can be split along the column axis to H parts, denoted as W i Q , W i K , and W i V ∈ R d × d/H , for 1 ≤ i ≤ H . Similarly, the output projection matrix W O is split along the row axis into H heads, with W i O ∈ R d/H × d . For each head, we define two input-independent interaction matrix: W i QK := W i Q W i K T ∈ R d × d and W i V O := W i V W i O T ∈ R d × d . Intuitively, W i QK captures the attention strength between token pairs, while W i V O models how attending to specific tokens influences the subsequent hidden state.

To address differences in attention head counts, we characterize each head using its interaction matrices and compute a similarity matrix via cosine similarity between heads from the old and new models, at the layers specified above. When interaction matrix dimensions differ, we align them by transforming to a common hidden size d n (typically the new model's hidden size) using the embedding-based transformation W h . We then determine the optimal head mapping by maximizing the sum of similarities, applying the Hungarian algorithm with the similarity matrix treated as a cost matrix. Further algorithmic details and theoretical analysis are provided in Appendix A.1.

Changes in attention mechanisms primarily affect the number of W K and W V heads. For instance, transitioning from GQA to MHA in MiniCPM-1B to MiniCPM-2B increases the number of K and V heads from 8 to 36. As in the forward pass, we replicate the K and V heads to match the number of Q heads before applying the head mapping algorithm.

Head Dimension. Following the attention head granularity used in the model's forward pass, we also adapt LoRA weights to the attention head level. As shown in Table 3, the hidden size and number of attention heads jointly determine the dimension of each head. Similar to earlier steps, we leverage known parameters from both the original and new models to adapt the LoRA weights. Specifically, we first compute the

Algorithm 2 Pseudocode of LoRASuite in PyTorch-like style.

weight update ∆ W o for each projection matrix by multiplying the corresponding B o and A o , then split it along the specified axis to extract updates for individual heads. Let ( W i Q ) o and (∆ W i Q ) o denote the original weights and weight updates for the i -th head in the original model. Based on the Hungarian algorithm, the weight update for the j -th head in the new model is computed as:

where, W h , derived from the embedding weights of both models, is omitted if the hidden size remains unchanged. Incorporating original weights improves numerical stability, allowing the transformed LoRA to be directly applied in some cases. Finally, after all head mappings are completed, a single SVD step decomposes (∆ W Q ) n into its low-rank B n and A n components.

We employ different strategies for layer and head mapping due to their distinct structural roles. For layer mapping, we adopt a dynamic programming approach inspired by studies on CNNs, where different layers capture different levels of abstraction, and higher layers build upon the representations of lower layers [40]. In contrast, attention heads within the same layer operate in parallel and independently [41], so we formulate head mapping as a bipartite matching problem and solve it using the Hungarian algorithm to obtain the optimal one-to-one correspondence.

3.2 Put All Together: A Final Recipe

Integrating the five components described above, the pseudocode of LoRASuite is presented in Algorithm 2. The CKA-based layer similarity and head similarity between the original and upgraded models can be precomputed offline, eliminating the need for repeated calculation during each adaptation.

Since the transformed LoRA parameters are generated via matrix multiplication rather than backpropagation, they may introduce numerical instability and degrade performance. To mitigate this, we introduce a lightweight fine-tuning step to help the upgraded model better adapt. Unlike standard Transformer Trainer that uses a linear learning-rate scheduler with warm-up to stabilize optimization from random initialization [42], our model is initialized with transformed parameters. Therefore, we omit the warm-up phase and increase the learning rate to compensate.

3.3 Complexity Analysis

The complexity of LoRASuite primarily arises from three key processes: layer mapping using the CKA method, attention head mapping utilizing the Hungarian algorithm, and the final SVD. As illustrated in Algorithm 1, the CKA-based layer mapping has a time complexity of O ( n layer ∆ 2 layer ) ,

where n layer denotes the number of layers in the original model, and ∆ layer represents the difference in the number of layers between the original and upgraded models. According to [43], the Hungarian algorithm exhibits a time complexity of O ( n 3 head ) , where n head indicates the smaller number of attention heads present either in the original or upgraded model. Based on[44], the complexity of the SVD process to obtain the B n,j , A n,j in j -th layer is O ( n 3 hidden ) . Combining these factors, the total computational complexity becomes O ( n layer (∆ layer 2 + n head 3 + n 3 hidden )) . Since n hidden >> n head , ∆ layer , the overall complexity is effectively dominated by the SVD and simplifies to O ( n layer n 3 hidden ) .

4 Experiments

Implementation Details. Our implementation 1 builds on the publicly available Huggingface transformers and peft libraries, with modifications to support initialization from transformed weights. All experiments were conducted on a Linux server equipped with 80 Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz cores, 500 GB RAM, and 8 NVIDIA V100 GPUs.

Experimental Settings. For commonsense reasoning tasks, we evaluate BoolQ [45], PIQA [46], SIQA [47], HellaSwag [48],WinoGrande [49], ARC-e, ARC-c [50], and OBQA [51]. For math tasks, we evaluate AQuA [52], GSM8K [53], MAWPS [54], and SVAMP [55].

To ensure reproducibility, we use the same training and evaluation datasets as LLM-Adapters [56]. Both the original and upgraded models' LoRA (10k) are trained on the full dataset with 10k samples, whereas LoRA (100) and LoRASuite with LFT (100) are trained on identical, randomly selected subsets of the LoRA (10k) dataset with 100 samples.

Hyperparameter Settings. Unless otherwise specified, we use the default LoRA settings: rank 32, alpha 32, and dropout 0. A sensitivity analysis of these parameters is provided in Section 4.2. By default, the target modules include q_proj , k_proj , v_proj , and o_proj , with performance for other modules detailed in the Appendix A.3.

Yi-6B to Yi-1.5-9B

(a)

0

20

40

60

80

Avg. Accuracy (%)

65.86

73.07

69.28

74.83

76.32

(b)

0

5

10

15

20

25

Avg. Accuracy (%)

3.87

3.89

3.87

9.52

22.45

0

Bloom-560m to Bloomz-1B1

(c)

5

10

15

Avg. Accuracy (%)

3.83

3.85

3.81

6.26

14.52

Llama-2-7B to Llama-3-8B

(d)

0

20

40

Avg. Accuracy (%)

16.07

15.51

17.40

40.18

49.52

Qwen1.5-1.8B to Qwen2.5-3B

(e)

0

20

40

60

Avg. Accuracy (%)

16.06

15.94

19.33

56.88

50.32

Vanilla

LoRASuite w/o LFT

LoRA (Small)

LoRASuite

LoRA (Full)

4.1 Overall Performance

As shown in Table 1, upgrading from MiniCPM-S-1B to MiniCPM-2B involves all six explicit aspects. Thus, we primarily use MiniCPM to demonstrate LoRASuite 's overall performance.

Accuracy. Tables 4 and 5 present performance results for math and commonsense tasks, respectively. " LoRASuite w/o LFT" denotes LoRASuite 's performance using only the transformed MiniCPMS-1B LoRA without further fine-tuning with small-scale data. " LoRASuite w LFT (100)" indicates LoRASuite with additional fine-tuning at a data scale of 100. For MiniCPM-2B, "LoRA (10k)" represents LoRA retraining using the complete dataset, while "LoRA (100)" serves as a baseline for retraining at the same scale. Under the default setting with rank 32 for MiniCPM-2B, the number of trainable parameters for LoRA (100), LoRASuite with LFT (100), and LoRA (10k) is the same-23.59M, which is approximately 0.78% of all model parameters. From the tables, we observe: First, LoRASuite achieves a notable average score of 43.80 on math tasks, nearly doubling the performance compared to same-scale fine-tuning (22.82) and surpassing full dataset retraining (42.39). Similarly, in commonsense tasks, LoRASuite outperforms both same-scale LoRA fin-tuning and full dataset retraining.

Additionally, " LoRASuite w/o LFT" exhibits performance nearly identical to the vanilla MiniCPM2B model on both math and commonsense tasks. This result supports the hypothesis that transformation relying solely on matrix multiplication without backpropagation leads to numerical instability. It further validates the effectiveness of additional small-scale fine-tuning.

Memory and time consumption. Figure 1 demonstrates that LoRASuite significantly outperforms retraining a new LoRA model, achieving memory savings of 5.5GB and reducing time consumption by 78.23%. Although LoRASuite requires a small-scale fine-tuning stage, it yields a modest memory reduction of 5.5 GB due to three factors. First, data pipeline overhead and auxiliary states (e.g., DataLoader prefetching, dataset shuffling, and caching) scale with both the number of samples and training

(a)

0

10

20

30

40

50

26.85

36.72

42.19

(b)

0

50

100

150

Time (min)

27.5

29.5

135.5

LoRASuite w/o LFT

LoRASuite

steps. Second, preloading the full dataset or pre-tokenized storage directly increases host DRAM usage. Third, for variable-length inputs, larger datasets increase the likelihood of long-sequence batches, resulting in excessive padding and transient memory spikes from dynamic allocation. For " LoRASuite w/o LFT," the memory reduction is more substantial, reaching 36.36% (15.34GB). This memory efficiency stems from loading only the parameters of the new and old models for matrix operations, thereby eliminating the overhead associated with optimizer states and gradients. The notable decrease in time consumption occurs because LoRASuite avoids the need for loading extensive datasets, performing backpropagation, or training for multiple epochs from scratch. Even with small-scale fine-tuning, the additional time is only approximately two minutes.

Multiple LLM backbones. Figures 2 and 3 present the average performance comparisons across math and commonsense tasks, respectively, for different LLM backbones. The required amount of additional fine-tuning data for LoRASuite varies across different LLM backbones. Henceforth, we mainly use ' LoRASuite ' to refer to the variant with lightweight fine-tuning ( LoRASuite w LFT (Small)), which also applies to the comparison method LoRA (Small). Our observations from these figures are as follows: First, LoRASuite consistently achieves performance comparable to or exceeding full-scale LoRA retraining in several scenarios. Second, LoRASuite significantly

Yi-6B to Yi-1.5-9B

(a)

0

20

40

60

80

Avg. Accuracy (%)

56.78

70.70

68.16

74.29

83.13

(b)

0

10

20

30

40

Avg. Accuracy (%)

18.27

16.74

34.12

35.43

35.02

0

Bloom-560m to Bloomz-1B1

(c)

10

20

30

40

Avg. Accuracy (%)

14.18

14.26

27.02

37.01

36.87

Llama-2-7B to Llama-3-8B

(d)

0

20

40

Avg. Accuracy (%)

31.34

31.39

44.72

46.25

48.91

Qwen1.5-1.8B to Qwen2.5-3B

(e)

0

20

40

60

Avg. Accuracy (%)

46.64

46.42

42.71

54.41

56.30

Vanilla

LoRASuite w/o LFT

LoRA (Small)

LoRASuite

LoRA (Full)

4

8

16

32

64

Rank r

(a)

25

30

35

40

45

Avg. Accuracy (%)

LoRASuite

LoRA (Full)

LoRA (Small)

1e-4

3e-4

6e-4

9e-4

1e-3

Learning Rate

(b)

25

30

35

40

45

Avg. Accuracy (%)

LoRASuite

LoRA (Small)

100

200

400

800

1000

Datascale

(c)

25

30

35

40

45

Avg. Accuracy (%)

LoRASuite

LoRA (Small)

outperforms same-scale LoRA fine-tuning (LoRA (Small)) across all cases, with a notably pronounced improvement-nearly threefold-in math tasks when upgrading from Qwen-1.5-1.8B to Qwen2.5-3B. Additionally, LoRASuite consistently surpasses " LoRASuite w/o LFT," highlighting lightweight small-scale fintuing's critical role in enhancing numerical stability. Finally, " LoRASuite w/o LFT" occasionally achieves performance similar to or slightly better than the vanilla model and "LoRA (Small)," suggesting that numerical stability may not be a significant concern for certain models. Future research could focus on developing more robust algorithms that avoid additional fine-tuning and accommodate diverse model update scenarios.

4.2 Sensitive Analysis

Different rank. Figure 4(a) presents the performance comparison between LoRASuite and baseline methods on math tasks across varying LoRA ranks. LoRASuite consistently outperforms vanilla LoRA in both small-scale and full-scale fine-tuning scenarios. Notably, at a LoRA rank of 4, LoRASuite achieves 93% of the performance obtained from training with 10K samples using only 100 samples. At a LoRA rank of 16, LoRASuite achieves its greatest improvement over full-scale LoRA, with an average increase of approximately 2.67 points.

Different LFT learning rate. The experimental results depicted in Figure 4(b) indicate that LoRASuite is highly sensitive to changes in the learning rate compared to vanilla LoRA (Small). Specifically, when the learning rate is 1e-4, LoRASuite marginally outperforms LoRA (Small) by only 0.27 percentage points on average. However, at a learning rate of 9e-4, LoRASuite substantially outperforms LoRA (Small) by an average of 21.30 points, representing approximately a twofold improvement in performance. This sensitivity likely arises because LoRASuite leverages previously trained LoRa weights to quickly identify crucial task-specific features, creating a steeper optimization landscape and greater dependence on the chosen learning rate.

Different LFT data scale. When trained on small datasets, LoRASuite maintains stable performance and consistently outperforms LoRA (Small). However, its performance declines with increasing dataset size, likely due to LoRASuite 's reliance on historically trained LoRA weights, making it more susceptible to overfitting.

4.3 Application to Other PEFTs

Table 6 and Table 7 report the average performance of two PEFT methods, AdaLoRA and DoRA, on math tasks when upgrading the LLM from MiniCPM-S-1B to MiniCPM-2B. As shown, LoRASuite yields a substantial improvement for DoRA adaptation: while the small-scale vanilla LoRA achieves only 23.54% accuracy, LoRASuite improves this by more than 1 . 8 × , reducing the performance

gap with full-scale LoRA to less than 1%. In contrast, the improvement offered by LoRASuite over LoRA (Small) is less pronounced for AdaLoRA, although it still surpasses full-scale LoRA retraining. This discrepancy is likely due to rank incompatibilities between corresponding layers of the original and upgraded models.

5 Conclusion and Discussion

We propose LoRASuite , a modular framework designed to handle diverse types of LLM upgrades. To the best of our knowledge, this is the first work to systematically address the challenge of efficient LoRA weights transfer during LLM upgrades across six explicit perspectives. LoRASuite consistently outperforms small-scale vanilla LoRA across diverse tasks and, in certain scenarios, even surpasses full-scale LoRA retraining with both memory and time reduction.

Limitations. Currently, LoRASuite requires an additional small-scale fine-tuning step to achieve superior performance. Future research could investigate strategies to eliminate this step without compromising performance, further minimizing memory usage. Additionally, our study primarily addresses explicit upgrade aspects of LLMs; future investigations could extend to implicit upgrades, such as variations in pre-training datasets and post-training methods, with knowledge editing methods [13, 57, 58]. Lastly, exploring LoRA adaptation methods applicable to different architectures remains an open avenue for future work.

6 Acknowledgments and Disclosure of Funding

The authors thank the anonymous reviewers for their insightful feedback and Longxi Gao for assistance with complementary experiments. This work was supported by the National Natural Science Foundation of China (No.62425203, No.62032003) and partly by the Supercomputing Platform of Beijing University of Posts and Telecommunications. Fanxu Meng and Muhan Zhang are supported by the National Key R&D Program of China (2022ZD0160300). Mengwei Xu was also supported by the National Natural Science Foundation of China (No.62522202) and the Beijing Natural Science Foundation (No.L253005). Mengwei Xu is the corresponding author.

References

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

Google, 'Gemini nano with the google ai edge sdk,' 2025. [Online]. Available: https://developer.android.com/ai/gemini-nano

H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y. Liu, Y. Zhang, and Y. Liu, 'Autodroid: Llm-powered task automation in android,' in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , 2024, pp. 543-557.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , 'Llama 2: Open foundation and fine-tuned chat models,' arXiv preprint arXiv:2307.09288 , 2023.

Meta, 'The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,' 2025. [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

A. Yang, B. Yang, B. Hui et al. , 'Qwen2 technical report,' arXiv preprint arXiv:2407.10671 , 2024.

A. Yang, B. Yang, B. Zhang et al. , 'Qwen2.5 technical report,' arXiv preprint arXiv:2412.15115 , 2024.

M. Xu, D. Cai, Y. Wu, X. Li, and S. Wang, 'Fwdllm: Efficient federated finetuning of large language models with perturbed inferences,' in 2024 USENIX Annual Technical Conference (USENIX ATC 24) , 2024, pp. 579-596.

Y. Xia, J. Kim, Y . Chen, H. Ye, S. Kundu, C. Hao, and N. Talati, 'Understanding the performance and estimating the cost of llm fine-tuning,' arXiv preprint arXiv:2408.04693 , 2024.

E. Strubell, A. Ganesh, and A. McCallum, 'Energy and policy considerations for modern deep learning research,' in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 09, 2020, pp. 13 693-13 696.

A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, 'What you can cram into a single vector: Probing sentence embeddings for linguistic properties,' arXiv preprint arXiv:1805.01070 , 2018.

A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, 'Towards automated circuit discovery for mechanistic interpretability,' Advances in Neural Information Processing Systems , vol. 36, pp. 16 318-16 352, 2023.

K. Meng, D. Bau, A. Andonian, and Y. Belinkov, 'Locating and editing factual associations in gpt,' Advances in Neural Information Processing Systems , vol. 35, pp. 17 359-17 372, 2022.

G. Dar, M. Geva, A. Gupta, and J. Berant, 'Analyzing transformers in embedding space,' arXiv preprint arXiv:2209.02535 , 2022.

D. Rai, Y. Zhou, S. Feng, A. Saparov, and Z. Yao, 'A practical review of mechanistic interpretability for transformer-based language models,' arXiv preprint arXiv:2407.02646 , 2024.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, 'Parameter-efficient transfer learning for nlp,' in International conference on machine learning . PMLR, 2019, pp. 2790-2799.

B. Lester, R. Al-Rfou, and N. Constant, 'The power of scale for parameter-efficient prompt tuning,' arXiv preprint arXiv:2104.08691 , 2021.

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, 'Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 1950-1965, 2022.

E. B. Zaken, S. Ravfogel, and Y. Goldberg, 'Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,' arXiv preprint arXiv:2106.10199 , 2021.

Y.-L. Sung, V. Nair, and C. A. Raffel, 'Training neural networks with fixed sparse masks,' Advances in Neural Information Processing Systems , vol. 34, pp. 24 193-24 205, 2021.

S. An, Y. Li, Z. Lin, Q. Liu, B. Chen, Q. Fu, W. Chen, N. Zheng, and J.-G. Lou, 'Input-tuning: Adapting unfamiliar inputs to frozen pretrained models,' arXiv preprint arXiv:2203.03131 , 2022.

Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, 'Adaptive budget allocation for parameter-efficient fine-tuning,' in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id= lq62uWRJjiY

Y. Li, Y. Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao, 'Losparse: Structured compression of large language models based on low-rank and sparse approximation,' in International Conference on Machine Learning . PMLR, 2023, pp. 20 336-20 350.

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, 'Dora: Weight-decomposed low-rank adaptation,' arXiv preprint arXiv:2402.09353 , 2024.

F. Meng, Z. Wang, and M. Zhang, 'Pissa: Principal singular values and singular vectors adaptation of large language models,' arXiv preprint arXiv:2404.02948 , 2024.

S. Hayou, N. Ghosh, and B. Yu, 'Lora+: Efficient low rank adaptation of large models,' arXiv preprint arXiv:2402.12354 , 2024.

Y. Su, X. Wang, Y. Qin, C.-M. Chan, Y. Lin, H. Wang, K. Wen, Z. Liu, P. Li, J. Li et al. , 'On transferability of prompt tuning for natural language processing,' arXiv preprint arXiv:2111.06719 , 2021.

B. Lester, J. Yurtsever, S. Shakeri, and N. Constant, 'Reducing retraining by recycling parameterefficient prompts,' arXiv preprint arXiv:2208.05577 , 2022.

Z. Wu, Y. Wu, and L. Mou, 'Zero-shot continuous prompt transfer: Generalizing task semantics across language models,' in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=26XphugOcS

R. Wang, S. Ghosh, D. Cox, D. Antognini, A. Oliva, R. Feris, and L. Karlinsky, 'Trans-lora: towards data-free transferable parameter efficient finetuning,' Advances in Neural Information Processing Systems , vol. 37, pp. 61 217-61 237, 2024.

N. Gu, P. Fu, X. Liu, K. Ma, Z. Lin, and W. Wang, 'Adapt once, thrive with updates: Transferable parameter-efficient fine-tuning on evolving base models,' arXiv preprint arXiv:2506.06844 , 2025.

J. Echterhoff, F. Faghri, R. Vemulapalli, T.-Y . Hu, C.-L. Li, O. Tuzel, and H. Pouransari, 'Muscle: A model update strategy for compatible llm evolution,' arXiv preprint arXiv:2407.09435 , 2024.

Y. Qin, C. Qian, X. Han, Y. Lin, H. Wang, R. Xie, Z. Liu, M. Sun, and J. Zhou, 'Recyclable tuning for continual pre-training,' arXiv preprint arXiv:2305.08702 , 2023.

R. Shahroz, P. Li, S. Yun, Z. Wang, S. Nirjon, C.-W. Wong, and T. Chen, 'PortLLM: Personalizing evolving large language models with training-free and portable model patches,' in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://openreview.net/forum?id=gyHoR6uFhU

K. He, X. Zhang, S. Ren, and J. Sun, 'Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026-1034.

T. Nguyen, M. Raghu, and S. Kornblith, 'Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,' in International Conference on Learning Representations , 2021. [Online]. Available: https://openreview.net/forum?id=KJNcAkY8tY4

S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, 'Similarity of neural network representations revisited,' in International conference on machine learning . PMLR, 2019, pp. 3519-3529.

L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt, 'Feature selection via dependence maximization,' The Journal of Machine Learning Research , vol. 13, pp. 1393-1434, 2012.

M. Yamada, Y. Umezu, K. Fukumizu, and I. Takeuchi, 'Post selection inference with kernels,' in International conference on artificial intelligence and statistics . PMLR, 2018, pp. 152-160.

M. D. Zeiler and R. Fergus, 'Visualizing and understanding convolutional networks,' in European conference on computer vision . Springer, 2014, pp. 818-833.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, 'On the variance of the adaptive learning rate and beyond,' arXiv preprint arXiv:1908.03265 , 2019.

J. Munkres, 'Algorithms for the assignment and transportation problems,' Journal of the society for industrial and applied mathematics , vol. 5, no. 1, pp. 32-38, 1957.

X. Li, S. Wang, and Y. Cai, 'Tutorial: Complexity analysis of singular value decomposition and its variants,' arXiv preprint arXiv:1906.12085 , 2019.

C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, 'Boolq: Exploring the surprising difficulty of natural yes/no questions,' arXiv preprint arXiv:1905.10044 , 2019.

Y. Bisk, R. Zellers, J. Gao, Y. Choi et al. , 'Piqa: Reasoning about physical commonsense in natural language,' in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432-7439.

M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi, 'Socialiqa: Commonsense reasoning about social interactions,' arXiv preprint arXiv:1904.09728 , 2019.

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, 'Hellaswag: Can a machine really finish your sentence?' arXiv preprint arXiv:1905.07830 , 2019.

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, 'Winogrande: An adversarial winograd schema challenge at scale,' Communications of the ACM , vol. 64, no. 9, pp. 99-106, 2021.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, 'Think you have solved question answering? try arc, the ai2 reasoning challenge,' arXiv preprint arXiv:1803.05457 , 2018.

T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, 'Can a suit of armor conduct electricity? a new dataset for open book question answering,' arXiv preprint arXiv:1809.02789 , 2018.

W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, 'Program induction by rationale generation: Learning to solve and explain algebraic word problems,' arXiv preprint arXiv:1705.04146 , 2017.

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al. , 'Training verifiers to solve math word problems,' arXiv preprint arXiv:2110.14168 , 2021.

R. Koncel-Kedziorski, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi, 'Mawps: A math word problem repository,' in Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies , 2016, pp. 1152-1157.

A. Patel, S. Bhattamishra, and N. Goyal, 'Are nlp models really able to solve simple math word problems?' arXiv preprint arXiv:2103.07191 , 2021.

Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria, 'Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,' arXiv preprint arXiv:2304.01933 , 2023.

P. Wang, Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, F. Huang, and H. Chen, 'Wise: Rethinking the knowledge memory for lifelong model editing of large language models,' Advances in Neural Information Processing Systems , vol. 37, pp. 53 764-53 797, 2024.

Y. Yao, N. Zhang, Z. Xi, M. Wang, Z. Xu, S. Deng, and H. Chen, 'Knowledge circuits in pretrained transformers,' Advances in Neural Information Processing Systems , vol. 37, pp. 118 571-118 602, 2024.

Q. Sun, M. Pickett, A. K. Nain, and L. Jones, 'Transformer layers as painters,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no. 24, 2025, pp. 25 219-25 227.

A. Morcos, M. Raghu, and S. Bengio, 'Insights on representational similarity in neural networks with canonical correlation,' Advances in neural information processing systems , vol. 31, 2018.

L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, 'Language models are super mario: Absorbing abilities from homologous models as a free lunch,' in Forty-first International Conference on Machine Learning , 2024.

P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, 'Resolving interference when merging models,' arXiv preprint arXiv:2306.01708 , vol. 1, 2023.

A Appendix

A.1 Details of Hungarian Algorithm

The Hungarian algorithm, also known as the Kuhn-Munkres algorithm, is a combinatorial optimization technique for solving the assignment problem. It guarantees to find the optimal one-to-one assignment that minimizes the total cost in a bipartite graph. As shown in the above algorithm, the procedure mainly consists of four phases: (1) Matrix Preparation : Convert the cost matrix to a square matrix by zero-padding if necessary. (2) Row/Column Reduction : Subtract the minimum value of each row from its elements, then repeat for columns, creating at least one zero per row and column. (3) Initial Matching : Identify a maximum matching using zero-cost entries. If a perfect match is found, the algorithm terminates. (4) Iterative Adjustment : If unmatched, iteratively (4-a) cover zeros with minimal lines, (4-b) compute the smallest uncovered element δ , (4-c) adjust the matrix by subtracting δ from uncovered elements and adding δ to doubly-covered elements, and (4-d) update the matching. This loop continues until the number of covering lines equals the matrix dimension, guaranteeing an optimal assignment. The algorithm operates in O ( n 3 ) time [43], where n is the number of nodes in the smaller partition.

A.2 Hyperparameters

Tables 8 and 9 detail the hyperparameters of LoRA and LoRASuite for commonsense and math tasks, respectively.

A.3 Different Target Modules

Figures 5 and 6 present the performance of LoRASuite using q_proj, k_proj, v_proj, o_proj, up_proj, and down_proj as target modules. As shown, the performance improvement on the math dataset becomes more pronounced as the number of target modules increases. A likely explanation is that the commonsense corpus significantly overlaps with the LLM pre-training dataset; consequently, LoRASuite's additional small-scale fine-tuning is more impactful for mathematical tasks.

Bloom-560m to Bloomz-1B1

0

5

10

15

20

Avg. Accuracy (%)

3.83

3.94

5.15

8.01

20.92

Llama-2-7B to Llama-3-8B

0

10

20

30

40

Avg. Accuracy (%)

16.07

17.27

16.79

41.11

43.49

Qwen-1.5-1.8B to Qwen-2.5-3B

0

20

40

Avg. Accuracy (%)

16.06

16.01

26.26

34.30

49.91

MiniCPM-S-1B to MiniCPM-2B

0

10

20

30

40

50

Avg. Accuracy (%)

23.85

24.18

24.37

44.01

44.25

Vanilla

LoRASuite w/o LFT

LoRA (Small)

LoRASuite

LoRA (Full)

Bloom-560m to Bloomz-1B1

0

10

20

30

40

Avg. Accuracy (%)

14.18

13.97

36.22

37.32

37.14

Llama-2-7B to Llama-3-8B

0

20

40

60

Avg. Accuracy (%)

31.34

31.23

45.07

45.13

57.12

Qwen-1.5-1.8B to Qwen-2.5-3B

0

20

40

60

Avg. Accuracy (%)

46.64

46.23

52.81

53.78

61.50

MiniCPM-S-1B to MiniCPM-2B

0

20

40

Avg. Accuracy (%)

32.72

27.51

42.07

42.34

47.96

Vanilla

LoRASuite w/o LFT

LoRA (Small)

LoRASuite

LoRA (Full)

A.4 Heatmap of Layer Similarity

Figures 7 to 10 illustrate the CKA layer similarities of various LLM backbones before and after upgrading. The similarity patterns vary significantly across models, with MiniCPM and Pythia showing the highest variance. Notably, all models exhibit a clear block structure, similar to the three distinct representation spaces-'beginning,' 'middle,' and 'end' described in [59].

0

10

20

30

40

50

MiniCPM-S-1B

0

5

10

15

20

25

30

35

MiniCPM-2B

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0

10

20

30

40

Yi-1.5-9B

0

5

10

15

20

25

30

Yi-6B

0.93

0.94

0.95

0.96

0.97

0.98

0.99

tween Yi-6B and Yi-1.5-9B.

0

5

10

15

20

pythia-1.4b

0.0

2.5

5.0

7.5

10.0

12.5

15.0

pythia-1b

0.2

0.4

0.6

0.8

0

10

20

30

Qwen2.5-3B

0

5

10

15

20

Qwen1.5-1.8B

0.94

0.95

0.96

0.97

0.98

0.99

A.5 Detailed Results for Different LLM Backbones

Tables 10-19 provide the raw data underlying the aggregated results in Figures 2 and 3.

A.6 Ablation Studies

Math Tasks

(a)

0

20

40

Avg. Accuracy (%)

43.80

41.99

41.09

40.28

Commonsense Tasks

(b)

0

20

40

Avg. Accuracy (%)

42.34

42.28

40.92

33.53

LoRASuite

First

Medium

Last

Math Tasks

(a)

0

20

40

Avg. Accuracy (%)

43.80

42.58

41.60

Commonsense Tasks

(b)

0

20

40

Avg. Accuracy (%)

42.34

41.95

33.62

LoRASuite

Direct Algebra

Math Tasks

(a)

0

20

40

Avg. Accuracy (%)

43.80

43.80

41.06

41.94

Commonsense Tasks

(b)

0

20

40

Avg. Accuracy (%)

42.34

37.81

41.47

40.29

LoRASuite

CCA

Procrustes

PWCCA

(a) Qwen-1.5-1.8B to Qwen2-1.5B

0

20

40

60

Avg. Accuracy (%)

38.03

38.75

34.23

52.75

59.54

(b) Llama-2-713 to Llama-3.1-70B

0

20

40

60

80

Avg. Accuracy (%)

30.71

30.24

71.71

73.44

78.43

Vanilla

LoRASuite w/o LFT

LoRA (100)

LoRASuite

LoRA (10K)

CKA-based Layer Mapping. Figure 11 compares the performance of different layer mapping algorithms: "LoRASuite" refers to our CKA-based method described in Section 3.1; "First" maps layers sequentially from beginning to end; "Medium" spreads mappings outward from the middle; and "Last" maps from end to beginning. Our CKA-based approach outperforms the others on both math and commonsense tasks, demonstrating the effectiveness of using representational similarity to guide layer mapping.

Figure 12 further evaluates different representational similarity metrics. Canonical Correlation Analysis (CCA) [37] identifies linear projections that maximize correlation between two sets of activations. Projection-Weighted CCA (PWCCA) [60] extends CCA by weighting individual canonical correlations based on their importance. Procrustes analysis seeks the optimal orthogonal transformation that minimizes the Frobenius norm between two matrices. Our minibatch CKA-based method achieves superior performance across both task types, demonstrating the advantage of using minibatch CKA for guiding layer mapping.

Attention-head Mapping. Figure 13 compares the performance of different attention head mapping algorithms. "LoRASuite" refers to our Hungarian-based method described in Section 3.1, while "LoRASuite w/o Head Mapping" uses the default ordering without enforcing one-to-one correspondence based on cosine similarity, and the "Direct Algebra" method calculates the new LoRA weights by ∆ W n = W o -W n +∆ W o + c where c is just a learnable set of weights need to fine-tune. Our approach consistently outperforms the baseline on both math and commonsense tasks, highlighting the effectiveness of using input-independent interaction matrices and similarity-based head mapping. For the direct algebra method, the performance gap, especially on commonsense tasks, suggests that directly aligning to the trained W o +∆ W o may discard valuable information encoded in the new weight W n through its pretraining and post-training. As a result, even with further lightweight fine-tuning, this method struggles to fully leverage the potential of the stronger target model W n .

A.7 Loss Convergence

5

10

15

Steps

0.6

0.8

1.0

1.2

1.4

Training Loss

LoRA (100)

LoRASuite with LFT (100)

In Figure 15, we visualize the step-wise training loss for "LoRA (100)" and " LoRASuite with LFT (100)" under the settings of batch size = 16 and micro batch size = 4 for MiniCPM-2B. As mentioned in the Section 3.2, the default linear learning-rate scheduler of vanilla LoRA (100) with a warm-up phase to stabilize training from random initialization. However, with limited data, a considerable steps is spent in the warm-up stage, which slows convergence and limits performance. In contrast, LoRASuite starts with transformed parameters already close to a good solution, allowing us to omit warm-up and use a higher learning rate to accelerate convergence. This leads to better performance, especially in low-data regimes.

A.8 Merged LoRA Adaptation

Prior studies, such as DARE [61] and TIES [62], have demonstrated the feasibility and effectiveness of merging task-specific LoRA modules. In this section, we evaluate whether LoRASuite maintains its effectiveness when LoRA weights are merged across different tasks.

As shown in Table 20, we merged MiniCPM-S-1B's math and commonsense LoRA adapters using TIES with equal weights (0.5 each) and applied the same merging procedure to the LoRASuitetransformed adapters of MiniCPM-2B. Compared with the simple average of individual task performances, the merged LoRASuite-transformed adapters improved math and commonsense scores by 1.37× and 1.96×, respectively, demonstrating that our method remains effective even when merging weights from distinct tasks.

A.9 More LLM Backbones

Figure 14 (a) presents the performance on math tasks when upgrading LLM backbones from Qwen1.5-1.8B to Qwen2-1.5B, covering five major architectural changes: hidden size, intermediate size, layer depth, head number, and attention type. Similarly, Figure 14 (b) reports results for the upgrade from Llama-2-13B to Llama-3.1-70B, encompassing all six key architectural changes. In both cases, LoRASuite remains consistently effective.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our contribution is outlined as a separate paragraph in § 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are outlined as a separate paragraph in § 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the time complexity analysis for the algorithms involved in § 3.3 and § A.1, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Consistent with previous studies, our evaluation employs the widely recognized open-source benchmark [56].

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use the same open-source dataset as [56] and our code is available in https://github.com/YananLi18/LoRASuite .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed instructions on how to reproduce the main experimental in § 4

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We primarily report average performance metrics rather than error bars in our experiments. This choice aligns with prior studies that also emphasize mean performance. Additionally, due to significant variability across different datasets, reporting deviations comparable in magnitude to the averages would not yield meaningful insights. Nevertheless, we provide comprehensive data, enabling interested readers to compute error bars independently.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide detailed hardware information in § 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and believe that our research conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In § 1, we discussed real-world examples illustrating how our approach can promisingly mitigate the negative impacts associated with the current status quo, including excessive time consumption, high costs, and environmental harm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is intended for LoRa adaptation and does not involve high-risk data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited the original code, data, and models in § 4.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: We will provide detailed documentation for the new assets upon acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects; therefore, Institutional Review Board (IRB) approval is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a modular framework for reusing LoRA adapters across newer versions of language models. The idea is novel and has practical relevance. Experimental results demonstrate that LoRASuite achieves better performance than conventional LoRA fine-tuning in low-data regimes.\n\nSTRENGTHS AND WEAKNESSES:\n## strengths\n\n- How to use LoRA weights in a new model version is a very interesting idea. \n- The paper is well-organized and well-written. \n- The experimental results are comprehensive, especially when the authors compare the results with many backbone models.  \n\n## weaknesses\n\n- Actually, training a LoRA for a new version of a model is not expensive. Although LoRASuite has achieved some improvement compared to the base model, the w/o version still needs some additional training datasets to conduct the lightweight fine-tuning, which is not flexible. \n\n- There is no code released. I suspect the reproduction of the results. \n\n- The LoRAsuit is sensitive to the learning rate. In this case, the experimental results should have more training loss analysis. For example, there should be a training loss figure and model size comparison.\n\nQUESTIONS:\n- What are the statistics of the training dataset? \n\n- What is the training dataset for lightweight fine-tuning? \n- The\n\n- Could you present the training curve of the lightweigh for LoRA (100), LoRASuite with LFT (100), and LoRA (10k)?\n\n- Could you show the training parameters size for LoRA (100), LoRASuite w LFT (100), and LoRA (10k)?\n- It is a little weird why LoRASuite w/o LFT and base model get almost the same results. For example, since you use the LoRA adapters from MiniCPM-S-1B. For MiniCPM-2B, the LoRASuite w/o LFT get (23.96) compared to the base (23.85). \n\n- \"Although LoRASuite requires small-scale fine-tuning, this\nresults in a modest memory reduction of 5.5GB,\nprimarily due to the smaller sample size\" \nWhy can a smaller sample size reduce memory?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of adapting LoRA adapters for large language models (LLMs) when upgrading to new model versions that differ in architecture or parameters. The authors identify six types of incompatibilities that may arise during such upgrades. To address these, they propose LoRASuite, a modular framework that applies a sequence of techniques, including dimensionality transfer, layer and attention head alignment, and a brief fine-tuning step, to transfer LoRA adapters from an older LLM to a newer one. The method is evaluated on several backbone LLM families (like Llama, Yi, MiniCPM, Qwen, etc.) and two types of tasks, reporting improved results on adapter transfer efficiency, compute savings, and downstream performance than standard LoRA retraining.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The paper provides a systematic and detailed analysis of six specific incompatibility types, such as changes in vocabulary size, hidden dimensions, and attention structure, that arise when transferring LoRA adapters across large language model upgrades.\n2. It introduces a modular framework, LoRASuite, that integrates linear transfer, layer-wise alignment using CKA and dynamic programming, and attention head matching via the Hungarian algorithm, offering a practical and potentially extensible solution for LoRA transfer.\n3. The proposed approach is empirically validated on a diverse set of LLM backbones and on both math and commonsense QA tasks, with improved performance metrics compared to standard LoRA retraining.\n\n\nWeaknesses:\n1. Although the paper proposes an interesting method for transferring LoRA adapters, it is primarily evaluated on small to medium-scale models (up to 9B parameters), whereas its practical value would be much greater for larger models (e.g., 13B to 70B+), where adapter retraining is significantly more resource-intensive. This leaves the impact on truly large-scale LLMs unverified.\n2. The selected models used for evaluation are relatively dated, for example, Llama-3 to Llama-3.1/3.2 and Qwen1.5 to Qwen2/2.5/3. As a result, it remains unclear whether the proposed method can bring notable improvements when applied to more recent or stronger model baselines.\n3. The main advantage of the proposed method, saving compute and improving performance in low-data regimes, is only demonstrated when the fine-tuning dataset is extremely small (e.g., 100 samples), and the benefit diminishes or disappears as the dataset size increases, according to results in Section 4.2. Furthermore, the paper does not describe how the small fine-tuning datasets are selected, nor does it report results averaged over multiple random draws. This omission is problematic given the potential instability and variance in performance when using such small data subsets.\n\nQUESTIONS:\n1. Please address the weaknesses mentioned before.\n2. line 292: what does \"full-scale LoRA retraining\" mean? Is it the same as \"LoRA (10k)\" in Table 4/5? Same for \"LoRA (small)\". Please define it in the paper explicitly and keep it consistent throughout the paper.\n3. Why did you choose to upgrade Qwen1.5-1.8B to Qwen2.5-3B, while skipping Qwen2-1.5B? It could be interesting to see a comparative study on the performance of LoRASuite when upgrading to Qwen2-1.5B vs. Qwen2.5-3B, as the former is a more direct upgrade in terms of model size and architecture.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LoRASuite, a modular approach tailored specifically to various types of LLM updates. It identifies structural differences (e.g., vocabulary size, hidden size, attention heads) between old and new models and adapts existing LoRA weights accordingly through transfer matrices, layer/head mapping (via CKA and the Hungarian algorithm), and a lightweight fine-tuning (LFT) step. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods and full-scale\nLoRA retraining, while LoRASuite significantly reduces memory con-\nsumption by 5.5 GB and computational time by 78.23%.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper studies an interesting yet practically important and under-explored problem of adapting LoRA weights during LLM upgrades.\n- The paper systematically explores the problem by explicitly identifying the potential changed dimensions (e.g., vocabulary size, hidden size, attention heads)\n- Extensive experiments across multiple model upgrades, task types (math, commonsense) and thorough baselines demonstrates strong empirical results. Extensive ablations on layer and head mapping.\n\nWeaknesses:\nSee my questions.\n\nQUESTIONS:\n- Why LoRASuite is much more sensitive to the learning rate compared to vanilla LoRA (fig4-b)?\n- How do you set the maximum offset constraint in Algo 1?\n- What happens if the optimal head mapping is not one-to-one, i.e., the information could be distributed across or aggregated from multiple heads)?\n- What happens if the tokenizer or embedding changes significantly?\n- What could be some potential solutions for implicit upgrades, such as changes in pre-training datasets and post-training methods?\n- Does the LoRASuite still work when the LoRA weights are for different tasks?\n- Curious whether there are relatively different sensitivity when matching various factors, e.g., is layer matching more important/sensitive than head mapping?\n\nLIMITATIONS:\nSee my questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a novel method to leverage existing LoRA weights for adaptation in the updated models. The core idea assumes that we have access to both old and newer weights of a specified model and therefore we can calculate a transfer matrix which we can then utilize to adapt the parameters. They use the Centered Kernel Alignment (CKA) method for layer mapping and the Hungarian algorithm (aka the Kuhn-Munkres algorithm) for attention head mapping. The CKA sequentially aligns corresponding layers to maximize the total similarity, and the Hungarian algorithm

STRENGTHS AND WEAKNESSES:
Strength:
- The paper is well-written and it is easy to understand. The application is technically sound and potentially useful as it can reduce the training costs.
- The method is innovative and tries to tackle an interesting problem.
- The experiments are informative and it can potentially help future research for solving this issue.

Weakness:
- The LoRASuite is an expensive method (in the order of cubic) which might limit the applicability of this method. 
- It has not been clearly explained why they used different methods for head and layer mapping.
- As shown in table 4 and 5, the performance of LoRASuite is not consistent across different datasets.
- Their method doesn’t seem to be applicable to customized architectures.

QUESTIONS:
- Why in figure 4-b, the change of learning rate has a significant impact on the performance of LoRASuite? What’s the justification?

- Why in Algorithm 2, the input to the cka_layer_mapping() is S? S has not been introduced properly.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03