LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Authors: Zihang Liu
Tianyu Pang
Oleg Balabanov
Chaoqun Yang
Tianjin Huang
Lu Yin
Yaoqing Yang
Shiwei Liu
Venue: ICML 2025 poster (2025)
Keywords: Sparse Fine-tuning
Low-Rank Approximation
Memory Efficiency
Primary area: deep_learning->large_language_models
TL;DR: We propose a memory-efficient Sparse Fine-tuning algorithm that selects principle weights by performing low-rank approximation.

Abstract: Recent studies have shown that supervised fine-tuning of LLMs on a small number of high-quality datasets can yield strong reasoning capabilities. However, full fine-tuning (Full FT), while powerful, is computationally expensive and susceptible to overfitting and catastrophic forgetting, particularly when data is limited. Sparse fine-tuning, which previously achieved notable success by updating only a small subset of model parameters, offers a promising trade-off between efficiency and effectiveness. Yet, it has lagged behind in the LLM era due to the difficulty of identifying parameters truly critical for reasoning. In this work, we state that weights with the largest magnitude after low-rank approximation are critical weights for fine-tuning, which we call *Principal Weights*. Surprisingly, while magnitude-based sparse fine-tuning performs poorly as a baseline on LLM fine-tuning, it becomes highly effective after rank reduction. These insights motivate our method: **L**ow-rank **I**nformed Sparse **F**ine-**T**uning ($\texttt{LIFT}$). $\texttt{LIFT}$ only updates the top 5% *Principal Weights* throughout training and consistently achieves better performance on reasoning tasks than Full FT, while maintaining memory efficiency on par with popular parameter-efficient fine-tuning methods.  In addition to strong performance on target domains such as arithmetic reasoning, $\texttt{LIFT}$ also retains up to 20% more source-domain knowledge, compared to Full FT and LoRA. Our code is available at: https://github.com/zihanghliu/LIFT.

LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Pine-Tuuiig
Zihang Liu' Tianyu Pang? Oleg Balabanov 134 Chaoqun Yang² Tianjin Huang 67
Lu Yin 87 Yaoqing Yang?
Shiwei Liu 97
Abstract
1. Introduction
Recent studies have shown that supervised fine- tuning of LLMs on small number of high-quality datasets can yield strong reasoning capabilities. However, full fine-tuning (Full FT), while power- ful, is computationally expensive and susceptible to overfitting and catastrophic forgetting, partic- ularly when data is limited. Sparse fine-tuning, which previously achieved notable success by up- dating only a small subset of model parameters, offers a promising trade-off between efficiency and effectiveness. Yet, it has lagged behind in the LLM era due to the difficulty of identifying pa- rameters truly critical for reasoning. In this work, we state that weights with the largest magnitude after low-rank approximation are critical weights for fine-tuning, which we call Principal Weights. Surprisingly, while magnitude-basee tuning performs poorly as a baseline on LLM fine-tuning, becomes highly effective after rank reduction. These insights motivate our method: Low-rank Informed Sparse Fine-Tuning
Large language models have recently undergone revolutionary advancement in reasoning capabilities through Supervised Fine-Tuning (SFT) (Ye et al., 2025; Muennighoff al., 2025) and Reinforcement Learning (RL) (Guo et al., 2025; Face, 2025). Performing SFT on small, high-quality dataset delivers remarkable reasoning performance on math problems (Muennighoff et al., 2025). However, Full Fine-Tuning (Full FT) is prone to overfitting on limited training data (Chu al., 2025) and incurs substantial computational costs due to the massive sizes of modern LLMs (Yin et al., 2023).
On the other hand, Sparse Fine- Sane-Tuning (Sparse FT) (Guo et al., 2020; Xu et al., 2021; Sung et al., 2021; Sanh et al., 2020a). standout approach for pre-LLM fine-tuning, has demonstrated promising performance by training only a small subset of the base model's ' parameters. However, the adoption of Sparse FT has significantly lagged behind its low-rank counterparts in LLMs, as struggles to identify parameters truly critical to fine-tuning, and its memory over- head is the same as Full FT with irregular sparse patterns.
sparse fine-
Tuning (LIFT). LIFT only updates the top 5% Principal Weights throughout training and consistently achieves bet- ter performance on reasoning tasks than Full FT. while maintaining memory efficiency on par with popular parameter-effiiien fine-tuning methods. In addition to strong performance on target do- mains such as arithmetic reasoning, LIFT also retains up 20% more source-domain knowl- edge, compared to Full FT and LoRA. Our code is available at:
In this paper, we propose Low-rank Informed Sparse Fine- Tuning (LIFT). an effective and efficient approach for feouueedg-focused LLM fine-tuning. LIFT builds on counter-intuitive finding: the most naive baseline for Sparse FT, i.e., magnitude- based fine-tuning, becomes remarkably effective after applying low-rank approximation. We hence identify the weights with the largest magnitude after rank reduction Principal Weights. The process of obtaining Principal Weights is illustrated in Figure 1. Empirical results show that LIFT outperforms state-of-the-ar PEFT methods, Sparse FT methods, and Full FT on a wide range of tasks. LIFT solves the challenges of Sparse FT in these ways:
Univeessity of California, Berkeley, CA, USA 'Dartmouth College, NH, USA 'International Computer Science Institute, CA. USA 'Lawrenee Berkeley National Laboratory, CA, USA 'Tsinghua University, China University of Exeter, Exeter, UK "Eindhoven University of Technology, the Netherlands University of Surrey, Guildford, UK Uniiversity of Oxford, Oxford, UK. Cor- respondence to: Zihang Liu < <zihang. cahhang.iu@berrkleee..dde berkeley.edu>.. Shiwei Liu <shiwei.liu @ maths.ox. ac.uk>.. ac..kkk>>.....
Prior Knowledge: LIFT identifies Principal Weights, which are critical for retaining pre-training knowledge and adapting to downstream tasks. The intuition aligns with recent findings that reasoning capacity is already in base models (Ye et al., 2025; Yue et al., 2025). LIFT further finds that this knowledge is encoded with Principal Weights and fine- tuning only these parameters is sufficient to achieve comparable- or even superior superior-reasoning reasoning performance.
Proceedingy of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025, Copyright 2025 by the author(s).

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Fine- Oin-Tuning
Weight Magnitude
W
4
0
5
8
4
1
2
4
3
2
Selecting Top K highest magnitude of W for finetuning W
2
LIFT
W
Σ
VT
W
W
2
0.1
-33
0.5
0.4
1.2
0.8
100
2.8
8
3
1
0.3
0.0
088
0.1
$6
10.5
0.4
0.7
0.2
1.9
1.1
2.2
0.9
3
-3
1
0
0.0
-044
0.1
0.9
12
10.4
0.1
-04
2.3
0.3
0.5
2.6
2.5
Rank submatrices
Rank approximation
Selecting Top K highest magnitude of W' for finctuning W
Figure I: Overview of LIFT. LIFT first performs SVD on the original weight matrix W to obtain a Rank-r approximation W'. It then selects the top-K parameters of W with the highest magnitudes to create a fine-tuning mask. This mask is then applied to the original weight matrix W for fine-tuning.
Memory Efficiency: LIFT offers significantly better memory efficiency than Full FT and is par with LoRA. LIFT updates and stores only small subset of parameters during fine-tuning, leading to substantial memory savings-particularyy in optimizer states, which reduced from 27GB in Full FT to just 1.3GB (<5%) on LLaMA-2-7B.
Principal Weights crucial for retaining pre-trained
knowledge and adapting to downstream tasks.
We show that LIFT consistently yields strong perfor- mance on diverse sets tasks. We evaluate LIFT on wide range of reasoning benchmarks, including GPQA Diamond, Commonsense Reasoning, Arith- metic Reasoning, and Natural Language Understand- ing. We show that LIFT outperforms PEFT methods, Full FT, and other Sparse FT strate- gies. Specifically, LIFT achieves up 4.42% better performance than LoRA commonsense reasoning, and to 2.02% higher overall performance than Full FT on GPQA Diamond.
Our analysis reveals that Principal Weights are more impor- tant for LLM fine-tuning compared other weight selection criteria: Adding random perturbation Principal Weights drastically affects model performance, substantially greater than other sparse selection metrics, both for pre-training knowledge and downstream tasks. In addition, the update matrix of LIFT has a substantially larger magnitude than LoRA and Full FT, and substantially larger rank than that of LoRA, close to Full FT, enabling a larger capacity to acquire new knowledge in tinettunig. Furthermore, LIFT can strongly affect principal eigenspace of the LLM, making significantly larger deviation than LoRA and Full FT, leading to better adaptation of the downstream tasks. We summarize our contributions as follows:
We provide a comprehensive analysis of LIFT, and show that LIFT has 1) Strong generalization perfor- mance, that it balances learning and forgetting, achiev- ing stronger performance on target domains while up to 20% better than LoRA on source domains; 2) Strong learning capacity, that it enables larger weight up- dates, with rank significantly higher than LoRA, close Full FT, as well as other fascinating observations.
We propose LIFT, a memory-efficient Sparse Fine- tuning algorithm, that selects and fine-tunes Principal Weights, parameters with the highest magnitude after low-rank approximation. LIFT has significantly lower memory overhead than Full FT (less than 5% memory for optimizer), similar that of LoRA. We show that
2. Related Works
Low-rank Approximation of Weight Matrices. Recent works (Sharma et al., 2024; Chen et al., 2025; Wang et al., 2024; Wei al., 2024; Jaiswal al., 2024; Geva et al., 2023)
2

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Fine- TineTTnnig
LIFT
Random
Weight Mag
Prediction Probability (1)
10
70-
Perplexity (
10
60
$0
Test Accuracy
10
50
40
10
30
10
20
10'
10
10
10
0
0
4
8 10
12
14
4
10
12
0
6
8
12
% of Parameters
% of Parameters
% of Parameters
(a) Wikitext Perplexity (1)
(b) Next-token Prediction (1)
(c) Arithmetic Reasoning (1)
Figure 2: Evaluating pre-trained -LLMA-22-7B model with random noise added to selected parameters.
study the low-rank structure within the weight matrices of LLMs. Specifically, Sharma et al. (2024) and Chen et al. (2025) found that low-rank approximation of the feedfor- ward layer improve reasoning capabilities. Sharma et al. (2024) proposes that higher-order components of weight matrix can introduce noise decision-making and eliminat- ing the higher-order components "denoises" the model and helps recover "hidden", less-fregg frequent information, improv- ing the model's performance questions whose answers are supported by less frequently occurring data. Chen et al. 2025 further theoretically confirmed this point in a two-layer transformer setting. The low-rank structures weight up- date also inspire line of adapter-based PEFT methods (Hu et al., 2022; Liu et al., 2024a; Meng al., 2024; Huang et al., 2025).
However, there is lack of metrics that identify critical weights for fine-tuning, making it challenging to scale up. Li et al. (2024) prioritizes layers with significant outliers and fine-tunes models with sparse layers. Recently, Yang et al. (2024) proposed a structured Sparse FT scheme that achieves strong generalization performance. this work, our proposed LIFT identifies the Principal Weights that are crucial model performance.
3. Methodology
In this section, we introduce LIFT in detail. In Section 3.1, we describe the background and foundations for our ap- proach. Section 3.2 presents the detailed algorithm of LIFT. An overview of LIFT is depicted in Figure 1.
Eigenspectrum Analysis in Training and Fine-tuning. Previous works (Ba al., 2022; 2023; Wang al., 2023) demonstrated theoretically that a two-layer neural network initialized with i.i.d. weights trained with one update step tends to exhibit a bulk+spike pattern in the empirical spectral density (i.e., the histogram of eigenvalues), which contains important signals in the corresponding eigenvectors. Recent works (Yang et al., 2023; Martin et al., 2020; Dandi et al., 2024) have investigated the eigenspectrum of weight/fcature matrices the trained model and found that shape metrics related the heavy-tail distribution of eigenspectrum can reflect the quality of the model. Other works (Sanyal al., 2019; Nassar et al., 2020; He & Ozay. 2022; Zhou et al., 2023; Liu et al., 2024b; Lu et al., 2024; Hu et al., 2025; Lingam et al., 2024; Meng et al., 2024) leveraged the infor- mation of the eigenspectrum and designed adaptive training and fine-tuning algorithms that improve generalization.
3.1. Observations on Sparsity and Low-rank Structures
Sparse fine-tuning involves choosing a small subset of Prin- cipal Weights to fine-tune. Intuitively, the Principal Weights should be critical to model performance on different tasks. To determine the critical components in the weight matrices, recent works (Sharma et al., 2024; Chen et al., 2025) pro- pose that lower-order components (corresponding to large singular values) contain information related to the task and context, while higher-order components (with smaller singu- lar values) contain more generic information, which can be considered noise. These noise terms could prevent the adap- tation process when adapting to domain -emeiff-peciffc task in fine-tuning. Based on this insight, we aim to design a sparse fine-tuning algorithm that finetunes parts of the weight ma- trices that contribute significantly to the model when the "noises" are filtered out through low-rank approximation. Essentially, the algorithm should finetune parameters with the largest magnitude after low-rank approximation.
Sparsity in Fine-tuning. Sparse Fine-tuning (Sparse FT) aims to train a small subset of the weight matrices that are critical for downstream tasks, while achieving smaller memory footprint than Full FT (Guo et al., 2020; Xu al., 2021; Sung et al., 2021; Sanh et al., 2020a; Ansell et al., 2024; Song et al., 2024; He et al., 2025). Sparse FT achieved notable success before the era modern LLM.
3.2. LIFT Algorithm
We now introduce LIFT detail. Given a model, for all trainable weight matrices {W,, W2..., W. LIFT first performs rank R approximation of the weight matrices
3

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Fine- Tin-Tunnng
Model
Method
Best Rank
BoolQ
PIQA
SIQA
HellaSwag
Wino
ARC-e
ARC-e
OBQA
Avg.
Full FT
73.8
84.2
81.0
94.7
85.2
88,9
75.6
84.8
83,53
LoRA
128
70.8
82.8
79.4
92.9
83.4
86.3
71.6
82.8
81.25
DoRA
128
71.3
83.4
80.1
92.3
84.0
86.1
71.4
85.8
81.80
LLaMA-2-7B
PISSA
128
72.5
85.3
80.8
87.2
86.1
87.1
74.3
85.6
82,36
S2FT
128
73.3
83.7
81.0
94.3
84.6
88.3
75.8
84.8
83.22
LIFT
128
74.8
84.7
82.2
94.4
86.0
89.2
76.4
89.6
84.66
Full FT
75.4
88.0
81.8
93.1
83.0
86.64
LoRA
64
71.8
85.3
80,9
93.4
84.5
90.0
77.0
84.8
LLMMA-3-B
DoRA
64
74.6
87.4
81.2
94.7
87.1
89.4
79.5
86.4
85.04
S2FT
64
67.7
89.8
82.5
95.2
87.8
93.1
84.6
88.6
86. 16
LIFT
32
75.7
90.5
83.2
96.5
89.4
93.6
83.9
90.2
87.88
Table 1: Commonsense reasoning. Fine-tuning Commonsense-1707 170K dataset.
to obtain (Wi, W W n ), such that
4. LIFT Finds Principal Weights
We now try to provide initial insights into LIFT. We expect that through low-rank approximation, the model weights discard higher-order components (corresponding to smaller singular values) and retain the parts that best represent the encoded knowledge (Ba al., 2022; Chen et al., 2025), leading to LIFT identifying the Principal Weights. To de- termine the importance of weights selected by LIFT, we design a simple experiment to randomly perturb different groups of weights and observe the resulting performance changes. If set of parameters is truly critical to the LLM, perturbing them should cause significant negative impact on the model performance. We empirically show that adding noise to parameters chosen by LIFT significantly affects model performance, compared other selection criteria.
W arg min IW: -ille. (w)Sr
(I)
By obtaining the Rank r approximation, we filter out the "noisy" information in higher-order components, while maintaining the proximity to the original weight matrix according to the Eckart-Young-irsks theorem (Eckart & Young, 1936).
Then, LIFT generates a binary mask, in which positions with the highest magnitude are set to 1, and the rest to O:
Mij = {
if W7y in top-k of W otherwise.
(2)
Experiment Setup. For an LLM, given a subset of parame- ters selected by a criterion, we add a Gaussian noise with a fixed scale of 0.01 to these parameters. Then, we evaluate the perturbed model on three tasks: 1) Wikitext Perplexity, 2) Next-token Prediction, and 3) Arithmetic Reasoning. For 1) and 2), we use pre-trained LLaMAA-277B model to verify Principal Weights's ' importance for attaining pre- trained knowledge, and for 3), we use a LLaMA-2-7B model fine-tuned the MATH-10K dataset verify the importance of Principal Weights on fitting downstream tasks. We compare the performance of the perturbed model under LIFT, and different parameter selection strategies.
where k is the number of chosen parameters in W. The mask is then applied to the original model during fine-tuning. Given any optimization algorithm with stochastic gradient updates, suppose at iteration & the gradient of the weight matrix WT is g: we apply the binary mask M to only store the gradient and corresponding optimizer states of the selected parameters as vectors:
of = vec(o{[M 1))
Wikitext Perplexity. We first evaluate the perplexity of the perturbed model on the Wikitext dataset. As shown in Figure 2a, we see that when LIFT -eeeeeeletedd parameters are added noise, the perplexity increases significantly, while other selection metrics remain stable. This implies that parameters selected by LIFT have a significant influence the model's basic language capabilities.
By storing only the optimizer states of Principal Weights, the memory expenses of optimizers such as Adam are dra- matically reduced. The detailed algorithm can be found in Appendix A. In Section 7.4, we show that the memory over- head of LIFT is significantly lower than Full FT, similar to that LoRA.
In addition, as the low-rank approximation and its largest components will also change, we need to adjust our estima- tion of the Principal Weights dynamically. We choose the update interval that balances effectiveness and efficiency. A detailed discussion is in Appendix B.I.
Next-token Prediction. Recent work (Sharma et al., 2024; Chen et al., 2025) showed that replacing weight matrices with their low-rank approximation improves reasoning, pre- dicting contextual answers instead of "generic" tokens by

---

Principal Weights Emerge after Rank Reduction for Ceasoning-Focusee Supervised Fine-Tuning
Model
Method
Best Rank
MultiArith
GSM8K
AddSub
AQuA
SingleBQ
SVAMP
MAWPS
Avg.
Full
98.50
33.13
92.15
24.02
94.29
51.1
87.40
LoRA
128
98.50
30.48
89.87
24.41
92.91
53.6
86.13
67.98
128
30.48
89.87
92.91
53.6
86.13
LLAMA-3.2-IB
PISSA
32.15
91.90
23.23
52.3
86.97
S2FT
128
96.17
30.17
90.38
23.62
92.13
49.4
81.93
66.26
LIFT
128
98.17
32.37
90.89
26.38
92.91
56.3
86.13
69.02
Fall
97.83
56.71
92.66
33.46
95.08
69.1
90.34
76.45
LoRA
128
98.50
55.95
91.39
27.95
94.29
70.3
91.60
75.71
DoRA
98.33
55.12
91.65
28.35
94.88
70.9
89.92
75.59
LLaMA-3.2-3B
PISSA
98.33
59.51
91.90
25,20
95.47
69.9
89.50
75,69
S2FT
128
98.33
55.65
91.90
29.53
68.9
88.66
LIFT
128
57.92
94.17
28.74
71.0
91.60
Fall FT
98.17
46.55
93.67
22.05
96.85
63.2
89.08
72.79
LoRA
128
98.00
47.76
92.41
23.62
95.08
62.9
90.76
72.93
DoRA
64
98.00
47.38
92.41
21.26
96.06
62.3
89.50
72.42
LLAMA-2-7B
PISSA
128
98.83
48.45
92.66
21.26
95.87
63.4
90.76
73.03
S2FT
128
99.17
44.43
91.39
29.13
95.47
62.6
89.50
73.10
LIFT
128
98.67
47.31
92.66
26.77
96.85
63.6
90.34
73.74
Full FT
99.00
69.83
93.42
28.74
97.83
79.6
92.86
80.18
LoRA
64
99.17
71.57
92.15
24.41
96,26
80.5
92.02
79.44 49
DoRA
64
98.83
70.96
90.89
29.53
96.65
8188
90.76
79.92
LLaMA-3-8B
PISSA
128
99.00
71.27
93.67
28.74
97.64
80.6
92.02
80.42
S2FT
64
99.67
70.89
92.91
32.68
97.64
78.2
94.12
80.87
LIFT
128
99.33
72.40
93.42
34.65
98.03
80.9
93.70
81.78
Table 2: Arithmetic reasoning. Fine-tuned on the MATH- 10K dataset.
the original matrix. Inspired by this, we aim to investigate the role of parameters selected by LIFT on the next-token prediction task. Specifically, we pre-trained LLaMA-2- 7B model and analyze its output probability given prompt such as "Madrid is located in the country of". Given this prompt sentence, the pre-trained model would successfully predict the ground truth answer "Spain". On the other hand, if the model fails to learn the context information, it more likely to behave like an n-gram model and predict generic words like "the". As shown in Figure 2b, after more LIF- selected weights are added noise, the output probability of "Spain"" decays drastically, and eventually becomes zero. While other selection metrics are not influenced by the noise, still predict the correct answer.
5. Experiments
In this section, we evaluate LIFT on various fine-tuning tasks, and compare it with state-of-the-arr fine-tuning meth- ods, including Full FT, LoRA (Hu al., 2022), DoRA (Liu et al., 2024a), Spectral Adapters (Zhang & Pilanci, 2024), S2FT (Yang et al., 2024), and PiSSA (Meng et al., 2024). We also compare LIFT with sparse fine-tuning methods such as SpIEL (Ansell et al., 2024) and SIFT (Song et al., 2024). We demonstrate that LIFT achieves superior per- formance than these methods across all datasets. To ensure fair comparison, we reproduce all results of the baseline methods with the codebases provided in previous papers.
5.1. Experimental Setup
Arithmetic Reasoning. Figure 2c shows the average test accuracy on 7 arithmetic reasoning tasks of the perturbed model. We can that when parameters selected by LIFT are added noise, the performance degrades drastically to 0, compared to other metrics where adding noise doesn't have a significant influence.
Tasks. We conduct experiments domains of major inter- ests modern L.M communities, including: 1) Reasoning Models, in which we perform SFT with Qwen-2.5 on s s dataset (Muennighoff al., 2025) and evaluated on GPQA Diamond (Rein al., 2023); 2) Commonsense Reason- ing, where fine-tune models
The above experiments show that parameters selected by LIFT are extremely crucial to model performance, and they are sensitive to tiny perturbations. Therefore it makes sense to focus on these parameters during funinnuning as they will potentially be more adaptive to downstream tasks, and be more robust after fine-tuning. In Appendix C, provide insights into LIFT from the perspective of spectral norm.
Commonsense--10k dataset (Hu et al., 2023) and evaluate them on eight com- monsense reasoning tasks; 3) Arithmetic Reasoning, in which we fine-tune models on MATH-10K dataset (Hu et al., 2023) and evaluate them on arithmetic datasets; 4) Natural Language Understanding, in which fine-tune and evaluate models on the GLUE datasets (Wang et al., 2018); 5) Code generation, where we perform instruct fine- tuning and evaluate the model on the Humaneval dataset;
5

---

Principal Weights Emerge after Rank Reduction for Reasoning- Focused Supervised Fine-Tuning Tuning
Method
Best Rank
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STSB
Avg.
Full FT
90.22
96.10
89.71
70.67
93.59
92.20
83.03
91.38
88.36
LoRA
64
89.92
95.87
89.95
68.35
92.88
90.62
81.95
90.81
87.57
DoRA
64
89.93
96.22
89.95
68.59
92.86
90.66
82.31
90.83
87.66
Spectral
64
89.89
96.22
88.73
69.65
93.45
91.21
82.67
90.92
87.84
PiSSA
128
90.12
95.87
89.46
68.48
93.48
91.72
81.95
91.10
87.77
LIFT
128
90.49
96.56
90.93
71.84
93.90
92.38
85.92
91.86
89.24
Table 3: Natural language understanding. Fine-tuning DeBERTa-v3 on GLUE datasets
Full FT
LIFT
Weight Mag
Movement
Grad Mag
Random
25
48
43
20
46
41
44
15
(a) LLaMA-3.2-3B
(b) LLaMA-2-7B
(c) Mistral-7B
Figure 3: Comparing different sparse parameter selection metrics GSM8K dataset.
6) Question Answering, where we fine-tune and evaluate models with the Strategy QA dataset.
fine-tuning on the slK dataset.
Metric
Qwen2.5-1.5B
Qwen2,5-3B
Models. For arithmetic and commonsense reasoning, we use LLaMA models with sizes ranging from 1B to 8B (Tou- et al., 2023a;b; Grattafiori et al., 2024). For reasoning models with test-time scaling, we choose the instruction- finetuned Qwen-2.5 (Qwen et al., 2025) 1.5B and 3B models. For natural language understanding, we use DeBERTa- V3- base (He al., 2023) and RoBERTa-large (Liu et al., 2019). For code generation, tion answering, we choose I.L.MA2-777 and I.LAMA---BB.
Full
26.77
33.33
LIFT
28.79 128)
34.85 (r 256)
Table 4: Test accuracies of Qwen 2.5 models GPQA- Diamond, trained with supervised fine-tuning on the s1K dataset.
LLaMA-2-7B model. For ques-
In Table 4, we compare LIFT with Full FT, which is the default method for supervised fine-tuning. We can see that LIFT can achieve better performance than Full FT on Qwen2.5 1B and 3B instruct model. This result shows the potential of LIFT on the training of large scale reasoning models.
To ensure fair comparison, we search the rank of PEFT methods in {16, 32, 64, 128. 256), and LIFT with the same parameter counts, and compare their best results among all the ranks. For detailed dataset configurations and hyperpa- rameter settings, please refer to Appendix D.
5.2. Reasoning Models GPQA Diamond
5.3. Commonsense Reasoning
Recently, reasoning models such DeepSeek R1 (DeepSeek-AL, 2025), and Qwen 2.5 (Qwen et al., 2025) have shown advanced reasoning capabilities by scaling up compute resources during test-time. This trend incentivizes the development of more efficient and effective methods to train these reasoning models. To evaluate LIFT on adapting reasoning models, we follow the settings of the recent s1 paper (Muennighoff et al., 2025), and train instruct-fine fintuut-inntuned Qwen-2.5 models with supervised
As shown in Table 1, LIFT achieves superior results on commonsense reasoning tasks than other fine-tuning meth- ods. When compared to PEFT methods, LIFT outperforms DoRA and PiSSA by 2.86% and 2.30% with LLaMA-2-7B model; When compared to Full FT, LIFT achieves 1.24% and 1.13% higher overall accuracy with LLaMA-3-8B and LLaMA-2-78 respectively. This highlights the effectiveness of LIFT in commonsense reasoning.
6

---

Principal Weights Emerge after Rank Reduction for Bessoning-Focussee Supervised Fine- TineTunnig
Full FT
LIFT
LoRA
44
53
77
40
52
94
76
36
51-
93 32
50
75
28
64
128
32
64
128
32
64
128
32
64
128
Rank
Rank
Rank
Rank
(a) Target Domain (Easy)
(b) Target Domain (Hard)
(c) Target Domain Overall
(d) Source Domain (Common- sense)
Figure 4: Generalization performance of LIFT on Near OOD and Far OOD tasks with LLaMA-3.2-3B.
5.4. Arithmetic Reasoning
Appendix B.I, study the effect of update interval on the performance of LIFT. Finally in Appendix B.2, we com- pare the performance of different rank reduction strategies.
In Table 2, present evaluation results seven arithmetic tasks. We can see that LIFT outperforms all PEFT methods, Sparse FT methods, and Full FT on LLaMA models. Specif- ically, compared to LoRA, LIFT achieves 1.79% higher overall performance LLaMA-3.2-3B In addition, LIFT outperforms the recent S2FT method by 2.76% on LLaMA- 3.2-1B. More significantly. compared to Full FT, LIFT achieves a noticeable improvement of 1.14% and 1.60% on LLLMA-2.77 and ILLMMA3-88B respectively. This sug- gests that LIFT can yield better test performance across wide range of model sizes. Furthermore, LIFT achieves significantly better results on the most difficult tasks, such as GSM8K and SVAMP. This implics that LIFT can obtain high-level arithmetic capabilities more effectively.
6.1. Comparing Different Parameter Selection Metries
To evaluate the superiority of weight selection with LIFT, we compare LIFT with different parameter selection met- ries on the task of fine-tuning on the GSM8K dataset. Specif- ically, chose the pre-trained model of LLaMA-3.2-3B. LLaMA-2-7B, and Mistral-7B (Jiang et al., 2023), and we ran each experiment on four random seeds. The weight se- lection criteria include: 1) Weight Magnitude, 2) Movement Score (Sanh et al., 2020b). 3) Gradient Magnitude, and 4) Random Selection. As shown in Figure 3, we can see that LIFT outperforms all other parameter selection metrics by large margin while surpassing Full Fine-tuning. This sug- gests that LIFT is robust and effective selection metric for fine-tuning, especially challenging tasks like GSM8K.
5.5. Natural Language Understanding
We further evaluate the performance of LIFT on Natural Language Understanding tasks, as shown in Table 3. We can every
6.2. More Ablation Studies
see that LIFT achieves the highest performance task, significantly better than other fine-tuning methods. Specifically, LIFT outperforms Full FT by 0.88% overall, and surpasses the recent Spectral Adapter method by 1.40%.
Due page limits, place more ablation study results in Appendix B. In Appendix B.1 we study the effect of update interval of LIFT on model performance, and in Ap- pendix B.2, we compare different rank reduction strategies.
5.6. Additional Results
In addition to the results provided above, have further evaluated LIFT on more diverse domains. This includes code generation under instruction fine-tuning, and question answering, which are presented Appendix E.2 and Ap- pendix E.3, respectively. We have also compared LIFT with other sparse fine-tuning methods, namely SpIEL (Ansell et al., 2024) and SIFT (Song et al., 2024), on diverse tasks, which are presented in Appendix F.
7. Discussions
In this section, we provide comprehensive analysis on LIFT. First, in Section 7.1, we study how LIFT learns more on the target domain and forgets less on the source domain. In Section 7.2, investigate the weight update of LIFT. Then in Section 7.3, we study the eigenspace and eigenspectrum of model fine-tuned by LIFT to investigate its learning dynamics. In Section 7.4, we analyze the memory efficiency of LIFT. Further discussion results are in Appendix
6. Ablation Study
7.1. LIFT Balances Learning and Forgetting
In this section, perform extensive ablation studies to sup- port the effectiveness of LIFT. In Section 6.1, we compare ATFT with other sparse weight selection methods. Then in
Balancing learning and forgetting is crucial in studying the generalization abilities of training algorithm. Follow-

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Fine- Sine-Tnning
LIFT
Full FT
LoRA
10
modellayers.O.mbll proj
Saa.lllayyrr....l..ooo.. proj
model.layers. 10. proj
model. 10. proj
10º
Count
10
10
10
-0.02 -0.01
0.00 0.01
0.02
-0.02 -0.01 0.00 0.01
0.02
-0.02 -0.01 0,00 0.01
0.02
-0.02 -0.01
0.00 0.01 0.02 Weight Difference
Weight Difference
Weight Differrncee
Weight Diffierence
Figure 5: Weight difference between the model before and after fine-tuning.
ing (Biderman et al., 2024; Yang et al., 2024), we study the learning and forgetting of LIFT with pre-traines LLM fine-tuned on MATH-10K, and evaluating it on both arith- metic reasoning tasks (target domain), including easy (i.e, MultiArith, AddSub, SingleEq, MAWPS) and hard (i.e, GSM8K, AQuA, SVAMP), and commonsense reasoning tasks (source domain). Specifically, we compare the perfor- mance of LIFT with Full FT and LoRA. To ensure a fair comparison, we match the number of trainable parameters of LIFT and LoRA with the corresponding rank.
7.3. Eigenspace and Eigenspectrum Analysis
To analyze how LIFT enables superior fine-tuning results, we investigate the dynamics of eigenspace and eigenspec- trum before and after fine-tuning. Specifically, we compute the 1) alignment score of top eigenspace (right singular vectors) before and after fine-tuning to measure the devia- tions in these directions, and 2) rank of weight update. The definition of alignment score can be found in Appendix H.1
Eigenspace. In Figure 12, we show the alignment score of each layer of LLaMA-3.2-3B model (a larger alignment score means more similar eigenspace). We can see that:
In Figure 4, we show the performance of LIFT with LLaMA-3,2-3B model. We can see that LIFT significantly outperforms Full FT and LoRA both easy and hard tasks. In addition, LIFT also surpasses the source domain per- formance of Full FT and LoRA by a large margin, more than 5% than Full FT and 12% than LoRA. This suggests that LIFT not only able to achieve superior results on the target domain tasks but also can retain previous knowledge, showcasing its ability to forget less. We believe that this strong generalization might come from the fact that LIFT only tunes parts of the weight matrices, keeping a large por- tion of the parameters unchanged, which makes the model forget less. In Figure 10 of Appendix G.1 we also show the performance of LIFT with LLaMA-3-8PP model.
Some layers' eigenspace are extremely robust to fine- tuning. For example, for Query, Key and Gate layer, the alignment score is almost 1. This implies that fine- tuning these layers is less effective, as they don't bring further rotation to the top eigenspace. On the other hand, for Output, Up, and Down layers, the change of alignment scores is 10x larger than the other layers. This indicates that these layers are more adaptive to fine-tuning tasks, as fine-tuning them is more effective in rotating their eigenvectors. In Appendix G.2, we em- pirically demonstrate the difference in the effectiveness of fine-tuning these layers. We note that this observa- tion is also observed in previous works (Sharma et al., 2024; Yang et al., 2024).
7.2. Weight Update
To analyze the changes brought by LIFT to the model, we plot the magnitude distribution of the weight update matrix AW of different layers in the model, as shown in Figure 5. We can see that LIFT brings a significantly larger weight update than Full FT and LoRA. These larger weight up- dates may reflect that the model is actively exploring and exploiting its capacity to capture new task-specific features, improving performance on the fine-tuning dataset. In the meantime, can that with LIFFT, only small set of parameters are changed while most weights remain un- changed (a large spike in the center of LIFT's magnitude distribution), the model retains its fundamental capacities that enable it to generalize OOD settings.
LIFT is especially effective on highly adaptive layers. We can see that for Output, Up and Down layers, LIFT can bring significantly larger rotations to the top eigenspace, resulting in much lower alignment scores than Full FT and LoRA.
Rank. In Figure 13, we show the rank of the update matrix of each layer, grouped by their layer type. We can see that compared to LoRA, LIFT doesn't have rank constraints, and that the rank of the update is significantly higher than LoRA, close to Full FT. On some finetnn--rruiil layers such as Up and Down projection in the MLP module, LIFT achieves almost the same rank update as Full FT on all
8

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focuser Supervised Fine-Tuning
Weight
Activation
Optimization
Weight Gradient
Others
L.LMA-2-7B
LLaMA-3-8B
Full FT
Full FT
LoRA
LoRA
LIFT
LIFT
LIFT MLP
LIFT MLP
20 40
60
25
50
75
Memory Cost (GB)
Memory Cost (GB)
(a) LLaMA-2-7B
LLaMA-3-8B
Figure 6: Breakdown of memory consumption of LIFT, LoRA and Full Fine-tuning.
layers. This suggests that LIFT has a larger capacity to learn task-related knowledge, which could explain its perior performance than other PEFT methods. Details of computing the rank are explained Appendix G.3.
8. Conclusion
In this paper, we propose that parameters with large magni- tude after low-rank approximation of weight matrices Principal Weights for reasoning- on this insight, we designed a memory-efficiem sparse fne-ttnnng algorithm LIFT, that fine-tunes only Princi- pal Weights. We show that LIFT has superior performance than state-of-the-art PEFT methods, Full Fine-tuning, and Sparse FT methods reasoning tasks. From extensive analysis, we find that: 1) LIFT learns more on the target domain while forgetting less the source domain; 2) the weight update matrix of LIFT has a large magnitude and rank, providing large capacity adapt to fine-tuning tasks; and 3) LIFT brings large rotation to the top eigenspace of important layers. We hope that our work can provide in- sights into how to find critical weights fine-tuning L.MSs. In addition, we find that there are few limitations that bring room for further exploration:
fine-tuning, Based
Combining the two metrics, we can see that LIFT serves as good method to 1) provide larger eigenspace rotation adapt fine-tuning tasks, and 2) provide large rank update to increase the capacity of learned knowledge in fine-tuning.
7.4. Memory Efficiency of LIFT
We analyze the memory cost of LIFT. Figure shows the memory overhead breakdown LIFT, Full FT, and LoRA with LLaMA-2-7B and LLaMA-3-4 3-BBB. We can see that the overall memory overhead is just slightly larger than that of LoRA, and significantly smaller than Full FT. Specifically, LIFT only takes up around 5% memory on the optimizer states than Full FT, due to usage of sparse momentum and variance. This suggests that LIFT is able effec- tively balance efficiency and performance. In Appendix G.4, we further show that the memory overhead of LIFT can be further reduced by only fine-tuning MLP layers, while achieving comparable performance (i.e. LIFT MLP).
How combine LIFT with RL algorithms like GRPO enhance the reasoning capacity of LLMs with better memory efficiency?
How does the eigenvector rotation phenomenon of LIFT connect to the learning dynamics of LLM fine- tuning?
7.5. More Discussions
We provide further discussions on LIFT in Appendix G. In Appendix G.5, create two-layer model on simple regression task to simulate the fine-tuning with LIFTT, and show that LIFT has stronger generalization abilities than Full FT. In Appendix G.6, we analyze the training loss curve of LIFT. In Appendix G.7, we explore the potential of LIFT for structured sparse fine-tuning, In Appendix G.8, we analyze the influence of different ranks for rank reduction in LIFT. In Appendix G.9, we investigate the pattern of weights selected by LIFT compared other methods.
Can LIFT be improved with GPU acceleration to fur- ther improve computation efficiency?
Currently LIFT uses a global rank to perform LRA. However, different layers have different capacities. Can we improve LIFT by designing adaptive rank reduction on each layer?
We hope that these problems inspire future research on the development of more effective fine-tuning approaches.
9

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Impact Statement
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika. 1(3):211-218, 1936.
This paper presents work whose goal to advance the field of Machine Learning and Deep Learning, especially the fine-tuning of Large Language Models. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Face, H. Open rl: A fully open reproduction of deepseek- r1, January 2025. URL https: / //g:://gitt /ttpt/tttbtb..//////// com/
References
Galanti, T., Xu, M., Galanti, L., and Poggio, T. Norm-based generalization bounds for sparse neural networks. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http://wwwwe..eeefffoooe...ooooo http///rrmmmmmccPP///// Ohz.
Ansell, A., Vulié, I., Sterz, H., Korhonen, A., and Ponti, E. M. Scaling sparse fine-tuning to large language models, 2024.
/ //openreview.. net
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. figh-ddmmensional asymptotics of feature learn ing:
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dis- secting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023.
How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35: 37932-37946, 2022.
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., and Wu, D. Learning in the presence of low-dimensional structure: A spiked random matrix perspective. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., and Others. The llama herd of models, 2024. URL https: // a arxiv. www.abssrgb/////// 2407 21783.
Guo, D., Rush, A. M., and Kim, Y. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020.
Bartlett, P., Foster, D. J., and Telgarsky, M. Spectrally- normalized margin bounds for neural networks, 2017. URL https::tp/// / //tpivaaaavvv//a////a//aa/aawa.a org/ abs / 706.. 08498.
Guo, D., Yang, D.,, Zhang, H., Song, Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Biderman, D., Portes, J., Ortiz, J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J.. Blakeney, C., and Cunningham, J. P. LoRA learns less and forgets less. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: openreview net http:/orrmmnddaloorrrueeee Fea- tured Certification.
He, B. and Ozay, M. Exploring the gap between col- lapsed & whitened features in self-supervised learning. International Conference on Machine Learning, pp. 8613-8634. PMLR, 2022.
Chen, L., Bruna, J., and Bietti, A. Distributional sociations vs in-context reasoning; A study of feed- forward and attention layers. In The Thirteenth International Conference on Learning Representations, 2025, URL https::////. httph::/topteeeeee httpss///peeervieeeeeee wwt.:/eeetffrrttr id=WCVMgRHWR5.
He, H., Li, J. B., Jiang, X., and Miller, H. SMT: Fine- tuning large language models with sparse matrices. In The Thirteenth International Conference on Learning Representations, 2025. URL tatttps://ppnreeee
Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q V., Levine, S., and Ma, Y. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, arXiv preprint 150.::5001.116.1. 2025.
He, P., Gao, J., and Chen, W. DeBERTav3: Improving de- BERTa using ELECTRA-style pre-training with gradient- disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, 2023. URL https:: htttp:://openrevieww net / forum? id=sE7
Dandi, Y., Pesce, L., Cui, H., Krzakala, F., Lu, Y. M., and Loureiro, B. A random matrix theory perspective on the spectrum of learned features and asymptotic gener- alization capabilities. arXiv preprint arXiv:2410. 18938, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y.. Wang. S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: /atpss////peeeeeee/e net forum? id=nZeWKeeFYf9.
DeepSeek-AN AI. Deepseek-rl: Incentivizing reasoning ca- pability in ilms via reinforcement learning, 2025. URL https: /aaxivvv. wwg/abs////eeeooooeoo. abs/2501 12948.
10

---

Principal Weights Emerge after Rank Reduction for Ceasoning-Foousse Supervised Fine- Fine-Tuning
Hu, Y., Goel, K., Killiakov, V., and Yang, Y. Eigenspec- trum analysis of neural networks without aspect ratio bias.
Liu, Z., Hu, Y., Pang. T., Zhou, Y., Ren, P., and Yang, Y. Model balancing helps low-data training and fine- tuning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1311-1331, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi: 10. 20.16633///22244..mmmmmmm...... aclanthology wrg/ 2024. e0022..mmngg.. www.lmmainnn...e.emaaamaammmmmam mao-mmmin..... aeaai..8// 78/.
In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. URL https: //i fimml.... cc/vi rtual /2025/22/222 https//poseer/////
Hu, Z., Wang, L., Lan, Y. Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Lee. R. LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, Singapore. December 2023. Association for Computational Linguistics. doi: 10. 00.18653/11/0233. emnlp- main..1..... URL https: http://caaatthhooc org/2023. emnlp-maae maapppmmin....m 319/.
URL https://
Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang. Y. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: oppnrevvewwwweeeeeeeeeeeee net forum? 1d=fHq4x2XXVV.
Huang, Q. Ko, T., Zhuang, Z., Tang, L., and Zhang, Y. HiRA: Parameter-efficient hadamard high-rank adap- tation for large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps://operrrevieet net forum? id-Tw3rTz9cRS.
Martin, C. H., Peng, T., and Mahoney, M. W. Predict- ing trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12, 2020. URL https: / //api //api.semante 211132500.
Jaiswal, A., Yin, L., Zhang, Z., Liu, S., Zhao, J., Tian, Y., and Wang, Z. From galore to welore: How low-rank weights non con-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.1 1239, 2024.
Meng, F., Wang, Z., and Zhang, M. PiSSA: Princi- pal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /ppp/eveeeeeeeeeeeeeeeeeee net./ forum? id=62BHIEtdP4.
Jang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, I.., Lavaud, I. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023, URL https: //arxiv. orgo abs/ 22100.3100.. 0 0625.
Muennighoff, N., Yang. Z., Shi, W., Li, X. L.,, Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L.,, Liang, P., Candès, E., and Hashimoto, T. s1: Simple test-time scaling, 2025. URL https: / wwwxxiiorggg abs/ 2501. 19333.
Li, P. Yin, L., Gao, X., and Liu, S. Owlore: Outieer-eiihed layerwise sampled low-rank projection for memory-efficient IIm fine-tuning. arXiv preprint mxXv:24405.18380 2024.
Nassar, J., Sokol, P., Chung, S., Harris, K. D., and Park, I. M. On 1/n neural representation and robustness. Advances in neural information processing systems, 13:6211-6222 2020.
Lingam, V., Neerkaje, A. T., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J., Choi, E., Dimakis, A., Bojchevski, A., and sujay sanghavi. SVFT: Parameter-efficient fine-tuning with singular vectors. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICH @CML 2024), 2024. URL https:// openreview net net//orumm forum? Ld=DouskkCqgs..
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based capacity control in neural networks. In Grünwald, P., Hazan, E., and Kale, S. (eds.). Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 1376-1401, Paris, France, 03-06 Jul 2015. PMLR. URL htts://: ptttttss:/ppoceeeee http///peeeennn..eeee/ee/ee/.eee mlr. ptt.s.pee////// Neyshabur15 html.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-. F, Cheng, K.TT., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024a.
Neyshabur, B., Bhojanapalli, S., and Srebro, N. A pac-bayesian approach to spectrally-normalized mar- gin bounds for neural networks, 2018. URL https: / //arxiv.orr abs 1707. 095644
Liu, Y. Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov. V. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: //a /ate:i///xiivvvvi www.//rg//bb.........//.oeeee..e /tt.s////...... 11692.
Qwen, :, Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., Lin, H.,

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J.. Dang, K., Lu, K., Bao, K., Yang. K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y. Cui, Z., Zhang, Z., and Qiu, 7. Qwen2.5 technical report, 2025. URL https: //arxiv. httg/bbb/// /412 15115.
Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X.,, Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., 7arov, I., Zhang. Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https://aat http://aaxvvv...//ee org/ abs 2207/23772220 09288.
Rein, D., Hou, B., Stickland, A., Petty, J., Pang, R., Dirani, J., Michael, J., and Bowman, S. Gpqa: A graduate- level google-prooe proof q&a benchmark, nov. arXiv preprint arXiv:2311.1200 2023.
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Linzen, T., Chrupata, G., and Alishahi, A. (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 353- 355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 18653vv\v W18-544 W18-5446. URL https: pettps:/aaclln http://laaaaattloa............aa htwpchoooogyogoggoo/ooo W18-5446/.
Sanh, V., Wolf, T., and Rush, A. Movement prun- ing: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33:20378-20389 2020a.
Sanh, V., Wolf, T., and Rush, A. M. Movement pruning: Adaptive sparsity by fine-tuning, 2020b. URL https: // //arxiv..rr
07683.
Sanyal, A., Torr, P. H., and Dokania, P. K. Stable rank normalization for improved generalization in neural net- works and gans. arXiv preprint Set.::990004659, 2019.
Wang. H. Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter-efficient IIm finetuning. arXiv preprint arXiv:2406.09044 2024.
Sharma, P., Ash, J. T., and Misra, D. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In The Twelfth International Conference on Learning Representations, 2024. URL ptttps::/opeereeeee wtwp:e/wwww.eee.eeeeeeeeeeeeeeee wwt.:eettoouuaeee id=ozX92bu8VA.
Wang. Z., Engel, A. W., Sarwate, A., Dumitriu, I., and Chiang, T. Spectral evolution and invariance in linear- width neural networks. In Thinty-seventh Conference on Neural Information Processing Systems, 2023.
Song. W., Li, Z., Zhang, L., hai zhao, and Du B. Sparse is enough in fine-tuning pre-trained large language mod- cls. In Forty-first International Conference on Machine http///peeeew....m net/
Wei, B., Huang, K., Huang, Y., Xie, T., Qi, X., Xia, M., Mittal, P., Wang, M., and Henderson, P. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint 1621 2024.
Learning. 2024. URL https: wttptps://ppeeneee
Srrumiid=1Chu2DD1
Xu, R., Luo, F., Zhang. 7.,, Tan, C.,, Chang, B., Huang, S., and Huang, F. Raise a child in large language model: Towards effective and generalizable fine-tuning, arXiv preprint wxiv:2109.05687, 2021.
Sung. Y.-L., Nair, V., and Raffel, C. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193- 24205, 2021.
Yang, X., Leng, J., Guo, G., Zhao, J., Nakada, R., Zhang, L., Yao. H., and Chen, B. SS (2) Efficient, scalable and generalizable LLM fine-tuning by structured spar- sity. In The Thihty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / //opneevview net www.mrirrrmrrrrr wtw.mmmmmdd1E1114444414
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A.. Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https: /ttpei///aavvvii/ org/ abs/ 2302. 13971.
Yang, Y., Theisen, R., Hodgkinson, L., Gonzalez, J. E., Ramchandran, K., Martin, C. H., and Mahoney, M. W. Test accuracy vs. generalization gap: Model selection in nip without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 3011-3021, New York. NY, USA. 2023. Associa- tion for Computing Machinery. ISBN 9798400701030.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel. D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G.. Esiobu, D., Fernandes, J., Fu, J. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.,
12

---

Principal Weights Emerge after Rank Reduction for Crasoning-Focuseee Supervised Sin-TTnnigg
doi: 10.11455358000555595518. URL hww.h:tp//////t //di..
Ye, Y., Huang, Z., Xiao, Y. Chern, E., Xia, S., and Liu, P. Limo: Less is more for reasoning. arXiv preprint
arXiv.2502.03387, 2025.
Yin, L., Wu, Y., Zhang, Z., Hsich, C.-Y., Wang, Y., Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., et al. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05115. 2023.
Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Zhang, F. and Pilanci, M. Spectral adapter: Fine- tuning spectral space. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: http::://///eeeeeee http:///ppeeeeeeeeeeeeeee wtw.lewwwee//frrri id=Uoxua0GV6B.
Zhou, Y., Pang, T., Liu, K., charles h martin, Ma- honey, M. W., and Yang. Y. Temperature balane- ing, layer-wise weight analysis, and neural network training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL https: //openreview..o net mnte/forume ttwe:/oooooddoooo9FF....... d=3o
13

---

Principal Weights Emerge after Rank Reduction for Reasoning-Foeusse Supervised Fine-Tume Tuning
A. LIFT Algorithm Detail
Algorithm Adam with LIFT
1: Input: 8, a, 31, 82 E [0, 1). 0,
2: mo f- 0, Ub - 0, t + 0, M + calc mask((0)
3: Mo.update - (ec VU.update +- vec(()
4: 1,
do
5: i mod update mask interval 0 then
6: Moia f M, MM + (8) binary mask]
7:
11-1,1 1 f 0,0
8:
1m-1|Ma = 1] = m Lepdate
9:
Ut-1 Moldd = 12- Lupdate
10:
my L.ppdate vec ( 11)
11:
Ut update = vec 1 1])
12:
end if
9t 4- (ofe(01-1)
Ituudate vec(gi = 1])
15:
mupuuttte - B1 11m uupdatte + (1 - B1 uppdote
16:
ULauddaee - 32 Lupdate + (1- 32)g uppaaee
17:
Muaudate Maupdate Vtaupaate & 1-03
Vt.update
1-9
18:
O2 + 1C-1, vec(O,[M = =1] t vec(Or = 1}) a
the updats
V Drapdete
19:
end for
LIFT adapter method. We note that LIFT selects fixed number of parameters to fine-tune at any training step. However, this does not guarantee LIFT has fixed overall sparsity the total count of parameters updated across the entire process can still fluctuate. Nevertheless, empirical evidence suggests that the total update matrix remains sparse (Figure 5),
To obtain fixed-size adapter like in LoRA, one can construct the update mask an accumulative manner, by gradually adding principal weights to already-chosen parameters until designated sparsity. Additionally, the principal weights and fix them during fine-tuning. Such new versions of LIFT can have better portability and adaptability, and represent a promising direction for future research.
pre-determine
B. More Ablation Studies
B.1. Analyzing the update interval of LIFT
LIFT uses dynamic scheme to update the selected parameters. To investigate the impact of update interval on model performance, we compare different update intervals, ranging from 50 to 1000, and compare the performance of LLaMA-2-7B model on the GSMSK dataset. Figure 7a shows the result. We can see that different intervals all significantly outperform the baseline, indicating the robustness of LIFT. Moreover, the interval should neither be too small nor too large, and median interval is the best choice. This aligns with empirical insight as smaller interval would change the fine-tuned weights more frequently, making some weights not fully trained, and from the perspective of compute efficiency, it would be also efficient we don't update the mask too frequently. On the other hand, not changing the selected weights also inferior as some weights may be already saturated in terms of training quality, and the low-rank approximation of the weight matrix also changes during training, therefore the Principal Weights also changes. Therefore, choosing an update interval that is not small and not too large can best benefit the performance of LIFT.
B.2. Comparing Different Rank Reduction Strategies
To prove that rank reduction by preserving the largest singular values and singular vectors (or ranks) is the best choice, compare it with other rank reduction strategies, including randomly preserving the rank (Random), preserving the smallest rank (Smallest), and preserving a combination of largest and smallest ranks (Hybrid). The results are shown in Figure 7b. Specifically, we choose the setting of fine-tunins fine-tuning LLaMA-2-7B model on the MATH-10P dataset and compare the average
14

---

Principal Weights Emerge after Rank Reduction for Ceasoning-Foeusse Supervised Fine- Tin-Tuning
LIFT
25.5
Full FT
82
24.5
81
23.5
80
22.5
79
50
100
200
250
500
1000
Largest Random
Hybrid
Smallest
Update Interval
(a) Update Interval
(b) Rand reduction strategies
Figure 7: More ablation results. (a) Comparing GSM8K accuracy of different update interval of sparse mask chosen by
LIFT. (b) Comparing different rank reduction methods on arithmetic reasoning tasks (mean accuracy).
performance on 7 tasks. We can see that LIFT, by preserving the largest ranks, yields significantly better performance than other rank reduction strategies.
C. More Intuitions on LIFT
In this section, we further analyze LIFT from spectral norm perspective. The spectral norm of a matrix represents its largest singular value, which is the largest "stretch" of the transformation by the weight matrix input data. Several theoretical works have also linked spectral norm to the generalization performance of the model (Neyshabur al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018; Galanti et al., 2023). Here we show that the update brought by LIFT can dramatically influence the spectral norm of the weight matrix, compared to other sparse update methods. We first analyze the influence of LIFT on random matrix. Then, we investigate the change of spectral norm corresponding Section 4.
C.1. Random Matrix Case
To begin, we consider adding noise to LIFT selected by LIFT on random matrix with different dimensions. Figure shows the results of the spectral norm and Frobenius weight selection strategies. We can see that all methods don't ton't differ from the Frobenius norm perspective. However, when it comes to spectral norm, while random selection and selection by weight magnitude doesn't 't change the spectral norm much, LIFT significantly increases the spectral norm, especially when the matrix size large. Weight Mag
of the matrix before and after adding the noise, with different
Original
LIFT
Random
250
2000
200
1500
150
1000
100
500
50
0
10"
10"
10"
10'
Dimension
Dimension
(a) Spectral Norm
(b) Frobenius Norm
Figure 8: Spectral Norm and Frobenius Norm of random matrices of different dimensions after adding random noise selected weights.
15

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focusee Supervised Fine- Sine-Tnning
C.2. Adding Noise LLM
Following the experiments in Section we analyze the spectral norm of pre-trained LLM after adding random noise weights selected by LIFT, compared to other sparse selection strategies. In Figure 9, we show the difference in spectral norm before and after adding the random noise with fixed scale of 0.1 to the LLaMA-2-77 model.
LIFT
Random
Weight Mag
Query
Key
Value
Output
1.4
A AMw
1.2
4.0
4.0
1.2
3.5
1.0
0.8
2.5
0.6
0.6-
2.0
0.4
0.4
1.0
0.21
10
15
20
25
30
0
5
10
15
20
25
30
10
15
20
25
30
10
15
20
25
30
Layer Index
Layer Index
Layer Index
Index
Gate
1.p
Down
2.5
3.5
2.0
3.0
2.5-
2.0
3.0
1.5
1.0
0
5
10
15
20
25
30
10
15
20
25
30
0
5
10
15
20
25
30
Layer Index
Lay er
Layer Index
Figure 9: Spectral norm difference between before and after adding random noise to selected weights pre-trained LLaMA-2-7B model.
We that LIFT brings significantly larger change to spectral than other selection strategies. This indicates that the effect of LIFT on the spectral norm of weight matrices is consistent not only in toy settings as in Appendix C.I, but also in practical settings of LLMs.
D. Detailed Experimental Setup
In this section, provide detailed experimental setup of LIFT of the main results. First, in Appendix D.1 we describe the detailed hyperparameters of experiments in Section 5. Then,
D.1. Detailed Hyperparameters
The following tables corresponds to the main results in the paper. Table 5 corresponds to the experiments shown in Table 1; Table 6 corresponds to the experiments in Table and 11; Table 7 corresponds to experiments in Table
Hyperparameters (LIFT)
LLAMA-2-7B
LLAMA-3-8B
Optimizer
AdamW
LR
le-4
2e-4
LR Scheduler
Linear
Batch size
16
Warmup Ratio
0.03
Epochs
Table Hyperparameter configurations for different LLaMA models on Commonsense- 170K dataset. Corresponding experiments in Table 1.
16

---

Principal Weights Emerge after Rank Reduction for Sesooninn--ocusse Supervised Fine- Tin-Tuning
Hyperparameters (LIFT)
LLAMA-332-1B
LLAMA-32-3B
LLAMA-7B
LLAMA-2-7B
LLAMA-3-8B
Optimizer
AdamW
LR
2e-4
le.4
2e-4
le-4
2e-4
LR Scheduler
Linear
Batch size
16
Warmup Ratio
0.03
Epochs
Table 6: Hyperparameter configurations for different LLaMA models on MATH-IOK dataset. Corresponding to experiments in Table and 11.
Hyperparameters (LIFT)
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STSB
Optimizer
AdamW
Epochs
1
5
13
8
2
10
30
LR
{6e-5, le-4, 2c-4, 3
LR Scheduler
Linear
Batch Size
32
Table 7: Hyperparameter configurations for different LLaMA models GLUE datasets. Corresponding experiments in Table
D.2. Searching Process for Number of Trainable Parameters
As shown in Section 5, we report the best result each method with number of training parameters equivalent to LoRA rank in range (16, 32, 64, 128, 256). Here we present the rank search process and provide detailed results of the performance of different methods under all ranks. Specifically, Table 8 is the rank search result of experiments in Table 1; Table 9 is the rank search result of experiments in Table 2; Table 10 is the rank search result of experiments in Table 3.
LoRA Rank
16
32
64
128
256
Full FT
86.64
86.64
86.64
86.64
86.64
LoRA
76.23
81.86
83.46
83.21
82.63
S2FT
81.07
85.03
86.16
83.82
82.76
LIFT
86.93
87.88
87.18
84.16
83.67
Table 8: Mean test performance 8 commonsense reasoning tasks of different methods on LLaMA-3-8B with number of trainable parameters equivalent to various LoRA ranks. Bold numbers are the best test results of different methods. Corresponding to experiments Table
LoRA Rank
16
32
64
128
256
Full FT
72.79
72.79
72.79
72.79
72.79
S2FT
67.78
71.78
72.48
73.10
72.63
PiSSA
71.57
71.82
72.54
73.03
72.54
DoRA
71.10
71.74
72.42
71.83
71.81
LoRA
70.91
71.74
72.81
72.93
72.24
LIFT
70.91
71.09
72.74
73.74
73.67
Table 9: Mean test performance on 7 arithmetic reasoning tasks of different methods on LLaMA-2-7B with number of trainable parameters equivalent to various LoRA ranks. Bold numbers are the best test results different methods. Corresponding experiments in Table
17

---

Principal Weights Emerge after Rank Reduction for Sestoninn--ocusse Supervised Fine- Tine-Tnnng
LoRA Rank
16
32
64
128
256
Full FT
88.36
88.36
88.36
88.36
88.36
LoRA
87.45
87.51
87.57
87.43
87.32
DoRA
87.35
87.56
87.66
87.45
87.41
PiSSA
87.13
87.21
87.40
87.77
87.33
Spectral
87.76
87.79
87.84
87.51
87.38
LIFT
85.17
86.93
88.19
89.24
88.74
Table 10: Mean test performance on GLUE tasks of different methods DeBERTa-v3-bass with number of trainable parameters equivalent to various LoRA ranks. Bold numbers are the best test results of different methods. Corresponding to experiments in Table 3.
E. More Experimental Results
E.1. Arithmetic Reasoning
To complement the arithmetic reasoning results Figure 11, we show the test performance of LIFT on another model LLaMA-7B, compared with the same fine-tuning algorithms.
Model
Method
Best Rank
MultiArith
GSM8K
AddSub
AQuA
SingleFQ
SVAMP
MAWPS
Avg.
FT
98.50
42.68
91.90
22.83
95.08
59.5
89.50
71.43
98.50
40.49
92.15
24.80
95.28
$7.2
128
98.17
41.77
92.66
25.98
96.26
59.5
87.81
71.74
LLaMA-7B
LoRA
97.67
43.29
90.38
22.83
95.08
60.8
84.87
70.70
DoRA
98.17
43.59
91.90
23.23
95.08
60.1
86.
71.17
LIFT
256
98.17
42.15
92.41
23.62
95.47
62.7
87.82
71.76
Table 11: Arithmatic reasoning performance on LLaMA-7B model. Fine-tuned the MATH- 0K dataset.
E.2. Code Generation
To test the code generation performance of LIFT, adopt the settings from the recent SIFT paper (Song al., 2024), where we fine-tune LLaMA-2-7B with the Alpaca dataset for one epoch, and evaluate the Humaneval dataset. From the table below, we can see that LIFT outperforms all other methods, in both pass@1 and pass @ 10 settings.
L.FT
Full FT
SIFT
LoRA
DoRA
Pass @
16.46
15.24
14.02
13.66
13.96
Pass 10
31.10
28.05
30.48
27.44
29.88
Table 12:
Code generation performance of LIFT and different PEFT methods evaluated on the Humaneval dataset (1), after training with the Alpaca dataset.
E.3. Question Answering
To evaluate the performance of LIFT on question-answering tasks, we adopt the experimental setup of the Strategy dataset following recent WeLore paper (Jaiswal et al., 2024). For PEFT methods, we consider ranks { (16. 32, 64, 128, 256): for LIFT, we use the same counts of trainable parameters. The learning rates are {le-5, 2c-5, 5e-5, lc-4] for Full FT and (5e-5, le-4, 2e-4, 5e-4) for others. We select the best best-performing config for each method and report the results below. We can see that LIFT achieves notable performance gains than all other methods, on both LLaMA-2-77 and LLaMA-3-8B model.
18

---

Principal Weights Emerge after Rank Reduction for Saaooninn--ocusse Supervised Fine- TineTTnnng
LIFT
Full FT
LoRA
DoRA
PiSSA
LLaMA-2-7B
72.53
70.61
71.78
71.98
71.26
LLaMA-3-8B
75.85
74.81
74.44
74.27
75.19
Table 13: Question answering performance (f) of LIFT and different PEFT methods evaluated on the StrategyQA dataset.
F. Comparison with Other Sparse Fine-tuning Methods
In this section, we further support the empirical success of LIFT by comparing with recent sparse fine-tuning algorithms, including SpIEL (Ansell et al., 2024) and SIFT (Song et al., 2024).
F.1. Comparing LIFT with SpIEL
We compare LIFT with SpIEL on the GSM8K dataset. We use the same training setting as Figure 3 in our paper. For both methods, we searched learning rate among (5e-5, le-4, 2e-4, 5e-4] and trainable parameters same LoRA rank among {16, 32, 64, 128, 256} } to obtain the best results. The table below shows that LIFT significantly outperforms SpIEL with both LLaMA-2-7B and LLMMA-3.2-3B
LIFT
SpIEL
Full FT
LLaMA-3.2-3B
46.46
43.76
44.50
LLMMA-3-BB
24.24
21.61
22.57
Table 14: Test accuracies on GSM8K dataset LIFT, SpIEL, and Full FT.
F.2. Comparing LIFT with SIFT
We compare LIFT with SIFT with two experimental setups following the SIFT paper. We use 1) Instruct tuning, where we fine-tune LLaMA-2-7B with the Alpaca dataset for one epoch, and evaluate the Humaneval dataset, which is discussed in Appendix E.2, and 2) Natural Language Understanding, where we fine-tune the RoBERTa-large model on GLUE datasets. For both methods, we use the same number trainable parameters (5% total parameters). We use the RoBERTa-large model, and search the learning rate of LIFT SIFT in {5e-5, 7e-5, Ic-4, 2c-4] and compare their best results. The table below shows that LIFT outperforms SIFT all GLUE tasks, while outperforming Full FT on almost all tasks.
Method
# Param (%)
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STSB
Avg.
Full FT
90.58
96.22
91.91
68.55
94.47
91.52
85.92
92.21
88.92
SIFT LIFT
5%
89.91
96.79
89.95
66.29
93.04
88.49
87.00
92.27
87.97
5%
90.79
96.67
90.93
70.44
94.69
92.38
87.00
92.58
89.44
Table 15: GLUE results comparing LIFT, SIFT, and Full FT using RoBERTa-large.
G. Complementary Discussions
G.I. More Generalization Results
In addition the results shown in Figure we present the generalization results on the target and source domain of LLaMA-3-8B model in Figure 10. We can see that LIFT still achieves significantly stronger generalization results than Full Fine-tuning and LoRA fine-tuning, especially in hard target domain tasks (Figure 10b). In addition, we can see in Figure 10d that LIFT is able to generalize to source domain extremely well, achieving more than 10% better result than Full Fine-tuning, and almost 20% better than LoRA.
19

---

Principal Weights Emerge after Rank Reduction for Bessoning-Focussee Supervised Fine- Tune-Tnnng
Full FT
LIFT
LoRA
62
50
96
45
61
40
60
80
35
95
59
30
79
16
32
8
16
32
8
16
32
16
32
Rank
Rank
Rank
Rank
(a) Target Domain (Easy)
(b) Target Domain (Hard)
(c) Target Domain Overall
(d) Source Domain (Common-
sense)
Figure 10: Performance of LIFT on target domain (arithmetic reasoning) and source domain (commonsense reasoning)
G.2. Model Component Analysis
Following Section 7.3, we see that fine-tuning some layers is ineffective in adapting downstream datasets. Therefore, we further study the effectiveness of LIFT of fine-tuning different model components. Specifically, in each experiment, only fine-tune one layer type of the model, chosen from (Key, Query, Value, Output, Gate, Up, Down). We evaluate the performance on arithmetic reasoning tasks with ILaMA-2-7B model. The results are shown in Figure 11.
Query
Key
Value
Output
Gate
Up
Down
98
2R
96
26
40
94
24
92
22
90
85
MultiArith
(b) GSM8K
(c) AddSub
(d) AQuA
95
64
88
93
71
91
69
89
56
84
67
82
65
(c) SingleEg
(1) SVAMP
(g)
MAWPS
(h) Average
Figure 11: Average performance on arithmetic reasoning tasks when applying LIFT with low-rank approximation of different ranks.
We that compared to other modules, fine-tuning only the Query and Key module yields significantly worse results on all tasks. In contrast, fine-tuning modules such as Value, Up, and Down modules achieves strong performance. This phenomenon is similar to the observation on the alignment of top eigenspace in Section 7.3, where the top eigenspace of Query and Key modules does not change much during fine-tuning, while Output, Up, and Down modules change relatively more significant. These observations could suggest that Query and Key modules are not very adaptable downstream tasks, or that fine-tuning them may not be effective. This could be because the Key and Query components, as part of the self-attention module, mostly store information related to token relations, rather than task task-specific knowledge. On the other hand, modules including the Output, Up. and Down projection layers mostly in the MLP module, tend to be more adaptive fine-tuning, and is able store more task-specific knowledge.
20

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine- TineTTnnig
G.3. Eigenspace and Eigenspectrum Analysis
Figure 12 and 13 shows the results corresponding the discussion in Section 7.3, Figure 12 presents the alignment score between the top eigenspace before and after fine-tuning, comparing LIFT, LoRA and Full Fine-tuning. Figure 13 shows the rank of the weight update matrix of LIFT, Full Fine-tuning and LoRA.
Rank computation. To compute the ranks of different methods in Figure 13, we use torch. linalg matrixx.aat function from PyTorch. It counts the number of singular values greater than threshold 7, which has the default value:
max (m, n) max X
where (m, n) the matrix shape, max the largest singular value, and precision of input data type. Since robust rank comparison requires the threshold to exceed the rounding error incurred during the update matrix evaluation, set it to 10 times the default value.
LIFT
Full FT
LoRA
Query
Key
Value
1.000
88g88oo Ba Bagg
1.000
1.000
88882
0.999
0.995
0.998
0.999
0.990
0.997
0.996
0.998
0.985
0.995
0.980
0.994
0.997
5
10
15
20
25
0
5
10
15
20
25
0
10
15
20
25
Layer Index
Layer Index
Layer Index
Up
Down
1.000
1.000
0.998
0.998
0.996
0.996
0.994
0.994
0.992
0.992
0.990
0
5
10
15
20
25
0
5
10
15
20
25
Layer Index
Layer Index
Figure 12:
Alignment score of the top eigenspace of the weight matrix before and after fine-tuning on LLaMA-3.2-3B model. lower alignment score indicates a larger deviation from the original eigenspace.
LIFT
LoRA
Query
Key
Value
Up
Down
4000
3500
3000
Rank
2500
2000
1500
1000
500
100000000
Www.0000000000000 WWW.
10000000000000 00000000000000000000000000000000 COM WWV 2000
500500500520005000 WWW.0000000000000 SOA
5
10
15
20
10
15
20
25
30
10
15
20
25
30
20
25
30
0
5
10
30
Layer Index
Layer Index
Layer Index
Layer Index
Layer Index
Figure 13: Rank the weight update matrix of each layer of the LLaMA-2-78 model trained on MATH-10K dataset.
G.4. Further memory saving of LIFT
In Appendix G.2, we show that fine-tuning MLP layers more effective than fine- tuning attention layers. Table 16 shows the results of LLLMM-2-7B on arithmetic datasets where we only fine-tune the ML. layers, LIFT MLP. We can see that
21

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focuset Supervised Fine-Tuning
LIFT.MLP has similar performance LIFT. From Figure 6, we can see that LIFT_MLP further reduces the memory usage on gradients and optimizer states, which achieves better memory efficiency than LoRA under optimal rank settings. We realize that selecting appropriate layers for LIFT fine-tuning is crucial problem for future research, as it provides insights on the importance of different layers during fine-tuning, and can potentially further reduce the memory overhead.
Method
Rank
MultiArith
GSM8K
AddSub
AQuA
SingleBQ
SVAMP
MAWPS
Avg.
LIFT
128
98.67
47.31
92.66
26.77
96.85
63.60
90.34
73.74
LIFT -MPP
128
99.66
47.61
91.90
25.59
95.67
62.60
90.34
73.34
Full FT
98.17
46.55
93.67
22.05
96.85
63.20
89.08
72.79
LoRA
128
98.00
47.76
92.41
23.62
95.08
62.90
90.76
72.93
Table 16: Arithmetic reasoning performance of LIFT.MLP, which only fine-tunes the MLP layers of an LLM.
G.5. A Toy Model Case
To demonstrate that LIFT works not only on large models, here we consider toy model task: two-layer neural network (z) for regression:
f(X) = o (XWWa
(4)
where X E RRaxd W E Rdxh, E R
Training Pipeline. We first pre-train this toy network with a curated training dataset then Apply LIFT during fine-tuning and compare its performance with that of Full Fine-tuning and other Sparse Fine-tuning methods. We consider Adamf optimizer and an early stopping strategy for training and fine-tuning.
Pre-training and fine-tuning Datasets. For pre-training dataset, we first random sample Xpre E Roo where Nore is the number of pre-training samples and d is the input dimension. We construct pre-training labels
Y pre [: sum (dim = 1) + 0.1 * sin (x, 32: 64)).sum(dinn =)
(5)
For fine-tuning dataset, we random sample X E xd where ns is the number of fine-tuning samples and d is the input dimension. We construct fine-tuning labels.
Ya = 0.2 [x 64] Xnl:; 65] * Xx 66] + 0.1 * sin (Xn[:.67] Xxl: 68))
(6)
For convenince, we assume 512, 128, Nyre 5000, nft 100.
Figure 14 shows the statistics during fine-tuning of different methods. We compare LIFT with Full Fine-tuning and Sparse Fine-tuning with parameters selected by weight magnitude and gradient magnitude. As we can see, in terms of training and validation loss in Figure 14a and 14b, Full Fine-tuning saturates the early stage, indicating overfitting. On the other hand, sparse fine-tuning methods are more adaptive and achieve better generalization, achieving lower validation loss. Among the sparse fine-tuning methods, LIFT outperforms other strategies by a large margin, suggesting that LIFT has superior generalization performance. In addition, as shown in Figure 14c LIFT achieves more stable gradient norm, with sharp decay towards the end, and converges to lower values. Furthermore, in Figure 14d, we can see that LIFT achieves a substantially lower spectral norm than other methods. This observation aligns with Appendix C, that LIFT can significantly change the spectral norm of weight matrices. These observations on the effectiveness of LIFT pave the way for future theoretical works.
G.6. Training Loss of LIFT
In Figure 15, we show the training loss curve of all methods of LLaMA-2-7B model from the experiments in Table 2. We can see that the convergence speed of LIFT is on par with Full FT, notably faster than other PEFT methods.
G.7. Exploring Structured Fine-tuning of LIFT
In our paper, LIFT selects and fine-tunes model parameters in an unstructured fashion. We realize that LIFT naturally lends itself to block-sparse fine-tuning. To validate the great potential of LIFT in this context, we conducted an experiment with LIFT suctuued that selects and fine-tunes Principal Weights 4 blocks. We compare with two common
22

---

Principal Weights Emerge after Rank Reduction for Reasoning-Focuset Supervised Fine- Tune-Tnnig
Full FT
LIFT
Weight Mag
Grad Mag
10
10
10
10
10
10
10"
10
10
10"
10"
Full MagRlight Mag
Epoch
Epoch
Epoch
(a) Training Loss
(b) Validation Loss
(c) Gradient Norm
(d) Spectral Norm
Figure 14: Fine-tuning statistics on the toy model setting of different fine-tuning strategies.
FT
0.6
LIFT
LoRA
Training Loss
0.5
DoRA
Mww Ma Myrne
PiSSA
0.4
S2FT
0.3
0.2
0.1
0
250
500
750
1000 Steps
1250 1500 1750
Figure 15: Training loss curves of different methods with LLaMA-2-7B model on MATH-10K dataset.
parameter selection metrics: 1) Top-k Magnitude, 2) Top-k Gradient. For all sparse methods, we fine-ture a subset of weights with the number of parameters identical to LoRA with rank Ra 128. We choose to fine-tune the LLaMA-2-7B model with the MATHIOK dataset and evaluate on seven arithmetic reasoning tasks. The results are in Table 17.
Method
Rank
MultiArith
GSM8K
AddSub
AQuA
SingleEQ
SVAMP
MAWPS
Avg.
T.FFT St rret
128
98.33
48.07
93.16
25.98
95.47
89.92
73.72
LIFT
128
98.67
47.31
92.66
26.77
96.85
63.6
90.34
73.74
Full FT
98.17
46.55
93.67
22.05
96.85
63.2
89.08
72.79
Weight Mag
128
98.00
49.13
91.39
23.62
93.90
63.3
89.08
72.64
Grad Mag
128
97.50
45.72
92.41
23.23
95.28
71.83
Table 17: Performance of different parameter selection strategies on arithmetic reasoning tasks.
can see that L.FT.Structured still achieves great performance under structured sparsity. LIFT LIFI.Structured achieves performance on par with LIFT, and outperforms Full FT and other common sparse selection metrics. This suggests that LIFT has the potential to be adapted for structured sparse fine-tuning, enabling further computational acceleration.
G.8. Rank Reduction of LIFT
In LIFT, we employ uniform rank to perform low-rank approximation on each weight matrix. We know that models with different sizes possess different learning capacities, and their "Important Ranks" could potentially be different. To find the trend how approximation rank influences the power of LIFT, we tne-tune models with different sizes using different approximation rank, ranging from 8 256.
23

---

Principal Weights Emerge after Rank Reduction for Ceasoning-Focusee Supervised Fine- Tine-Tunig
80
16
70
Rank
60
Rank
32
Rank
32
Rank
78
70
76
50
32 64 128 LRA Rank
256
128 256
16 32
64 128 256 LRA Rank
LRA Rank
LRA Rank
(a) LLaMA-3.2-IB
(b) LLaMA-3.2-3B
(c) LLaMA-2-7B
(d) LLaMA-3-8B
Figure 16: Average performance on arithmetic reasoning tasks when applying LIFT with low-rank approximation of different ranks.
Figure 16 shows the heat map of different models on seven arithmetic reasoning tasks. In each plot, the x-axis represents the rank used for low-rank approximation, which we call the LRA rank. The y-axis rank represents the number of trained parameters, matching the number of parameters selected by the LoRA method with corresponding ranks, which we call Selected Rank. We can see that a larger LRA rank doesn't yield better performance. Instead, LIFT works best when the LRA rank is close to the Selected Rank. Moreover, when the model size is larger, the Selected Rank of best performance tends to be smaller than that of smaller models. This suggests that smaller models store knowledge more densely, therefore maintaining a larger effective rank, while the larger model has larger capacity, and the effective rank for each layer may be smaller to store the same amount of knowledge.
G.9. Parameter Selection of LIFT
To analyze the parameters chosen by LIFT, we compare them with parameters selected by Weight Magnitude, and analyze their overlap.
In Figure 17, we present the overlap of parameters selected by LIFT and Weight Magnitude on different layers. We can see that the overlap between the two methods is small for all layers. Specifically, for Query and Key layers, the overlap is relatively larger, generally around 20% and most 40%, while on the Up and Down layers in the MLP module, the overlap significantly smaller, around 5% and matrices are close to the original matrix, the weights with the largest magnitude more similar. This could mean that Query and Key modules are low-rank in nature, and fine-tuning them is not as effective other modules. This aligns with our observation from Section 7.3 and Appendix 11.
20%. This suggests that the low-rank approximation of the Query and Key
In addition, when selecting the same amount of parameters, we increase the rank of low-rank approximation, the overlap between LIFT and Weight Magnitude also becomes larger. This is also intuitive as perform precise approximation of the original matrix as we increase the LRA rank.
H. Metric Definitions
H.1. Eigenspace Alignment Score
To calculate the alignment score of the top eigenspace before and after fine-tuning, take the top 128 right singular vectors of each weight matrix before and after fine-tuning. For each singular vector v after fine-tuning, we calculate the alignment score d, as the sum of squared cosine similarity with all top vectors of the pre-trained model V1, V2, Vn!
n d;
Σ (v. v2)2 j=1
Since all singular vectors are orthonormal, the alignment score has the range [0, 1], representing the projection length of the fine-tuned singular vectors on the subspace spanned by the top pre-trained singular vectors. Then, the total alignment score the mean of the alignment score of all singular vectors,
24

---

Principal Weights Emerge after Rank Reduction for Jassoning-Focusee Supervised Fine-Tuning Tuning
Query
Key
Value
Output
30
30
40
40
25
25
30
30
20
20
15
20
20
10
10
10
5
2
8
10
12
2
4
6
8
10
12
0
2
4
6
8
10
12
10 % of Parameters
12
% of Parameters
% of Parameters
% of Parameters
Gate
Up
Down
20
LRA Rank:
20
15
15
LRA Rank: 16
15
10
LRA Rank: 32
LRA Rank: 64
10
5
5
LRA Rank: 256
0
2
8 % of Parameters
4
6
10
12
10
12
2
4
6 8
10
12
% of Parameters
% of Parameters
Figure 17: The overlap ratio of parameters selected by LIFT and by Weight Magnitude.
d
Σ d 1=1
(8)
This should reflect the alignment of fine-tuned singular vector and the cigensubspace spanned by the pre-trained model. An aligmment score of represents that the singular vector aligns perfectly with the pre-trained eigenspectrum, and 0 represents that the singular vector is orthogonal to the pre-trained eigenspectrum.
25


Reviews:
[{"review_id": "uxckGlYtdD", "reviewer": "Reviewer_kFAr", "summary": "This paper introduces a sparse fine-tuning approach, LIFT, which identifies and updates what the authors call “Principal Weights” in LLMs. The central claim is that the most critical parameters for downstream fine-tuning can be found by first applying low-rank approximation (e.g., SVD) to each weight matrix, then selecting the largest-magnitude weights from the approximated matrix. These selected parameters are subsequently updated during training, while the rest remain frozen. The paper reports that LIFT achieves performance on par with or exceeding full fine-tuning across various tasks (e.g., arithmetic and commonsense reasoning), all while using a memory footprint comparable to parameter-efficient fine-tuning (PEFT) methods like LoRA.\n\n======\n\nAfter reviewing the full discussion again, I find that two issues still remain unresolved and, in my opinion, are unconvincing:\n\n- The authors mention that they follow the experimental setup of WeLore, yet they do not include WeLore in their comparisons or discussions in the manuscript.\n\n- The implementation of the method directly modifies original model weights, causing position changes during training that violate sparsity constraints. Thus, the approach aligns more with memory-efficient or sparse training methods rather than PEFT.\n\nFor the reasons stated above, I maintain my current rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for the constructive feedback. We'd like to address your concerns as follows. **Please find the supplementary figures/tables in the [rebuttal link here](https://github.com/icml12437/ICML2025_12437).**\n# Q1: Higher-level analysis on LIFT\nIn our paper, we conduct higher-level analysis on LIFT through empirical analysis and eigenspectrum analysis. \n\n**First, we show that parameters selected by LIFT are crucial to model performance.** In Sec. 4 and Fig. 2, we show that LIFT selects principal weights that are sensitive to random perturbations, which has significant impact on the model performance, thereby validating that these weights are good candidates for fine-tuning. \n\n**Second, we also provided in-depth analysis of the eigenspace of LIFT.** Sec. 7.2 and 7.3 show that LIFT induces larger weight changes (Fig. 5), and significantly increases the rank (and effective rank, see the Q4 response to Reviewer ceyi) of weight updates (Fig. 11), and exhibits greater singular subspace divergence (Fig. 10), all of these demonstrates LIFT shows a greater learning capacity in fine-tuning. Additionally, spectral norm analysis suggests LIFT enhances generalization (Fig. 8, 9), where random perturbations sharply increase the spectral norm in both random matrix and LLM settings, and LIFT's lower gradient norm curves (Fig. 12) further demonstrate its effectiveness, as supported by PiSSA [1].\n# Q2: Evaluation on diverse domains\nWe conduct more experiments on QA and Code Generation datasets. These experiments further showcase the effectiveness of LIFT on more diverse domains.\n\nFor QA, we use the experimental setup of StrategyQA dataset following recent WeLore paper [2]. For PEFT methods, we consider ranks {16, 32, 64, 128, 256}; for LIFT, we use the same counts of trainable parameters. The learning rates are {1e-5, 2e-5, 5e-5, 1e-4} for Full FT and {5e-5, 1e-4, 2e-4, 5e-4} for others. We select the best-performing config for each method and report the results below. We can see that LIFT achieves notable performance gains than all other methods on both LLaMA-2-7B and LLaMA-3-8B.\n|StrategyQA|LIFT|Full FT|LoRA|DoRA|PiSSA|\n|-|-|-|-|-|-|\n|LLaMA-2-7B|**72.53**|70.61|71.78|71.98|71.26|\n|LLaMA-3-8B|**75.85**|74.81|74.44|74.27|75.19|\n\nFor code generation, we adopt the settings from the recent SIFT paper [3], where we fine-tune LLaMA-2-7B with Alpaca dataset for one epoch, and evaluate on the **Humaneval** dataset. From the table below we can see that LIFT outperforms all other methods in both pass@1 and pass@10 settings.\n|**Humaneval**|LIFT|Full FT|SIFT|LoRA|DoRA|\n|-|-|-|-|-|-|\n|Pass@1|**16.46**|15.24|14.02 |13.66|13.96|\n|Pass@10|**31.10**|28.05|30.48|27.44|29.88|\n\n# Q3: Is the results of LIFT robust under multiple seeds?\nIn Fig. 3 in our paper, we show the results of LIFT on GSM8K datasets with 4 random seeds. In table 8 in the rebuttal link, we present the results on arithemetic datasets with four random seeds (same as Table 2 in the paper), which further prove that LIFT has robust performance, where it outperforms other baselines under four seeds.\n\n# Q4: Comparison with dynamic/structured pruning methods\nIn Fig. 3, we have compared LIFT with a wide range of other dynamic sparse fine-tuning methods, including Top-k Magnitude, Top-k Gradient, and Movement, and showed that LIFT outperforms all other sparse fine-tuning methods. We note that unstructured sparse fine-tuning usually serves as the performance ceiling of structured and block-wise sparsity. In addition, in our main results, we have compared LIFT with S2FT, which is the state-of-the-art structured fine-tuning method by far. These results suggest that LIFT is indeed superior to other common sparse selection methods.  \n\n# Q5: Hyperparameter Tuning\nHere we explain the hyperparameter tuning for experiments. We will incorporate detailed information in the revised version of our paper. \n## Training Hyperparameters\nIn all experiments in the paper, we **compare the best results of different methods among a range of parameter sizes**. Specifically, when comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and **LIFT with the same parameter counts** to ensure fair comparison, and pick their best results. We find that LIFT and LoRA-like methods typically have the best performance at **rank = 128**, similar to results from PiSSA paper [1].\n## LIFT Hyperparameters\nThe main hyperparameter of LIFT is the **LRA Rank**, which **controls the rank for low-rank approximation** of weight matrices. In **Appendix G.5 and Fig. 15** of our paper, we study the influence of LRA Rank on the performance of LIFT. We found that the LRA Rank of best performance increases as we select more parameters to train. In practice, when we tune LIFT with the same number of parameters as LoRA, we find that **using LRA Rank similar to LoRA rank yields optimal performance**.\n## References\n[1] Meng et al, 2024  \n[2] Jaiswal et al, 2024  \n[3] Song et al, 2024", "claims_and_evidence": "The paper’s main claims—that “Principal Weights” identified via low-rank approximation are the most important for fine-tuning and that the resulting LIFT method outperforms both full fine-tuning and prior PEFT approaches—feel somewhat overstated. While the empirical results are suggestive, the authors don’t convincingly prove that this exact selection strategy is uniquely optimal.", "methods_and_evaluation_criteria": "They use typical LLM benchmarks—like arithmetic and commonsense tests—to judge performance. While these tasks do show some variety, there’s no deeper or more specialized evaluation that might reveal the approach’s limits or weaknesses. The metrics are standard accuracy-based scores, which is fine, but there’s little beyond that (e.g., no real-world or higher-level analysis).", "theoretical_claims": "There isn’t much formal proof offered", "experimental_designs_or_analyses": "Their experimental setup mostly follows standard fine-tuning procedures, but there’s little discussion of hyperparameter tuning or random seed variability. It’s unclear how stable these results are across multiple runs.", "relation_to_broader_scientific_literature": "They do reference recent efforts like LoRA, low-rank approximation approaches, and sparse-adapter methods. However, the paper mostly leans on known results about low-rank structures in large language models (e.g., Eckart–Young–Mirsky) and doesn’t deeply compare against or incorporate broader theoretical and empirical work on compressive adaptation (like structured pruning, block-wise sparsity, etc.).", "essential_references_not_discussed": "They mention some sparse and low-rank methods (like LoRA), but skip other key variants on dynamic or structured pruning (e.g., top-k fine-tuning approaches) that also use magnitude- or gradient-based selection.", "other_strengths_and_weaknesses": "A notable plus is that the paper lays out an easy-to-replicate method and backs it up with multiple experiments. The authors do a decent job packaging a simple rank-then-sparsify idea into a seemingly effective pipeline. However, the novelty is fairly modest. The paper also sometimes feels repetitive when explaining the approach, and the clarity on hyperparameters and training details could be improved.", "other_comments_or_suggestions": "A clearer, more concise discussion of hyperparameters for each experiment and how they’re tuned would help readers reproduce results.", "questions_for_authors": "You focus on arithmetic and commonsense tasks—have you tested more diverse domains (e.g., structured QA, code generation)? If so, how does LIFT fare there, especially on tasks known for high domain shift or specialized knowledge?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AobENVwiIg", "reviewer": "Reviewer_ceyi", "summary": "This paper proposes a novel sparse fine-tuning method, LIFT, that identifies and fine-tunes the critical parameters (Principal Weights) through SVD-based rank reduction. Extensive experiments demonstrate that LIFT significantly outperforms existing parameter-efficient fine-tuning (PEFT) approaches, including LoRA, as well as full fine-tuning. The paper also provides valuable insights through thorough analyses, including eigenspace and eigenspectrum investigations and detailed ablation studies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for the constructive feedback. We'd like to address your concerns as follows. **The supplementary figures/tables are in the [rebuttal link here](https://github.com/icml12437/ICML2025_12437).**\n# Q1: Comparison with recent sparse fine-tuning methods\nWe compare LIFT with sparse fine-tuning methods, SIFT [1] and SpIEL [2] (since SMT is not opensourced, we compare with other two). We will cite these papers in the revised version.\n\n**First, we compare LIFT with SIFT on GLUE tasks** following the SIFT paper, with the same number of trainable parameters (5% total parameters). We use the RoBERTa-large model and search LR of SIFT and LIFT in {5e-5, 7e-5, 1e-4, 2e-4} and compare their best results. The table below shows that LIFT outperforms SIFT on all GLUE tasks, while outperforming Full FT on almost all tasks.\n||MNLI|SST2|MRPC|CoLA|QNLI|QQP|RTE|STSB|Avg.|\n|-|-|-|-|-|-|-|-|-|-|\n|**LIFT**|**90.79**|96.67|90.93|**70.44**|**94.69**|**92.38**|**87.00**|**92.58**|**89.44**|\n|SIFT|89.91|**96.79**|89.95|66.29|93.04|88.49|87.00|92.27|87.97|\n|Full FT|90.58|96.22|**91.91**|68.55|94.47|91.52|85.92|92.21|88.92|\n\n**Second, we compare LIFT with SpIEL on GSM8K** (See Q2 response to Reviewer AcQs). We show that LIFT significantly outperforms SpIEL on both LLaMA-2-7B and LLaMA-3.2-3B models.\n# Q2: Number of trainable parameters and GPU memory cost of LIFT\n## 2.1. Number of trainable parameters\nIn all experiments in the paper, we **compare the best results of different methods in a range of parameter sizes**. When comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and **LIFT with the same parameter counts** to ensure fair comparison. We find that LIFT and LoRA-like methods typically perform best at **rank = 128**.\n\nIn addition, in our analytical results (e.g. Fig. 11), we compare LIFT and LoRA both with **rank = 128**. Some layers of LoRA have rank larger than 128 is likely due to numerical errors when computing the rank.\n\n## 2.2. GPU memory cost of LIFT\nWe note that although the memory cost of LIFT (as in Fig. 6 in Appendix B) is slightly larger than LoRA, we can further reduce the memory cost of LIFT while preserving performance.  \n\nIn Appendix G.4, we showed that fine-tuning MLP layers is more effective than fine-tuning attention layers. The table below shows the results of LLaMA-2-7B on arithmetic datasets where we **only fine-tune the MLP layers (LIFT_MLP)**. We see that LIFT_MLP has similar performance to LIFT. Furthermore, only fine-tuning MLP layers further reduces the memory usage on gradients and optimizer states. In Fig. 17 of the rebuttal link, we see that LIFT_MLP achieves better memory efficiency than LoRA under optimal settings.\n||MultiArith|GSM8K|AddSub|AQuA|SingleEQ|SVAMP|MAWPS|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|**LIFT**|98.67|47.31|92.66|**26.77**|**96.85**|**63.6**|**90.34**| **73.74**|\n|**LIFT_MLP**|**99.66**|**47.61**|91.90|25.59|95.67|62.6|**90.34**| **73.34**|\n|Full FT|98.17|46.55|**93.67**|22.05|96.85|63.2|89.08|72.79|87.54|\n|LoRA|98.00|47.76|92.41|23.62|95.08|62.9|90.76|72.93|\n# Q3: Could LIFT negatively impact the performance on OOD tasks?\nWe believe although model weights undergo significant changes during fine-tuning, because only a small set of parameters are changed while most weights remain unchanged (see the center of histogram in Fig. 5), the model retains its fundamental capacities that enable it to generalize to OOD settings.\n\nIn Sec. 7.1, we also analyzed the generalization performance of LIFT by training models with arithmetic datasets and evaluating on commonsense tasks (OOD). This study verifies that LIFT achieves stronger OOD performance compared to other baselines.  \n# Q4: Metrics like effective rank\nHere we compare LIFT with HiRA [3] and evaluate the effective rank of other baselines.  \n\nThe table below shows the results of LIFT and HiRA on arithmetic datasets (same as Table 2 in our paper). We search HiRA rank in {16, 32, 64, 128, 256, 512} and found that HiRA performs best at rank = 512. We see that LIFT (rank = 128) outperforms HiRA with rank = 512.\n||MultiArith|GSM8K|AddSub|AQuA|SingleEQ|SVAMP|MAWPS|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|**LIFT**|**98.67**|**47.31**|**92.66**|**26.77**|**96.85**|**63.6**| **90.34**|**73.74**|\n|HiRA|98.50|46.70|91.65|25.59|95.67|61.5|89.50|72.72|\n\nIn Fig. 19 of the rebuttal link, we show the effective rank of different methods. We see that the update matrix of LIFT has larger effective rank than LoRA and PiSSA, close to Full FT, while slightly lower than HiRA. We believe although effective rank is a good indicator of model capacity, it's insufficient to soly rely on it to predict performance.\n# Q5: Training curve of LIFT\nIn Fig. 18 of the rebuttal link, we show the training loss curve of all methods in Table 2 results. We see that the convergence speed of LIFT is on par with Full FT, notably faster than other PEFT methods.\n## References\n[1] Song et al, 2024  \n[2] Ansell et al, 2024  \n[3] Huang et al, 2025", "claims_and_evidence": "The claims presented in the paper are well-supported by comprehensive empirical results across various tasks, demonstrating consistent effectiveness.", "methods_and_evaluation_criteria": "The benchmarks selected in this paper are suitable for evaluating the proposed method's performance.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental setup is clearly structured and adequately covers a broad spectrum of tasks. However, the comparison baselines are primarily related to LoRA-based methods, while Sparse FT methods are less extensively compared, with only S2FT included as a baseline.", "relation_to_broader_scientific_literature": "The paper introduces a straightforward yet highly effective approach for sparse FT. Additionally, it explores multiple parameter selection strategies, providing valuable insights and analysis.", "essential_references_not_discussed": "The paper adequately discusses relevant literature but omits discussions and comparisons with several recent sparse fine-tuning methods, including:\n\n[r1] Sparse is Enough in Fine-tuning Pre-trained Large Language Models, ICML 2024\n\n[r2] SMT: Fine-Tuning Large Language Models with Sparse Matrices, ICLR 2025\n\n[r3] Scaling Sparse Fine-Tuning to Large Language Models, arXiv preprint arXiv:2401.16405, 2024.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is clearly structured, well-written, and easy to follow.\n- The proposed method (LIFT) is conceptually sound, with thorough analyses and ablation studies to support its effectiveness.\n- LIFT achieves superior performance compared to Full FT, while maintaining memory efficiency similar to LoRA and other parameter-efficient methods.\n\nWeaknesses:\n\n- The tables presented in the paper do not explicitly report the actual GPU memory consumption or the exact number of trainable parameters for each method. Additionally, details such as the specific rank setting for LoRA are unclear; from Figure 11, it seems the LoRA rank is at least 512, significantly different from the common settings (typically 8-32 for LLaMA), potentially leading to unfair comparisons. It is advisable to supplement this information explicitly.\n\n- According to Appendix B, LIFT still has higher GPU memory usage during training compared to LoRA and its variants. The part of GPU memory consumed by activations can be significantly reduced using checkpointing techniques, which may make LoRA’s memory efficiency advantage even clearer.", "other_comments_or_suggestions": "- In the analysis of eigenspace and eigenspectrum, incorporating metrics like effective rank (as presented in Figure 8 of HiRA [r4]) might better represent the full singular value spectrum.\n- Expand the method comparison by including recent sparse fine-tuning methods as well as advanced high-rank fine-tuning techniques, like HiRA, which could further demonstrate LIFT’s strengths.\n\n[r4] HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models, ICLR 2025", "questions_for_authors": "Q1: The paper states that LIFT balances learning and forgetting effectively, yet from Figure 5, the weights undergo significant changes, especially Principal Weights. Could such substantial modifications negatively impact the performance on Out-of-Distribution (OOD) tasks to some extent?\n\nQ2: Could the authors add loss convergence curves for each method in the main experiments, similar to Figure 12, to further clarify the training behavior and convergence speed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ZQybGz2pq", "reviewer": "Reviewer_AcQs", "summary": "The authors propose a novel sparse fine-tuning approach, LIFT, which identifies so-called Principal Weights. By only training these Principal Weights, LIFT outperforms full-parameter fine-tuning in multiple benchmarks including commonsense reasoning, math reasoning, and GLUE tasks. Specifically, Principal Weights are the weights that have the largest magnitude after performing low-rank reduction.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for the insightful and constructive feedback. We'd like to address your concerns as follows.\n\n# Q1: Overlap between Principal Weights and the original largest-magnitude weights\n\nIn **Appendix G.6** of our paper, we discussed the overlap between parameters selected by LIFT and parameters selected by weight magnitude. In Fig. 16, we plotted the overlap ratio between LIFT and weight magnitude on different layer types as the number of trainable parameters varies. We showed that the overlap is generally below 20%, while different layers have different overlap ratios. We note that the overlap on MLP layers are significantly lower than that on attention layers. This suggests that the low-rank approximation of the Query and Key matrices are close to the original matrix, and Query and Key modules are low-rank in nature. \n\nThe above results indicate that the *Principal Weights* of LIFT and original large-magnitude weights do not overlap heavily, and how to combine the two types of weights is indeed a promising direction of future work.\n\n# Q2: Missing discussions on recent sparse fine-tuning paper.\n\nHere we discuss the paper \"Scaling Sparse Fine-Tuning to Large Language Models\" [1]. This paper proposed SpIEL approach to reduce the need for computing all gradients during backward propagation, which scales the gradient memory linearly with the selected weights. While our method LIFT proposed a novel way to select  Principal Weights for sparse fine-tuning and store only the optimizer states of Principal Weights to significantly reduce the memory overhead.\n\n**We also compare LIFT with SpIEL on the GSM8K dataset.** We use the training setting as in Fig. 3 in our paper. For both methods, we searched learning rate among {5e-5, 1e-4, 2e-4, 5e-4} and trainable parameters corresponding to LoRA rank among {16, 32, 64, 128, 256} to obtain the best results. The table below shows that LIFT significantly outperforms SpIEL with both LLaMA-2-7B and LLaMA-3.2-3B model.\n\n|**GSM8K**|LIFT|SpIEL|Full FT|\n|-|-|-|-|\n|LLaMA-3.2-3B|**46.46**|43.76|44.50|\n|LLaMA-2-7B|**24.24**|21.61|22.57|\n\nWhat's more, combining LIFT with the way SpIEL reduces memory overhead to further pursue memory efficiency is a promising direction for future work. \n\n\n# Q3: Can LIFT technique be adapted to structured sparse finetuning?\n\nTo evaluate whether LIFT can work in a structured fine-tuning fashion, we perform initial experiments on structured block-sparsity, where we select a number of $4\\times4$ blocks to fine-tune (**LIFT_Structured**). We select 5% of all parameters to train (corresponding to LoRA rank $\\approx$ 128). We choose to fine-tune LLaMA-2-7B model with MATH-10K dataset and evaluate on seven arithmetic reasoning tasks. The results are as follows.\n\n||MultiArith|GSM8K|AddSub|AQuA|SingleEQ|SVAMP|MAWPS|Avg.|\n|-|-|-|-|-|-|-|-|-|\n|**LIFT_Structured**|98.33|**48.07**|93.16|25.98|95.47|**65.1**|89.92|**73.72**|\n|**LIFT**|**98.67**|47.31|92.66|**26.77**|**96.85**|63.6|**90.34**|**73.74**|\n|Full FT|98.17|46.55|**93.67**|22.05|96.85|63.2|89.08|72.79|87.54|\n\nFrom the table above, we can see that LIFT still achieves great performance under structured sparsity. LIFT_Structured has almost the same performance as original LIFT, while outperforming Full FT. This suggests that LIFT has the potential to be adapted to structured, sparse fine-tuning to achieve further computation acceleration.\n\n# Q4: Typos\n\nWe thank the reviewer for pointing out the typo, and we will fix it in the revised version of the paper.\n\n### References\n[1] Ansell et al, 2024", "claims_and_evidence": "The paper introduces the Low-rank Informed Sparse Fine-Tuning (LIFT) method and provides substantial evidence supporting its effectiveness. The claim of LIFT's superior performance is validated through extensive experimental evaluation across three distinct tasks: (1) Commonsense Reasoning, (2) Arithmetic Reasoning, and (3) Natural Language Understanding. Additionally, the paper robustly supports the claim that LIFT identifies Principal Weights through experiments in Figure 2.", "methods_and_evaluation_criteria": "The motivation for identifying principal weights via low-rank decomposition is clearly justified by Figure 2. The effectiveness of LIFT is thoroughly demonstrated through extensive experiments across diverse tasks and varying model sizes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "(1)Good evaluation, comparing against a number of alternative state of the art models. I appreciate the fact that all experiments are run with four random seeds, which demonstrate the performance gain of LIFT is significant. \n\n(2)It would be helpful if the paper explicitly analyzed the ratio or overlap between Principal Weights and original largest-magnitude weights across different layers. Understanding whether Principal Weights significantly differ from massive weights, and how this ratio varies across model layers, could offer deeper insights into LIFT’s effectiveness and behavior.", "relation_to_broader_scientific_literature": "The low rank finds principle weights is related to the paper “ From Galore to Welore: how low-rank weights no-uniformly emerge from low-rank gradients” (https://arxiv.org/abs/2407.11239.)", "essential_references_not_discussed": "https://arxiv.org/pdf/2401.16405 this is very early sparse finetuning methods for LLMs, which is missing in this paper.", "other_strengths_and_weaknesses": "Strengths:\nNovel idea nicely explained and motivated. It is interesting to see that crucial weights emerge after performing SVD.\nLIFT demonstrates stronger generalization, balancing the learning of new task-specific knowledge with minimal forgetting of the source domain knowledge. \n\nWeakness:\nThe paper's identification of Principal Weights is insightful; however, recent research (e.g., the Massive Weights in LLMs paper) suggests original large-magnitude weights are also crucial. It might be beneficial to explore combining both Principal Weights and original Massive Weights to further enhance performance, rather than relying exclusively on one set.", "other_comments_or_suggestions": "Typos: state-of-the-srt -> state-of-the-art in line 250.", "questions_for_authors": "Considering that unstructured sparse fine-tuning is not well supported by GPUs, whereas structured sparse fine-tuning enjoys efficient GPU support, Can this LIFT technique be adapted to structured sparse finetuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BN2lwgDVAx", "reviewer": "Reviewer_yeg4", "summary": "This paper presents LIFT (Low-rank Informed Sparse Fine-Tuning), which introduces the idea of Principal Weights—parameters with the largest magnitude after low-rank approximation—as the most critical ones for LLM fine-tuning. This research in very important to the community, as sparse fine-tuning methods have been largely ignored in the LLM era, even though they were effective for pre-LLM models. The authors argue that one of the reasons for this is that sparse fine-tuning struggles to identify which parameters matter, and they address this with a fundamentally new approach, i.e., using low-rank decomposition to first filter noise and then select key weights based on magnitude.\n\nThe method is novel, well-motivated, and well-executed. Sparse fine-tuning has been overlooked in LLMs, and this paper does a fantastic job of showing that it can be revived and improved with low-rank decomposition. The results are strong, the insights are compelling, and the memory efficiency makes it highly practical.\n\nThere are some minor things that could be improved (discussion of computational cost, layerwise analysis), but these are not major flaws—more like areas for further exploration. Overall, this is a clear strong accept. It pushes the field forward in an important way, and I expect it will have significant impact in both research and practical applications of LLM fine-tuning.\n\n## update after rebuttal\nAuthors have addressed my concerns. Thus, I keep it as accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for the insightful and constructive feedback. We'd like to address your concerns as follows. **The supplementary figures/tables are in the [rebuttal link here](https://github.com/icml12437/ICML2025_12437).**\n# Q1: Layer-wise analysis on LIFT\n\nIn Appendix G.4 of our paper, we analyze the **layer-wise effect of LIFT**, where we compare the performance of LIFT in **fine-tuning a single type of layers**. Our results show that fine-tuning only MLP layers yields significantly better results than fine-tuning attention layers. We hypothesize that attention components mostly store information on token relations rather than task-specific knowledge. On the other hand, MLP layers are more adaptive to downstream tasks, and fine-tuning them is more effective.\n\nBased on that insight, we explore the possibility of further improving the efficiency of LIFT by **only fine-tuning MLP layers**. The table below shows the results of LLaMA-2-7B on arithmetic datasets where we only fine-tune the MLP layers (LIFT_MLP), and only fine-tune the attention layers (LIFT_Attn). We can see that LIFT_MLP has similar performance to the full version of LIFT, while LIFT_Attn has drastically worse performance. This suggests that for LIFT, fine-tuning MLP layers is more effective than fine-tuning Attention layers.\n\n||MultiArith|GSM8K|AddSub|AQuA|SingleEQ|SVAMP|MAWPS|Avg|\n|-|-|-|-|-|-|-|-|-|\n|**LIFT**|98.67|47.31|92.66|**26.77**|**96.85**|**63.6**|**90.34**|**73.74**|\n|**LIFT_MLP**|**99.66**|**47.61**|91.90|25.59|95.67|62.6|**90.34**|73.34|\n|**LIFT_Attn**|95.00|43.75|91.14|25.59|91.73|60.1|86.55|70.55|\n|Full FT|98.17|46.55|**93.67**|22.05|96.85|63.2|89.08|72.79|87.54|\n|LoRA|98.00|47.76|92.41|23.62|95.08|62.9|90.76|72.93|\n\n\n\n# Q2: How does the rank-level affect the effectiveness of LIFT?\n\nIn **Appendix G.5 and Fig. 15** of our paper, we study the influence of rank in low-rank approximation (LRA Rank) on the performance of LIFT. We found a correlation between LRA Rank and the number of trainable parameters, that **the optimal LRA Rank increases as we select more parameters to train**. In practice, when LIFT has the same number of parameters as LoRA, we find that **using LRA Rank similar to LoRA rank yields optimal performance**. We note that LRA Rank may differ with different models, using some layer-adaptive metrics could bring further performance gain to LIFT.\n\n# Q3: More discussions on related references\nThe SVFT paper [1] introduces a PEFT method called SVFT, which performs SVD on pre-trained weights and fine-tunes the resulting SVD factors. In contrast, our work considers sparse fine-tuning and proposes a novel approach for selecting Principal Weights to update. Furthermore, by storing only the optimizer states associated with these Principal Weights, LIFT significantly reduces memory overhead. We will cite SVFT in the revised version of our paper.\n\n# Q4: Detailed training settings and memory costs\n\n## Training Settings\n\nIn Fig. 3 in our paper, we showed the results of LIFT on GSM8K datasets with 4 random seeds. Due to resource limitations, for other experiments we only report the results of one seed. In Table 8 in the rebuttal link, we present the results on arithemetic datasets with four random seeds (Table 2 in the paper), which further prove the robust performance of LIFT, where it outperforms other baselines under four seeds.\n\n## Number of Trainable Parameters and GPU Memory Costs\nIn all experiments in the paper, we **compare the best results of different methods among a range of parameter sizes**. Specifically, when comparing LIFT with LoRA-like methods, we search the LoRA rank in {16, 32, 64, 128, 256}, and **LIFT with the same parameter counts** to ensure fair comparison, and pick their best results. In practice, we find that the LIFT and LoRA-like methods typically have the best performance at **rank = 128**, similar to results from PiSSA paper [2].\n\nIn Fig. 6 of Appendix B, we present the memory breakdown of LIFT compared to Full FT and LoRA under optimal settings. We showed that LIFT achieves similar memory overhead as LoRA, significantly lower than Full FT. Furthermore, we explored only fine-tuning MLP layers with LIFT, which further improves memory advantages over LoRA, while maintaining test performance (details are in the Q2 response to Reviewer ceyi).\n\n# Q5: Typos\n\nWe thank the reviewer for pointing out the typo, and we will fix it in the revised version of the paper.\n\n### References\n[1] Lingam et al, 2024  \n[2] Meng et al, 2024", "claims_and_evidence": "The claims in the paper are well supported. The insights that Principal Weights are important to LLMs’ pre-trained knowledge are well supported by their pre-liminary study in Section 4. The effectiveness of LIFT is further supported with extensive experiments in Section 5. I didn’t see how many runs are provided in Section 5. I believe the results will be more significant, if this is the averaged performance over >3 runs. Another claim of LIFT learns better on target domains and preserve better on source domains is also supported in Figure 4. \n \nOne issue is that the computational overhead of LIFT isn’t fully explored. While the authors have mentioned that the comparison in their experiments are fair, it is important to report the exact memory costs in the main tables to provide a full picture to audiences.", "methods_and_evaluation_criteria": "The idea of Principal Weights itself is new and makes sense. To my best knowledge, denoising first, and then selecting weights for fine-tuning is non-trivial and well-motivated. Previous work of LASER (https://arxiv.org/abs/2312.13558) demonstrate that rank reduction using SVD has a denoising effect on LLMs, enhancing performance, which provides insights to the selection of Principal Weights. \n\nRegarding the evaluation criteria, multiple commonly used benchmarks are evaluated, including commonsense reasoning, math reasoning, and GLUE. The baselines in this paper include SOTA low-rank approaches LoRA, DoRA, and PiSSA, as well as a recent sparse fine-tuning baseline, i.e., S2FT. Some might say that there are many more PEFT approaches over there, but I believe that two most important baselines are already included, (1) PiSSA: SVD-based low-rank adaptation; (2) S2FT: a strong baseline of sparse fine-tuning. \n\nOne natural follow-up question to authors is how the rank-level affect the effectivenss of LIFT.", "theoretical_claims": "This paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes, the authors provide strong evidence that this approach revives sparse fine-tuning, making it competitive with (and often better than) Full FT and LoRA-like approaches. One question is whether the results are averaged over multiple runs, given the fact that LLM fine-tuning can be quite noisy. The analysis in this paper is intensive. \n\nMost existing PEFT approaches like LoRA and its variants focus on adding low-rank adapters, they are reported to struggle with large-scale settings (https://arxiv.org/abs/2405.09673). The observation that low-rank decomposition can help sparse fine-tuning by denoising first, then selecting weights, is non-trivial and well-motivated.", "relation_to_broader_scientific_literature": "LIFT algorithm is highly related to previous literature of using low-rank approximation for LLM denoising (https://arxiv.org/abs/2312.13558, https://arxiv.org/pdf/2406.03068). Moreover, sparse fine-tuning is an active research direction before the popularity of LLMs, where the authors have provided a paragraph of related work.", "essential_references_not_discussed": "https://arxiv.org/abs/2405.19597", "other_strengths_and_weaknesses": "Strengths：I really like the title “LIFT the Veil for the Truth” which subtly incorporates the algorithm name in a meaningful way, so kudos to the authors for that.\n\nWeaknesses：One thing that’s missing is a layerwise breakdown. We know from previous work that FFN layers and Attention layers behave very differently in fine-tuning. Does LIFT work equally well on both? Or are some layers more important than others? A study on layerwise sensitivity would be really interesting and could also help optimize LIFT further.", "other_comments_or_suggestions": "Some minor typos:\nLine 144: contains more -> contain more\nLine 250: state-of-the-srt -> state-of-the-art\nLine 437: We hope that these problems can lead to -> We hope that these problems can inspire", "questions_for_authors": "Does LIFT work equally well on both? Or are some layers more important than others? How about depth?\nHow the rank-level affect the effectivenss of LIFT?\nWhat are the exact memory costs in the main tables to provide a full picture to audiences?\nHow many random seems are reported in main tables?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews. After discussion, it was agreed that this paper is  above the ICML acceptance bar.
The authors are encouraged to address the comments of the reviewers to improve this work.
The author’s rebuttal has been carefully read and discussed. The author’s message has been carefully read and considered.
Decision: Accept (poster)