ICML 2025

LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for
Reasoning-Focused Supervised Fine-Tuning
Zihang Liu 1 Tianyu Pang 2 Oleg Balabanov 1 3 4 Chaoqun Yang 5 Tianjin Huang 6 7
Lu Yin 8 7 Yaoqing Yang 2 Shiwei Liu 9 7
Abstract
Recent studies have shown that supervised fine-
tuning of LLMs on a small number of high-quality
datasets can yield strong reasoning capabilities.
However, full fine-tuning (Full FT), while power-
ful, is computationally expensive and susceptible
to overfitting and catastrophic forgetting, partic-
ularly when data is limited. Sparse fine-tuning,
which previously achieved notable success by up-
dating only a small subset of model parameters,
offers a promising trade-off between efficiency
and effectiveness. Yet, it has lagged behind in the
LLM era due to the difficulty of identifying pa-
rameters truly critical for reasoning. In this work,
we state that weights with the largest magnitude
after low-rank approximation are critical weights
for fine-tuning, which we call Principal Weights.
Surprisingly, while magnitude-based sparse fine-
tuning performs poorly as a baseline on LLM
fine-tuning, it becomes highly effective after rank
reduction. These insights motivate our method:
Low-rank Informed Sparse Fine-Tuning (LIFT).
LIFT only updates the top 5% Principal Weights
throughout training and consistently achieves bet-
ter performance on reasoning tasks than Full FT,
while maintaining memory efficiency on par with
popular parameter-efficient fine-tuning methods.
In addition to strong performance on target do-
mains such as arithmetic reasoning, LIFT also
retains up to 20% more source-domain knowl-
edge, compared to Full FT and LoRA. Our code is
available at: https://github.com/zihanghliu/LIFT.
1University of California, Berkeley, CA, USA 2Dartmouth
College, NH, USA 3International Computer Science Institute,
CA, USA 4Lawrence Berkeley National Laboratory, CA, USA
5Tsinghua University, China 6University of Exeter, Exeter, UK
7E...