S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
Authors: Xinyu Yang
Jixuan Leng
Geyang Guo
Jiawei Zhao
Ryumei Nakada
Linjun Zhang
Huaxiu Yao
Beidi Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: PEFT; LLM Efficiency; LLM Post-training
Primary area: generative_models

Abstract: Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable serving, but not all three simultaneously.  
To address this limitation, we investigate sparse fine-tuning and observe a remarkable improvement in generalization ability. 
Utilizing this key insight, we propose a family of Structured Sparse Fine-Tuning (S${^2}$FT) methods for LLMs, which concurrently achieve state-of-the-art fine-tuning performance, training efficiency, and inference scalability. S${^2}$FT accomplishes this by "selecting sparsely and computing densely". Based on the coupled structures in LLMs, \model selects a few attention heads and channels in the MHA and FFN modules for each Transformer block, respectively. Next, it co-permutes the weight matrices on both sides of all coupled structures to connect the selected subsets in each layer into a dense submatrix. Finally, S${^2}$FT performs in-place gradient updates on all selected submatrices.
Through theoretical analyses and empirical results, our method prevents forgetting while simplifying optimization, delivers SOTA performance on both commonsense and arithmetic reasoning with 4.6% and 1.3% average improvements compared to LoRA, and surpasses full FT by 11.5% when generalizing to various domains after instruction tuning. 
Using our partial back-propagation algorithm, S${^2}$FT saves training memory up to 3$\times$ and improves latency by 1.5-2.7$\times$ compared to full FT, while achieving an average 10\% improvement over LoRA on both metrics. We further demonstrate that the weight updates in S${^2}$FT can be decoupled into adapters, enabling effective fusion, fast switch, and efficient parallelism when serving multiple fine-tuned models.

FT Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
Xinyu Yang', Jixuan Leng', Geyang Guo?, Jiawei Zhao Ryumei Nakada",
Linjun Zhang', Huaxiu Yao Beidi Chen'
1CMU .Georgia Tech, *Caltech, 'Rutgers, UUNC-hhppel Hill xinyuya2, beidic@andrew... c edu
https://infini-aii github. Lo/S2FT-Page
Abstract
Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable serving, but not all three simultaneously. To address this limitation, we investigate sparse fine-tuning and observe a remarkable improvement in generaliza- tion ability. Utilizing this key insight, propose a family of Structured Sparse Fine-Tuning FS² methods for LLMs, which concurrently achieve state-of-the art fine-tuning performance, training efficiency. and inference scalability. S accomplishes this by "selecting sparsely and computing densely". Based on the coupled structures in LLMs, FT selects few attention heads and channels the MHA and FFN modules for each Transformer block, respectively. Next, it co-permutes the weight matrices on both sides of all coupled structures to connect the selected subsets in each layer into a dense submatrix. Finally, S performs in-place gradient updates on all selected submatrices. Through theoretical analyses and empirical results, our method prevents forgetting while simplifying optimiza- tion, delivers SOTA performance on both commonsense and arithmetic reasoning with 4.6% and 1.3% average improvements compared to LoRA, and surpasses full FT by 11.5% when generalizing to various domains after instruction tuning. Using our partial back-propage tokkapopaattion algorithm, FT saves training memory up 3x and improves latency by 1.5-2.7 compared to full FT, while achieving an average 10% improvement over LoRA both metrics. We further demonstrate that the weight updates in FT can be decoupled into adapters, enabling effective fusion, fast switch, and efficient parallelism when serving multiple fine-tuned models.
Introduction
Recently, Large Language Models (LLMs) have achieved significant success [16, 66]. With these models being applied in diverse domains, full fine-tuning (FT) is commonly employed to enhance their downstream capabilities [56, 6, 74]. However, retraining all parameters comes with three drawbacks: (i) Full FT suffers from catastrophic forgetting, where model forgets pre-trained knowledge while acquiring new information [44, 8]. (ii) As the model and dataset sizes grow at scale, full FT becomes increasingly -enuuatiooo-eemddinn and memory-intent
[70]. (iii) It is impractical to store and serve thousands of fine-tuned LLMs on modern GPUs each requires full parameter storage [81, 60].
Parameter-efficient fine-tuning (PEFT) methods propose to address these bottlenecks by updating a small fraction of parameters [21]. Rather than merely reducing the number of learnable parameters, an ideal PEFT method should possess three key properties to be practically effective and efficient:
High Quality: It should exhibit both memorization and generalization capabilities, balancing the acquisition of new information from fine-tuning tasks with the retention of pre-trained knowledge.
Efficient Training: It should minimize the memory footprint for model gradient and optimization states, and further translate such memory efficiency into less computation and fine-tuning speedup.
Scalable Serving: It should avoid adding inference overhead when serving single PEFT model. For multiple models, new parameters should be partially stored as adapters to save memory, and allows for effective fusion [78], fast switch [33], and efficient parallelism [60] among thousands of adapters.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Step 1: Select sparsely with coupled structures
Astiaation
Modifed:
Parameeer
Trakabic
Prece:
NILA
FTX Tepe
Malishool Alcetion
Feod-frreadd Netoarh
Step 2: Compute densely after co-permutation
200
Vilad
Value Hoad'
First, we perform sparse selection of specific attention heads and channels within the coupled structures of the MHA and FFN modules. Next, we apply co- oo-pemuuationn the weight matrices both sides of these structures, enabling dense gradient computation only for the selected components. While we demonstrate S'FT by selecting the same heads/channels on both sides for clarity, approach also supports asymmetric selection strategies.
Figure 1: An Overview the FT Family for LLMs:
However, achieving all the aforementioned goals simultaneously is challenging. Common PEFT approaches, such as LoRA [27], DoRA [38], and Galore [80]. project the model's weights or gradients onto low-rank subspace. While this significantly reduces memory footprint, their performance lags behind full fine-tuning in most large-scale scenarios. Recent state-of-the-art PEFT methods have aimed improve performance but at the cost of serving efficiency. ReFT operates on a frozen base model and learns task-specific interventions on hidden representations that cannot be merged into the original model, leading to a 2.2x increase in inference latency. LISA [48] employs a coarse coarse-grained selective method by randomly freezing most Transformer blocks during optimization, which requires significantly more trainable parameters. Consequently, in scaled serving settings like S-LoRA [60], LISA can only serve at most
1 as many fine-tuned models as LoRA under the same memory budget.
TO
Prior to the of LLMs, PEFT methods based unstructured sparse fine- finettuning (SpFT) have shown a strong trade-off between low number of parameters and high model performance without sacrificing serving efficiency [63, 3, 71J. We hypothesize that SpFT, which selectively updates a small subset of model parameters, can outperform LoRA and its variants in generalization capabilities. In Figure 2, our findings across various generalization tasks support this hypothesis. However, the unstructured nature of SpFT necessitates sparse operations in computation, hindering its efficient training and scalable serving on modern hardware. This makes SpFT less practical for adapting LLMs at scale.
In this work, we propose a family of Structured Sparse Fine-Tuning (S FT² methods to "select sparsely and compute densely" (See Figure 1), thereby closing the efficiency gap in SpFT. Inspired by structured weight pruning techniques [45, 42], we first identify several coupled structures inherent in LLMs that connected by intermediate activations. For example, in the multi-heat head attention (MHA) module, each attention head in the query, key, and value projections is linked to only a few rows in the output projection. Similarly, in the feed-forward network (FFN) module, each column in the up and gate projections corresponds to a single row in the down projection. By co-permuting the matrices on both sides of these coupled structures, we can preserve the original output of these structures, with only the order of the intermediate activations changed. Exploiting this property, our ST strategically selects subset of attention heads for the MHA module and subset of channels for the FFN module. We then permute the coupled structures to connect the selected components within each linear layer into a dense submatrix. Finally, through our partial pook-propggation algorithm with only two-line code modification, S2FT performs in in-place gradient updates exclusively for all selected submatrices, boosting training efficiency by eliminating redundant forward activations and backward calculation.
Through our theoretical analysis, S2FT mitigates forgetting under distribution shifts while simplifying optimization. Empirically, F2 outperforms other PEFT methods on LLaMA and Mistral family models, improving 1. 1.2-4.14 1% on commonsense reasoning tasks and 0.6-1.9% on arithmetic reasoning ones. It also surpasses full FT by 11.5% when generalize to various domains after instruction tuning.
Finally, we conduct a comprehensive analysis to verify the training efficiency and serving scalability of F²F Compared to existing PEFT methods, FT not only saves 1.4-3.0x memory, but also increases latency by 1.5 to 2.7x, making LLM fine-tuning more accessible. Additionally, S2FT's parameter updates can be decomposed into adapters, enabling adapter fusion with smaller performance drop than LoRA. Our method also results in more scalable and efficient adapter switch and parallelism through reduced matrix multiplications, showcasing strong potential for large-scale L.M serving scenarios.

---

2 Memorization or Generalization?
In this section, we evaluate the memorization and generalization capabilities of various fine-tuning methods, including full FT. LoRA, and SpFT. We hypothesize that SpFT can generalize better to downstream tasks. To support this hypothesis, we present detailed observations and analyses. Further theoretical analysis about the generalization capabilities of the S2FT family can be found in Section 4.
Hypothesis. We hypothesize that SpFT offers superior generalization than both full FT and LoRA, while maintaining comparable memorization to LoRA with the same number of trainable parameters.
Experimental Setup. We fine-tune the Llama3-8B on the Math 10K data [28] using SpFT, LoRA, and full FT. In addition to training losses, accuracies are measured on downstream tasks in LLM- LLM-Adapeers including near aut-of-distributionn (OOD) generalization on both easy (i.e, MultiArith, AddSub, SingleEq. MAWPS) and hard (i.e, GSM8K, AQuA, SVAMP) arithmetic reasoning tasks, and far OOD generalization on commonsense reasoning ones. For PEFT methods, we set three ratios of trainable parameters (p 10,, 1%, 0.1%) and search for the optimal hyperparameters on the valid set. In SpFT, trainable parameters are selected randomly with given ratios. See details in Appendix C.
SpFT
LoRA
Full FT
Vanilla
0.87
96
57
50
0.60
46
LLMMA3-0
0.15
40
0.11
94
54
36
0.07
24
30
0.03
10
10"
10
-3
10-2
10"
19.5
10
10-2
10
25
10
10-
(a) Training Loss
(b) Near OOD Asc. (Easy)
(c) Near GOD Acc. (Hard)
(d) Far ODD Acc.
Figure 2: Accuracy comparison of SpFT, LoRA and Full FT at varying ratios of trainable parameters in various settings. SpFT exhibits strong generalization ability while full FT excels in memorization.
Observations. Figure 2 indicates several key findings. First, SpFT achieves lower training losses than LoRA when using the same ratio of trainable parameters, especially at very small ratios. This gap arises from the more complex optimization process in LoRA, which requires the simultaneous updating of two matrices [23]. Second, we observe both elevated training loss and reduced average accuracy on easier math tasks as the ratio decreases, suggesting a positive correlation between memo- rization abilities and trainable parameters. Notably, with only 10% of the parameters updated, PEFT methods learn comparable memorization abilities to full FT when trained on 10k-sample dataset.
When generalizing to complex mathematical problems or commonsense reasoning tasks, the perfor- mance ranking emerges as: SpFT > Full FT > LoRA. SpFT effectively transfers reasoning abilities to commonsense domains, while LoRA exhibits significant performance drops in far OOD gener- alization. This indicates (i) freezing a larger fraction of the parameters can retain more pre-trained abilities, and (ii) approximating high-dimensional gradients with low-rank decomposition may overfit fine-tuned data and hinder the model from generalization. Since LLMS are pre-trained on high-quality data, SpFT emerges as the preferred choice for fine-tuning on task-specific data of varying quality.
3 The F family of methods
While SpFT demonstrates strong generalization ability and good overall performance in Section 2, its unstructured nature poses challenges for efficient training and scalable serving modern hardware GPU). This is because of the need for sparse operations when storing and computing weights, gradients, and optimization states, which are significantly slower than their dense variants on GPU. This motivates our investigation into structured sparsity approaches that utilize only dense operations:
Can structured sparsity improve hardware efficiency while preserving performance by selecting sparsely but computing densely? so, how far can the the flexibility of selection be pushed in this context?
To answer this question, we design family of Structured Sparse Fine- Fine-Tuning FT² methods with dense-only computations, making PEFT effective, efficient and scalable. We begin by discovering the coupled structure in LLMs in Section 3.1. Leveraging this property, Section 3.2 introduce the selection and permutation strategies of F2F with overall pipeline illustrated in Figure 1b. In Section 3.3, we present our partial packppropagation algorithm that enables end-to-end training latency reduction.
3.1 Discover Coupled Structures in LLMs
We initiate our pursuit of flexible structured sparsity by examining the coupled structures in LLMss.
3

---

Rosidaal Connection
Residaal Cossecton Poussaticn
Prnmute
Peunue
W,
W;
Wi
w"
W
W;
W";
W';
(a) Basic Structure
(b) Residual Strueture
Figure 3: Grouped model weights with basic structure and residual structure. All highlighted weights must be permuted simultaneously. Residual structures require additional permutation during runtime.
Structure Dependency in LLMs. Inspired by prior work structured pruning [45, 17], our study start by building the dependencies between activations and weights for LLMs. Let denote an activation and W denote a weight in the model. We define In(AA) the set of parameters that directly contribute to the computation of A, and Out (A) as the set of parameters that depend on A in the com- putation of subsequent activations. The dependency between structures can be defined follows:
W1 E In(A) A Deg- (W1) = A is dependent on W1 W2 Out(A) A Deg (W2)
W2 is dependent on A
(2)
where Deg"T (W1) represents the out-degree of weight W1, and Deg (W2) represents the in-degree of weight W2. Each equation represents unqiue directional dependency between activations and weights. When both equations hold simultaneously, a coupled structure exists between W1 and W2. In Figure 3, we employ deep linear networks to illustrate two types of coupled structures in LLMs:
Basic Structures: In Figure 3a, these structures exist in both the multi-head attention (MHA) and feed-forward network (FFN) modules. Taking LLaMA as an example, in the MHA module, we consider the Query (Q), Key (K), and Value (V) projections as W1, and the Output (O) projection as W2. while Softmax QK" Vv()) acting as the activation between weight matrices. Similarly, in the FFN module, the Up (U) and Gate (G)) projections function as W,, with the Down (D) projection corresponding to W2. Here, U(x) SwiGLU(((()) serves as the activations connecting W1 and Wz-
Residual Structures: In Figure 3b, this type of coupled structures exists between the MHA and FFN modules. We further consider how residual connections influence the activations in these structures.
Permutation Invariance of Coupled Structures. Figure 3 demonstrates that Wi and W2 can be co-permuted using the same order, which only affects the order of activations between them while preserving the original output from the coupled structure. Since residual dependencies require an additional run-time step to permute the residuals, we will focus on basic dependencies in our method.
3.2 Sparse Selection and Permutation
At this point, all coupled structures within the model have been identified. The subsequent sparse selection and permutation processes are straightforward, with overall pipeline illustrated in Figure 1b.
MHA Module:
There are four linear layers in MHA module: Q, K, V, 0, 12dx For a model with h attention heads, each head i E [h] has its own projections denoted as Qi E Rdxd, K; E dxdd,
V; E dddx, and O, E Rdh xd where d, d/h is the dimension per head. Let SMHA S [b] denote a small subset of attention heads. By permuting SMHA the beginning of each weight matrix, we able to update these selected heads using dense-only operations, while keeping the other ones frozen.
FEN Module: There are three linear layers in an FFN module: U. E R and D E Rd In FT only a few channels require gradient updates. Let SFFN S [d] denote the selected channels. We can permute SPFN to the beginning of each weight matrix and only fine-tune this compact subset.
Next, we provide several strategies for identifying and selecting important subsets in each module.
S'FT-R FT²: In this strategy, a subset channels randomly selected and set to be trainable.
S'FT-W: This variant selects subsets based on the magnitude of the weights for linear layers.
3. S*FT-A: This variant selects subsets based on the magnitude of activations on a calibration set.
4. F'FT-S: Top-K subsets are ranked and selected by the product of weight and activation magnitudes.
5. S'FT-G: This variant selects subsets based on the magnitude gradients on a calibration set.
Here, and can be applied directly without pre-processing. and 4 only require a forward pass on small calibration dataset. While 5 necessitates a backward pass on this dataset, it does not store optimization states and can mitigate memory footprints for activations through gradient checkpoint- ing [18]. By default, we use F2FT-R R for fair comparison and discuss other variants in Section 5.4.

---

3.3 Partial Back- pack-propagation Algorithm
Finally, we introduce our partial back- paok-propagation algorithm with only two line modifications in PyTorch. our algorithm stores trainable channels based on their start and end positions, thereby improving training efficiency by eliminating redundant forward activations and backward calculations. def
inputs, output):
activation, weight, bias, start, end inputs
# only save partial input tensors for gradient calculation in forward ctx.
start:end], weight, bias, start, end)
def srngient_uudateepaaracco gradient, start, end): # only modify the assigned positions of weight matrices during optimization parameter[:, start:end] add_d
4 Theoretical Analysis
In this section, we theoretically explain why F demonstrates stronger generalization capabilities compared to LoRA. Following previous work [23, 79, 53, 52], we further show that S is simple and efficient in optimization by maintaining stability in both the magnitude and direction of updates.
4.1 Stronger Generalization Capability
First, we theoretically explore why S2FT demonstrates stronger generalization capabilities compared to LoRA. We consider a pre-trained L-layer deep linear network, which has been widely used to facilitate the theoretical analysis of complex DNNs [59, 30, 43, 22, 34, 5]. Let f (x) := W. Wh 1-1 W!Th. be the pre-trained deep linear network, where W E Bdexxde1,, with do p and d, q. We fine-tune the ε-th layer with low-rankness level S min din
sparsity level
8 [r. detde de-1 + Denote a class of adaptation with parameters U E Redexd and V RRd-ixdd (e, == We- UV "W
where Wite = w
L-1 Wine E [Rdxxde- and W = W.T Wirre Ip and L Lg. a transformer-basee LLM, each row of We can represent the parameters in a single attention head for the MHA module or in single channel for the FFN module.
2-1 WiT E Rdexds with
(i) (1) C RP x R9, we fine-tune frre by minimizing the empirical risk Given observations (x:" M1
R (fruvv)= = (L/n) Eie(m) [m] II - fe,.vv (x, ) via gradient descent. For LoRA, we train both low-rank matrices (U, V) in Equation (3) with d
For FT. train only V in Equation (3) with d + s and fixed U US = [ear : capi...: Ca,l where S {aI.... ag} C [de] and e is the a-th standard basis. Similar conclusions hold when we fine-tune only U. Motivated by the implicit regularization in gradient descent [77, 19. 5], we directly consider minimum norm solutions.
We consider a multiple linear regression setting. Assume that the in-distribution training data (z,)), y E RP+4 and out-of-distributtio test data (z:()), y/")) RP+4 are generated i.i.d. according to ( (x)) + c(k), k (i.
where ((k) RR is the coefficient matrix, (k) and e (k) are mean zero sub-Gaussian signal and noise with covariance matrices E11 and ! respectively. The generalization capacity is measured by the fine-tuned model's excess risk &(f) = Ellly(o) - f(x ) - inf, Ellly f'(
For these OOD data, LoRA suffers from forgetting, while FT can maintain pre-train pre-training knowledge. Assumption 4.1 (Distribution Shift). Assume that () = x Σg for some Eg RRxx,, and FTT Ug
- / < 20g (ree for some E > 0.
Assumption 4.1 states that while the covariate distribution remains unchanged, the label distribution conditioned on covariates may shift, but not exceeding a factor of 2 of the OOD risk of fere, This holds for fine-tuning with proper channel selection, where primarily the output distribution is changed. Excess Risk, Informal). Suppose Assumption 4.1 holds. Consider n + 0c. B = Whre 2+1 B00 Wire holds for some BO0) E Rdexde-1, and S rank(E1)) then,
Theorem 4.2
(S S (1 + 36 gl ()1r), (JE,TrORA ( 2 [I(B(o)
B() - 13
5

---

Theorem 4.2 indicates that the OOD risk of S'FT is bounded above by that of f while that of LoRA is bounded below by the label shift magnitude. If f already has a low risk for OOD tasks, and the label shift is significant, FT is expected to outperform LoRA. Essentially, when the OOD task deviates significantly from the FT distribution, LoRA may forget pre-trained knowledge and overfit to the FT data, compromising its generalization capabilities. See formal statements in Theorem E.8.
4.2 Simple and Efficient Optimization
Next, we explain why S is a simple and efficient optimization method. In Equation (3), F can be viewed as a LoRA variant that fixes U3 as combination of multiple orthogonal standard basis vectors while optimizing yS'FT with zero initialization. The gradient is given by 0vs"
(W T ac UF Ignore P-1 Withe and denote 8C
as G, at step t with learning rate n,
ow 6+1
DW E+1
(sea()) = [x)(2) Se,c- = UŠ ( x - US FT'' G ||0|]
x=
Since U is an orthogonal matrix, the update simplifies to (fr,(()) = -nG || |x1|.. Following LoRA+ [23], assuming that x O,, where is the width of the layers in LLMs, we expect (z) = (1) to ensure stability and feature learning in the infinite-widhe width limit [72]. S can achieve this when 7 = O while LoRA requires iJ = (1) and fγ = (nn for optimal performance. These rates become impractical for modern LLMs with very large n. Therefore, F aligns with LoRA variants that fix one matrix [52, 79], offering more stable and efficient optimization.
Furthermore, under a given sparsity level as regularization, our model simplifies optimization when approximating the full fine-tun tee-ttunngg gradients at non-zero positions. Similar to LoRA-SB [53], let Gy denote the gradient of /SSFT, The equivalent gradient G. which describes the virtual gradient of the pretrained weight matrices, can be expressed as US USSFT GV. Then, the gradient with respect to Vs
can be expressed in terms of the gradient of the pretrained weight Wie as: Gp USSFT' G. Using this relationship, our objective is to minimize the distance between the equivalent gradient and the full gradient as mingy ||G G where the optimal solution is given by Gv = (U3"FT' FFr" (ggFTTy Since USS FF is orthogonal, we have Gv = GQ. This shows that FT can keep the optimal update directions throughout the training process, establishing it as an efficient sparse optimization method.
5 Experiments
In this section, we conduct a series of experiments across three diverse benchmarks covering more than 20 datasets. Our goal to provide a rich picture of how S performs in different scenarios. Here, we compare our method with different fine-tuning strategies and categories including: (i) Full fine-tuning (FT), (ii) reparameterized fine-tuning: LoRA [27]. DoRA [38], and Galore [80], (iii) adapter-based fine-tuning: Series Adapter [26]. Parallel Adapter [24]. and LoReFT [69], (iv) prompt- based fine- finettunin: Prefix- Peeii-Tuning [36]. (v) sparse fine-tuning: LISA [48]. For a fair comparison, we keep a comparable number of trainable parameters in ST to that of LoRA. The design choices for trainable parameter allocations in FT will be detailed in Section 5.4. All other hyperparameters are selected via cross-vaas validdtiii.t........ Detailed setups and dataset descriptions are provided in Appendix E.
5.1
Commonsense Reasoning
The results of eight common sense reasoning tasks in Table show that F2 consistently outperforms existing PEFT methods in the LLaMA-7B / 13B, LLaMA2-7B and LLaMA3-SB models. Compared to LoRA and DoRA, it achieves average performance gains of 4.6% and 2.8%, respectively. Furthermore, FT also shows superior performance against recent approaches, including Galore, LoReFT, and LISA, with improvements of at least 1.0%. Remarkably, despite using less than 1 % of trainable pa- rameters, our method surpasses full FT by 0.5%. The 3.0% improvement on the LLaMA3 3-8 suggests that keeping most pre-trained parameters frozen enables better generalization to test distributions.
5.2 Arithmetic Reasoning
As showcased in Table 2, FT consistently outperforms other PEFT methods for different base models. On average, it achieves improvements of 1.3% and 0.9% over LoRA and DoRA, respectively. These results highlight the versatility and effectiveness of our approach across a diverse range of tasks. Additionally, we observe substantial improvements even when compared to Full FT for the LLaMA3-8B 8B model, particularly on complex tasks such as GSM8K and AQuA. This suggests that S'FT better preserves the original reasoning capabilities of this stronger model while acquiring new skills from the fine-tuning data, thereby validating the enhanced generalization ability of our method.

---

Table 1: Comparison among various fine-tuning methods for the LLaMA-7B/138, LLaMA2-7B, and
LLaMA3-80 models on eight commonsense reasoning tasks. Non-PEFT methods are marked in gray.
( from DoRA paper, from ReFT paper, : reproduced by us, projected trainable parameters)
Model Method
Param
BoolQ PIQA SIQA HellaSwag Wino ARC-e ARC-e OBQA Avg.
ChatGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
Full FT
100
70.3
84.2
80.1
92.3
85.4
86.6
72.8
83.4
81.9
Prefix [36]
0.11
64.3
76.8
73.9
42.1
72.1
72.9
54.0
60.6
64.6
Series [26]'
0.99
63.0
79.2
76.3
67.9
75.7
74.5
57.1
72.4
Parallel [24]]
3.54
67.9
76.4
78.8
69.8
78.9
73.7
57.3
75.2
72.2
LLaMA-7B
LoRA [27]
0.83
69.2
81.7
78.4
83,4
80.8
79.0
62.4
78.4
76.7
DoRA [38]'
0.84
68.5
82.9
79.6
84.8
80.8
81.4
65.8
81.0
78.1
Galore [80]*
0.83'
68.6
79.0
78.5
84.7
80.1
80.3
62.1
77.3
LoReFT [69]
0.03
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
LISA [48]*
9,91
70.4
82.1
78.7
92.4
82.9
84,9
70.2
78.4
80.0
SFFT (Ours)
0.81
72.7
83.7
79.6
93.4
83.5
86.1
72.2
83.4
81.8
Full FT
100
74.5
86.3
81.3
94.4
86.9
89.7
77.9
88.8
85.0
Prefix [36]
0.03
65.3
75.4
72.1
55.2
68.6
79.5
62.9
68.0
68.4
Series [26]*
0.80
71.8
83.0
79.2
88.1
82.4
82.5
67.3
81.8
79.5
LLaMA-13B Parallel [24]]
2.89
72.5
84.9
79.8
92.1
84.7
84,2
71.2
82.4
81.4
LoRA [27]
0.67
72.1
83.5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
DoRA [38]'
0.68
72.4
84.9
81.5
92.4
84.2
69.6
82.8
81.5
LoReFT [69]
0.03
72.1
95.1
86.2
73.7
84.2
S'FT (Ours)
0.65
74.2
85.7
80.7
94.9
86.4
88.4
76.3
87.8
84.3
Full FT
100
74.7
84.9
78.7
93.7
84.1
87.5
75.2
85.0
ILAMA2-7B
LoRA [27]'
0.83
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
DoRA [38]'
0.84
71.8
83.7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
S (Ours)
0.81
72.9
86.1
80.2
94.3
85.5
87.2
74.6
83.4
83.0
Full FT
100
73.9
86.2
79.1
93.1
85.8
88.1
78.2
84.0
83.6
LLaMA3-8B
LoRA [27]'
0.70
70.8
85.2
79.7
92.5
84,9
88.9
78.7
84.4
82.5
DoRA [38]'
0.71
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
S (Ours)
0.70
75.0
89.0
80.7
96.5
88.0
92.5
83.4
87.8
86.6
Table 2: Comparison among various fine-tuning methods for different models on seven math reasoning tasks. Non-PEFT methods marked in gray. (!: from LLM-Adapters paper, 2: reproduced by us)
Model
Method
Param(%) MultiArith GSMSK AddSub AQuA SingleEq SVAMP MAWPS Avg.
GPT-3.5
83.8
56.4
85.3
38.9
88.1
69.9
87.4
72.8
FT
100
98.8
43.1
91.1
20.9
94.3
60.6
88.2
71.0
LLaMA-7B
LoRA [27]
0.83
98.0
40.0
91.2
21.7
93.1
56.7
85.3
69.7
DoRA [38]*
0.84
97.3
38.9
89.6
22.4
93.9
58.4
85.3
69.4
F (Ours)
0.81
98.8
41.3
91.4
21.3
93.5
58.4
86.1
70.1
Full FT
100
98.3
47.6
92.9
26.0
95.1
65.7
88.7
73.5
LLaMA-13B
LoRA [27]
0.67
97.5
47.8
89.9
20.5
94.3
61.2
87.4
71.2
DoRA [38]*
0.68
48.1
90.6
20.9
93.9
63.8
88.2
S'FT (Ours)
0.65
48.4
90.4
22.8
95.5
63.9
87.8
72.4
Full FT
100
99.3
47.5
91.1
24.4
96.7
62.5
89.1
72.9
LLaMA2-7B LoRA [27]
0.83
97.5
44.0
91.2
20.9
94.1
59.2
85.7
70.4
DoRA [38]
0.84
98.2
43.8
90.1
24.4
94.5
59.1
89.1
71.3
S'FT (Ours)
0.81
98.5
44.3
91.1
25.2
94.7
61.8
88.2
72.0
FT
93.9
26.8
96.7
74.0
77.7
LoRA [27]
0.70
99.5
61.6
92.7
25.6
96.3
73.8
90.8
77.2
LLaMA3-8B
DoRA [38]
0.71
98.8
62.7
92.2
26.8
96.9
74.0
91.2
77.5
S'FT (Ours)
0.70
99.7
65.8
93.7
31.5
97.8
76.0
92.4
79.6
5.3 Instruction Following
Table 3 comprchensively compares various methods on eight tasks in the MT-Bench dataset [82]. observed that S²FT LISA Full FT > LoRA/Galore Vanilla for both the Mistral-7B and LLama2-7B model. This is because sparse FT methods like FT and LISA retain more pre-trained knowledge while acquiring new skills on the FT dataset, thereby generalizing better to diverse tasks in the MT-Bench dataset. Moreover, our method outperforms LISA due to its fine- greigraided and flexible selection strategy, enabling all layers to learn follow instructions on the full fine-tuning set.

---

Table Performance comparison of L.M fine-tuning methods trained on the Alpaca GPT-4 dataset.
We report the MT-Bench score as the evaluation metric. All baseline results cited from LISA.
Model
Method Writing Roleplay Reasoning
Code Math Extraction STEM Humanities Avg.
Vanilla
5.25
3.20
4.50
2.70
4.65
4.32
Full FT
5,50
4.45
5.45
2.50
3.25
5.78
4.75
5,45
4.64
LoRA
5.30
4.40
4.65
2.35
3.30
5.50
5.55
4.30
4.41
Mistral-7B
Galore
5.05
5.27
4.45
1.70
2.50
5.21
5.52
5.20
4.36
LISA
6.84
3.65
5.45
2.20
2.75
5.65
5.95
6.35
4.85
Ours
4.40
5.50
2.70
3.55
6.35
6.75
5.27
Vanilla
4.40
2.80
1.55
1.80
3,20
5.25
4.60
3.29
Full FT
5.55
6.45
3.60
1.75
2.00
6.45
7.50
4.75
LoRA
6.30
5.65
4.05
1.60
1.45
4.17
6.20
6,20
4.45
LLaMA2-7B
Galore
5.60
6.40
3.20
1.25
1.95
5.05
6.57
7.00
4.63
LISA
6.55
6.90
3,45
1.60
2.16
4.50
6.75
7.65
4.94
Ours
6.60
4.75
8.38
5.20
5.4
Design Choices for Trainable Parameter Allocations
Finally, we detail how F distribute trainable parameters across layers, modules, and channels. Uniform across Layers: Following Chen et al. [10], allocate parameters each layer uniformly. Fine-tune Important Modules: Figure analyzes the effectiveness different components in LLaMA-like Transformer Block for fine-tuning, including Query, Key, Value, Output, Up. Gate, and Down projections. To ensure a fair comparison, we maintain fixed number trainable parameters when fine-tuning each component. The results show that the effectiveness of components in fine- tuning follows the order: Query/Key < Value/Up/Gate < Output/Down. This is because Query/Key only used to measure token similarities, while others serve persistent memories of training data. Based on this finding, we allocate our parameter budget fairly to the Output and Down projections. For the LLama3-88 8B and Mistral- 7B models, only fine-tune the Down projection due to the inflexible selection in multi-query attention. Further analysis of this setting left for future research.
Query
Value
Output
Gate
Down
100
60
BoalQ
PIQA
SIQA
HollaSwag
WinoGrande
Figure 4: The impact of different components fine-tuning, including Query, Key, Value, Output, Up. Gate, and Down projection. We fix the trainable parameter budget and only fine-tune one component.
Table 4: Comparison of various channel selection strategies on the commonsense and arithmetic reasoning datasets for the LLama3-88 -B. We report the average accuracy (%) as the evaluation metric.
F-FT-W
S
FTF
S FT--
Task
F'FT-R
Large
Small
Large
Small
Large
Small
T Large
Small
Commonsense 86.6
85.9,mm
85.3
84.7.19
87.3
85.1.rs
87.2
85 1..4
86 2,00
Arithmetic 79.6
78.4.13
78.4.1
77.1.a5
80.0.
76.8.as)
79.8
77.8.
77.8
79.5pom
Selection across Channels: In Section 3.2, we discuss several strategies for channel selection. In our main experiments, we employ random selection to ensure fair comparisons with baseline methods, as these approaches treat all channels with equal importance. However, the sparse structure of FT offers controllability during fine-tuning, allowing us to prioritize important channels in the selection process to further boost performance. Table compared nine different strategies, incorporating five varying selection metrics (i.e.. random, weight, activation, weight- activatiiiinn product, and gradient), each choosing either the largest smallest values. For FT-T F'FT-S, and FT we employ 1 % the fine-tuning data as calibration set, introducing only negligible overhead during inference.
Our results demonstrate that random selection serves strong baseline due to its unbiased nature. Among heuristic metrics, selecting channels with the smallest activations (i.e., F and F²FT-S) outperforms random selection. This indicates that these channels contain less task-specife informa- tion, enabling us to inject new knowledge through fine-tuning while preserving pre-trained capabilities in other channels. In contrast, other strategies introduce bias that compromises model performance. Notably, the counterintuitive accuracy decrease F2FT-G (Large) suggests that channels with large gradients contain taskerelated pre- pre-trained knowledge, and modifying them will disrupt these abilities.

---

6
Analysis
Having demonstrated the strong generalization capability and overall performance of S we now further explore its training efficiency and serving scalability compared to other fine-tuning techniques.
6.1
Training Efficiency
To evaluate training efficiency, we examine two crucial metrics: peak memory footprint and average training latency. These numbers are measured on a single Nvidia A 100 (80G) SXM GPU, We keep comparable number of parameters for all methods. To obtain the average latency, we fine-tune the model for 50 runs, each run including 200 iterations, with 10 warmup runs excluded in measurement.
FFFT(Our))
LaRA
DoRA
LISA
LoReFT
Gal
Full FT
Memory (GB)
CUN
bsm1
2
bsm
bsm2
bsml
bsm2
bsm1
bsm2
Labencc (ms)
bs-1
bs-2
bs=1
b5-2
bo-1
bs-2
bs=1
bs+2
LLaMA2-78, 78, length 512
LLaMA2-78, length 1024
L33MA2139, length 512
LLaMA2-138, length 1024
Figure 5: Comparison of memory and computation efficiency during training on the LLaMA2-7B/ 7B/13B with varying sequence lengths and batch sizes. Average latency and peak memory usage are reported. F significantly improves training latency while reducing memory footprint compared to baselines.
In Figure 5, we thoughtfully profile F2 on various model sizes, sequence lengths, and batch sizes. Compared to Full FT, F saves 1.4-3.0x memory, and speedups fine-tuning by 1.5-2.7 times. When benchmarking against other PEFT methods, S establishes new standards for efficiency, offering average reductions of 2% in memory usage and 9% in latency. Notably, S outperforms the widely adopted LoRA, achieving about 10% improvement both metrics by avoiding the need to store new parameters and perform additional calculations. Our partial back nack-propagation algorithm further improves efficiency by saving unnecessary forward activations and backward calculations.
6.2 Serving Scalability
While FT avoids additional inference overhead for single fine-tuned model through in in-place gradient updates, we will now discuss its scalability for serving thousands of fine-tuned models. To begin, we introduce the unmerged computation paradigm of FT Given pre-trained weight matrix W/rre E d and its corresponding fine-tuned weight matrix W with sparsity level 8, we define the weight difference as AW -PPe Similar to Section 4, AW can be decomposed into the product of a weight matrix V E f and a permutation matrix U Rxx. This decomposition allows us to "unmerge" an adapter AW UVT from W. thereby sharing similarities with other adapters during inference. Following Zhong et al. [83], we consider three different adapter composition scenarios:
A dapter Fusion. To combine knowledge from multiple trained adapters, we employ weighted fusion when fine-tuning is impractical due to limited data access or computational resources. However, this approach degrades performance. In Table 5, compare the effectivvness of LoRA and S when combining adapters trained separately on commonsense and arithmetic reasoning tasks, where we consider both fine-tuning overlapped and non-overlapped parameters for different adapters in FT. Our results show that FT with non -oneovellppped parameters achieves the best performance, while the overlapped variant shows inferior results. This is because FT (non-overlap) modifies orthogonal low-rank spaces for different tasks. Similarly, LoRA largely retains task-specific capabilities during adapter fusion by optimizing low-rank projection matrices to create separate spaces for each adapter.
Table 5: Adapter Fusion Results for LoRA and FT trained on the commonsense and arithmetic reasoning datasets using the LLama3-88 8B, We report the average accuracy (%) as the evaluation metric.
LoRA
FF
Task
Commonsense
Arithmetic
Fused
Commonsense
Arithmetic
Fused (overlap)
Fused (Bon-overlap)
Commonsense 83.1
32.1
79.8.a3)
86,6
42.3
82.0
84.0.3m
Arithmetic
12.0
77.2
71.6,5m
12.8
79.6
72.2 24
75.3.43)

---

300
st scatter_ add
STT:: scatter LeRA, add
5'17/WG1 in sscattrr
Latency (itas
2.0
LoRA vatrul+ + add
Lateney (Ins
80
Latency (iss)
20
14R4 2%x Kometmul + add
(WT in tett -getter -add
1.5
60
15
40
20
0.0
1024
4096
9192
32758
1024
2048
4006
15384
32768
1000 10000 100003
Base Weight Dimersion
Basc Weight Dimcosion
Number of Adapters
(a) Switch Time on GPU
(b) Switch Time on CPU
(c) Parallelism Time on GPU
Figure 6: Comparison of latency for adapter switch and parallelism on a single linear layer. FT improves scalability for switch GPU and CPU, while saving 22% time during parallelism on GPU.
Adatter Switch. Another way to leveraging multiple adapters is to dynamically switch between them. This process involves four steps: unfusing the old adapter, unloading it from memory, loading the new adapter, and fusing it into the model. In such setting, LoRA needs two matrix multiplications (matmul) and two additions (add) on GPU whereas F only requires two sparse addition (scatter _add). In Figure 6a, we increase the base weight dimension while maintaining sparsity of 32 for F and low-rankness of 16 for LoRA. Notably, we observe that LoRA's switching time scales quadratically, while FT remains nearly constant. Moreover, in c/O-constrained scenarios such as deployment on CPU, S further accelerates adapter switch by only updating a small fraction of the original weights, reducing the volume of 1/O transfers, time compared between scatter Ladd and add in Figure 6b.
Adapter Parallelism. To serve thousands of adapters in parallel, we decompose the computation into separate batched computations for Wpre and AW following S-LoRA A [60]. While LoRA requires two matmul and one add on GPU, FT reduces this to a ma matmul, an add, and either a scatter or gather for Wi and W22 in Section 3.1. Figure 6c shows that FT achieves up to 22% faster inference
than LoRA under the same memory constraints, with more speedup as the number of adapters scales.
7 Related Work
PEFT methods reduce the fine-tuning cost for large models, which can be categorized into 4 groups:
Adapter-based Fine-tuning introduces additional trainable module into the original model. Series Adapters insert components between MHA or FFN layers [51, 26]. while parallel adapters add modules alongside existing components [24]. Recently, ReFT [69] was introduced to directly learn interventions on hidden representations. However, they introduce additional latency during inference.
Prompt-based Fine-tuning adds randomly innttaiiiii soft tokens to the input (usually as prefix) and train their embeddings while freezing the model weights [36, 40, 35]. These approaches result in poor performance compared to other groups, while come at the cost of significant inference overhead.
Reparameterized Fine-tuning utilizes low-rank projections to reduce trainable parameters while allowing operations with high-dimensionnl matrices. LoRA[27] and its recent variants like DoRA[38], AsyLoRA [84], and FLoRA [61], use low-rank matrices approximate additive weight updates during training. To alleviate the limitations of low-rank structure, other work also add or multiply orthogonal matrices to enable high-rank updating, including MoRA [29], OFT [54], and BOFT [39]. These methods require no additional inference cost as the weight updates can be merged into models.
Sparse Fine-tuning aims reduce the number of fine-tuned parameters by selecting a subset of pre-trained parameters that are critical to downstream tasks while discarding unimportant ones. This kind of methods are commonly used in the pre-LLMM era [20, 75, 64]. However, they cannot reduce the memory footprints due to their unstructured nature. Recent approaches address this limitation through three directions: (1) developing structured variants that sacrifice selection flexibility for better hardware efficiency [48, 85]. (2) incorporating sparsity into LoRA [68, 15, 41] but yield limited efficiency gains, or (3) using sparse operators for lower memory cost but slow down training [4, 49, 7].
Our work is based on the last category but achieving better performance and efficiency simultaneously. Additionally, we focus on scalable inference of PEFT methods, with FT being the only approach that enables effective fusion, rapid switching, and efficient parallelism when serving multiple adapters.
8 Conclusion
This paper introduces S²FT, a novel PEFT family that simultaneously achieves high quality, efficient training, and scalable serving for LLM fine-tuning. S2FT accomplishes this by selecting sparsely and compute densely. It selects a subset of heads and channels to trainable for the MHA and FFN modules, respectively. The weight matrices from the two sides of the coupled structures in LLMS are co-permuted to connect the selected components into dense matrices, and only these parameters are updated using dense operations. We hope FT can be considered as a successor to LoRA for PEFT.
10

---

9 Acknowledgement
We would like to thank Songlin Yang, Kaustubh Ponkshe, Raghav Singhal, Jinqi Luo, Tianqi Chen, Hanshi Sun, and Chris De Sa for their helpful discussions, and the authors of LLM-Adapters, ReFT, and DoRA for providing detailed results.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 27X1v:2303.0874, 2023. 1
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi::2303.08777 2023. 17
[3] Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560) 2021. 2
[4] Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16400 2024. 10
[5] Sanjeev Arora, Nadav Cohen. Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019. 5, 18
[6] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023. 1
[7] Kartikeya Bhardwaj. Nilesh Prasad Pandey, Sweta Priyadarshi. Viswanath Ganapathy, Rafael Esteves. Shreya Kadambi, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, et al. Rapid switching and multi-adapter fusion via sparse high rank adapters. arXiv preprint arXiv:2407. 16712, 2024. 10
[8] Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less, arXiv preprint 2005 09673, 2024.
[9] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqu: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on arrificial intelligence, volume 34, pages 7432-7439, 2020. 16, 17
[10] Jinao Chen, Aston Zhang, Xingjian Shi, Li, Alex Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces. arXiv preprint 23Xx:23311008222 2023. 8
[11] Yuxin Chen, Yuejic Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends@ in Machine Learning, 14(5):566-88) 806, 2021. 31
[12] Christopher Clark, Kenton Lee, Ming-Wes Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 10044, 2019. 16, 17
[13] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0 1803.05457, 2018. 16, 17
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reüchiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10.11110.4446.. 2021. 16, 17
[15] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint 1rXiv:2311. 16696, 2023. 10
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI- Dahlee Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The lama 3 herd of models. arXiv preprint a0Xiv:240..21788, 2024. 1

---

[17] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16091-16101, 2023,
[18] Jianwei Feng and Dong Huang. Optimal gradient checkpoint search for arbitrary computation graphs. In Procedingg of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1 11433- 11442, 2021.
[19] Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017. 5, 18
[20] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint 2rXiv:2012.07463, 2020. 10
[21] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, ct al. Paramcter-effiiient fine-tuning for large models: A comprehensive survey. arXiv preprint 24033 44008, 2024. 1
[22] Moritz Hardt and Tengyu Ma. Identity matters in deep leamning arXiv preprint arXiv: 2016.
5, 18
[23] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation large models. arXiv preprint arXiv:2402. 12354, 2024. 3, 6
[24] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpaikka, and Graham Neubig. Towards a unified view of parameter-effiiient transfer learning. arXiv preprint arXiv:21 10.004660000666.. 2021. 6, 10
[25] Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verh categorization. Proceedings of the 2014 Conference on Empirical Methods Natural I Language Processing (FMNLP), pages 523-533, 2014. 16, 17
[26] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP, In International conference machine learning, pages 2790-2799. PMLR, 2019. 6, 7, 10
[27] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 100. 09685, 2021. 2, 6, 7, 10
[28] Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Lim-adapters: An adapter family for parameter-efficiet fine-tuning of large language models. arXiv preprint arXiv:2304.011 019333, 2023, 3, 16, 17
[29] Ting Jiang, Shaohan Huang. Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun. Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter-efficient fine-tuning, arXiv preprint or 24v:2405.. 2.30, 2024. 10
[30] Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016. 5, 18
[31] Rik Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597. 2015. 16, 17
Hannaneh Hajishirzi, Ashish Sabharwal, Oren Ftzioni, and Siena Dumas Ang.
[32] Rik Koncel-Kedzions Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157, 2016. 16, 17
[33] Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, and Yunxin Liu. Lora-switch: Boosting the efficiency of dynamic 1lm adapters via aalgom-algoorthm co-design. arXiv preprint 2xXiv:2405.17741 2024.
[34] Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global.
In International conference on machine learning, pages 2902-2907. PMLR, 2018. 5, 18
[35] Brian Lester, Rami AI- Af-R,ou, and Noah Constant. The power of scale for srrameter-efficient prompt tuning. arXiv preprint arXiv:2104.0866) 2021. 10
12

---

[36] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:2101.00190 2021. 6, 7, 10
[37] Wang Ling, Dani Yogatama. Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning solve and explain algehraie word problems. arXiv preprint arXiv:/705.04440 2017. 16, 17
[38] Shih-Yang Liu, Chien-Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-Deommosed low-rank adaptation. arXiv preprint arXiv:2402.0933 09353, 2024. 2, 6, 7, 10
[39] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint 2r3iv:231 1.06243, 2023. 10
[40] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. Al Open, 2023. 10
[41] Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham, Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403. 161127403.16187, 2024. 10
[42] Zichang Liu, Jue Wang. Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song. Anshumali Shrivastava. Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient ilms at inference time. In International Conference on Machine Learning, pages 22137 221762 PMLR, 2023. 2
[43] Haihao Lu and Kenji Kawaguchi. Depth creates no bad local minima. arXiv preprint 2017. 5, 18
[44] Yun Luo, Zhen Yang. Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning arXiv preprint arXiv:2308.08747 2023.
[45] Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023. 2,
[46] Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: (80900279, 2018. 16, 17
[47] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understand- ing multimodal contrastive learning and incorporating unpaired data. arXiv preprint wrXiv:2302.06233 2023. 32
[48] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. LISA: Lay- erwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403. 17919, 2024. 2, 6, 7, 10
[49] Ashwinee Panda, Berivan Isik, Xiangyu Qi. Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in 1lms. arXiv preprint arXiv:2406. 21t.::4006.6777, 2024. 10
[50] Arkil Patel, Satwik Bhattamishra, and Navin Goyal, Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online, June 2021. Association for Computational Linguistics. 16, 17
[51] Jonas Pfeiffer, Ivan Vulié, Tryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052, 2020. 10
[52] Lai-Man Po, Yuyang Liu, Haoxuan Wa, Tianqi Zhang, Wing-Yin Yu, Zhuohan Wang. Zeyu Jiang, and Kun Sbora: Low-rank adaptation with regional weight updates. arXiv preprint arXiv:2407.05411 2024. 5,
[53] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint arXiv:2411.19552 2024. 5,
[54] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information P Processing Systems, 36:793200799977 -73622 2023. 10
13

---

[55] Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv: wet.:..0000113333 2016. 16, 17
[56] Baptiste Roziere. Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llaaa: Open foundation models for code. arXiv preprint arXiv:23008. 12950, 2023.
[57] Keisuke Sakaguchi. Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)99-106, 2021. 16, 17
[58] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint 4rXiv:1904.09725, 2019. 16, 17
[59] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. 5, 18
[60] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper. Nicholas Lee, Shuo Yang. Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 2023. 1, 2, 10
[61] Chongjie Si, Xuchui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xiaokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405. 14739, 2024. 10
[62] GW Stewart, On the continuity of the generalized inverse. SIAM Journal on Applied Mathematics, 17(1):33-45, 1969. 33
[63] Yi-Lin Sung. Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, Advances in Neural Information Processing Systems, 34:24193-244 2420-,2225522222 2021. 2
[64] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, Advances in Neural Information Processing Systems, 34:24193-24205, 2021. 10
[65] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follwwee LLaMA model. https://github com/ tatttttts-uuua 1ab/ stanford_alpaca, 2023. 17
[66] Gemini Team, Rohan Anil, Schastian Borgeaud, Yonghui Wa, lean-Baptistt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXiv preprint arXiv:2312.1180. 2023.
[67] Leo van Hemmen and Tsuneya Ando. An inequality for trace ideals. Communications in Mathematical Physics, 76:143-16 148, 1980. 33
[68] Haoyu Wang, Tianci Liu, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adapta- tion of treitriind language model for knowledge editing and fine-tuning. arXiv preprint ar ArX:::2406. 10777, 2024. 10
[69] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. ReFT: Representation finetuning for language models. arXiv preprint arXiv:2404.0359/ 2024. 6, 7. 10
[70] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine- tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. 12148,111121 2023. 1
[71] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang. Songfang Huang. and Fei Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXiv:2109.0560 006877 2021.
[72] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34:17084-77997, 2021. 6
[73] Yi Yu, Tengyao Wang, and Richard Samworth. A useful variant of the davis-kahan theorem for statisticians. Biometrika, 102(2)315-323 2015. 31
[74] Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. Chatdoctor: A medical chat model fine-tuned on Ilama model using medical domain knowledge. arXiv preprint 1400002003.14000 2023. I
14

---

[75] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprini 1xXiv:206..0119 2021. 10
[76] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830 2019. 16, 17
[77] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):077-115, 2021.
18
[78] Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter--ffliient modules with arithmetic operation. Advances Neurul Information Processing Systems, 36: 166:12989-16111 2023.
[79] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficieet low-rank adaptation for large language models fine-tuning, arXiv preprint arXiv:2308. 03303, 2023. 5,
[80] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory--fficient IIm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507 2024. 2, 6, 7, 16
[81] Justin Zhao, Timothy Wang. Wael Abid, Geoffrey Angus, Amav Garg, Jeffery Kinnison, Alex Sherstinsky. Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned Ilms that rival gpt-4, a technical report. arXiv preprint 0007 2024. 1
[82] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li. Eric Xing, et al. Judging Ilm as-a as-a-judge with mt mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 7, 17
[83] Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402.168 2024.
[84] Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi. Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson. Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters foundation models. arXiv preprint arXiv:2402. 16842, 2024, 10
[85] Ligeng Zhu, Lanxiang Hu, and Song Han. Lift: Efficient layer-wise fine-tuning for large model models. arXiv preprint arXiv:2410.1172, 2023, 10
15

---

A Limitations
While work demonstrates the effectiveness of FT for LLM fine-tuning, several promising directions remain unexplored. First, extending S to other architectures with coupled structures, such CNNs and RNNs, broaden its applicability. Second, verifying our approach beyond language tasks, particularly in large vision/multi- exploring more selection strategies can provide deeper insights into optimal fine-tuning protocols due to the controllability in F*FT. Fourth, scaling our method to larger models requires further experiments. Finally, although our work confirms the feasibility of scalable and efficient deployment during inference, developing a practical serving system for FT remains an important next step.
models, will enhance its versatility. Third,
B Broader Impacts
Since our work focuses on PEFT, it leads to reduction in hardware resource and energy consumption. Given the growing adoption of LLMS across diverse domains and the corresponding surge in fine- tuning demands, F should represent an important step toward more sustainable Al development.
C Detailed Experimental Setups for Section 2
In this study, we used SpFT, LoRA, and Full FT to fine-tune the LLaMA-3-S8 model on the Math dataset [28]. The Mathl0K dataset combines training sets from GSM8K [14], MAWPS [32], and AQuA [37], augmented with chain-of-thoughh steps generated by language models. We conducted training for 3 epochs with batch size of 64. For both PEFT methods SpFT and LoRA-we fine- tune with three ratios of trainable parameters for all linear layers: = 10%, 1%, 0.1%. The model's s performance is evaluated on both arithmetic and commonsense reasoning tasks, representing near out- of-distribution (OOD) and far OOD generalization scenarios, respectively. The arithmetic reasoning dataset comprises seven subtasks: MultiArith [55], GSM8K, AddSub [25], AQuA, Singletqq [31],
SVAMP [50], and MAWPS. The commonsense reasoning dataset includes eight subtasks: BoolQ [12].
PIQA [9]. SocialQA [58], HellaSwag [76]. WinoGrande [57]. ARC-challenge [13]. ARC-easy [13],
and OpenbookQA [46]. Based on task complexity within arithmetic reasoning (accuracy 2 90%).
we group MultiArith, AddSub, SingleEq, and MAWPS as easy subtasks, while the remaining ones are classified as hard subtasks. This stratification enables us to evaluate whether the model develops advanced reasoning abilities beyond memorizing basic arithmetic operations from the training data.
D Detailed Selection Strategies in Section 3
For the five selection strategies described in Section 3.2, will detail the methods for identifying and selecting important subsets within each linear layer of both MHA and FFN modules in L.MSs.
1. F'R FT): In this strategy, we will randomly select some heads for the MHA modules and select few channels for the FFN modules. For the output projection, all channels in the selected heads will be included to enable dense-only computation. In the up and gate projections, we will select subset of columns, while for the down projection, few trainable rows will be chosen.
2. FFF-W: This variant selects subsets based on the weight magnitudes (i.e., ||W]|2) in the MHA and FFN modules. We will test subsets corresponding to both the largest and smallest weights.
FTF-A This variant selects subsets based on the magnitude of activations (i.e., ||All2) on calibration set, using 1% of the fine-tuning data. Since collecting activations requires only forward passes, this approach maintains the same memory footprint as inference and incurs a negligible increase in training time. Similarly, we evaluate both the largest and smallest activation variants.
4. S2FT-S: The Top-K subsets are ranked and selected by the product of the weight and activation magnitudes (i.e, || [|]]) The activation values are collected in manner similar to FA-
5. FFF-G: This variant selects subsets based on the magnitude of gradients on the calibration set. Since gradients are collected without updating the model, we calculate and discard gradients layer by layer during back pnok-propagation similar to Galore [80], requiring minimal additional memory.
16

---

E Detailed Experimental Setups for Section 5
Detailed selection strategies and number of trainable parameters are presented in Section 5.
E.1 Dataset Description
Commonsense Reasoning. The commonsense reasoning dataset comprise eight subsets: BoolQ [12],
PIQA [9], SocialQA [58], HellaSwag [76]. WinoGrande [57], ARC-challenge [13], ARC-easy [13], and OpenbookQA [46]. Following the experimental setup of LLM-Adapters [28]. we split each dataset into training and test sets. Subsequently, we combine the training data from all eight tasks
into a single fine-tuning dataset and evaluate performance on the individual test dataset for each task.
Arithmetic Reasoning. We followed Hu et al. [28] and evaluated S2 on seven math reasoning tasks, including MultiArith [55], GSM8K [14], AddSub [25], AQuA [37], SingleBq [31], SVAMP [50] and MAWPS [32]. Our fine-tuning employed the Math10K dataset [28], which combines training sets from GSM8K, MAWPS, and AQuA, augmented with LM-generated chain-of-thought steps. Therefore, these three tasks are considered ID, while the remaining four are classified as OOD tasks.
Instruction Following, To further showcase F superior generalization ability, we employ the intructioo-followinng fine-tuning task with Alpaca GPT-4 dataset, which comprises 52k samples gen- erated by GPT-4 [2] based on inputs from Alpaca [65]. Performance is measured on MTT-Becch [82], featuring 80 high-quality, multi-turn questions designed to assess LLMs on eight different aspects.
E.2 Hyperparameter Description
Additional hyperparameter configurations for all tasks provided in Table 6. We maintain the same hyperparameter settings across the LLaMA-7/11 3B, LLaMA2-7B, LLaMA3-8B. and Mistral-7B models.
Table 6: Hyperparameter configurations of STFT on various base models across three tasks.
Hyperparameters Commonsense Reasoning
Arithmetic Reasoning Instruction Following
Optimizer I.R
AdamW
AdamW lc-3
Adam 2c-5
2c-4
LR Scheduler Batch size
linear
linear
cosine
16x4
16x4
16x4
Warmup Steps
100
100
Epochs
F Proofs for Theoretical Results in Section 4
Here we provide proofs for the results in Section 4.
F1 Notation
For a vector a, let ||a|| be the & norm of For d1 d2, denote set of orthogonal matrices by Od == {R E Rdd xd2 : R R = Ta For matrix A E Rddixd,, let ||A||e and || be the Frobenius norm and spectral norm of A, respectively. Denote the condition number of A by k. := ||Alllop/ A, (A). Let A be Moore-Penross inverse of A. For symmetric matrix A. denote its effective rank by (A) == tr(A)/]l A|lpp Note that r (A) < rank(A) always holds. For a, E R. we let a V b := max(a, b) and a A := min b). For a matrix A E Rdd,,,, let SVD,(A)= == $ (A)A), V (A) the top-r singular value decomposition of A, where (A) e Oair and V (4) E Ods are top-r left and right singular vectors of A, respectively, and (A) diag(A .,),, E xr is a diagonal matrix of singular values of A, where X; (A) denotes the j-th largest singular value of A. Define 4, (A) (rank(A) (A) and (A) V Mrauk( A) (A) as the left and right singular vectors of A corresponding to non-zero singular values, respectively. Define the smallest positive singular value of A as (A) Arank( (A() and let (A) Arank( (A).
For deep learning model fine-tuned on n i.i.d. samples (1r ()) C Rp x R', we say an event F occurs with high probability when P(F) 1 exp(=S((lgg (n + + 9))).
17

---

F.2 Setup
We consider multivariate regression task. Using i.i.d. samples (a. 0 in-distribution task, we fine-tune a pre-trained network frr IR + R for better prediction.
-v.") C CR IR9 from
Deep Linear Networks We consider deep linear networks of the form + WL Wi R + R", where W₁ E Rd xd 1, with d,, = q and dn = p. In comparison to multi-head attention transformers, each row of We can be viewed corresponding the parameters in single head. Let (x) = WT W L-1 Wie I TRP - IR9 represent a pre-trained neural network. pre
We denote W : WT W
L-1... We E Rdexde as the weights up to the f-th layer, and W = w W 2-1 2-1... w E Rdexdo as the weights above the f-th layer, with the promise that W = I. Deep linear networks have been widely used to facilitate the theoretical analysis of modern complex deep neural networks [59, 30, 43, 22, 34, 5].
Fine-Tuning We employ & distance as the emror metric. Given pre-trained network fee, we fine- tune its l-th layer by minimizing the empirical in-distribution risk R (f) (4/n) llv,"
(i)
f(x where (x!") y/ ) TRP IR9 are n i.i.d. observations from in-distribution task. More specifically, we consider class of rank-d adaptation defined as
fI,,,(() == W
e+1 (W?" UV W t-1
(4)
where E Rede and V E - 1xd are parameters to fine-tune. Note that by regarding multiple consecutive layers as single layer, our settings can be extended multi-layer fine-tuning.
We specifically compare two fine-tuning methods: LoRA and S
LoRA. For a fixed é E [L]. and low-rankness level 1 < r < min (de, dz-1), we train the low- rank matrices (U, V) in (4) by minimizing the empirical in-distribution risk via gradient descent. Motivated from the previous results that gradient descent has implicit regularization [77, 19, 5], we directly consider the minimum norm solutions:
(ULORA, VloRA) E arg min (U, V)Il s.t. (U. minimizes R (fe..v)-
(5)
U.V
F*FT. For a fixed l [4]. and a sparsity level 8 [r det J. we train only V in (4) with the d;-1
fixed choice of U + U FF == (ear Easi..... :..i...; which specifies s channels to fine-tune, where S {a,, 02) .,,....,.,, ] C [dr]. Here ea the standard basis vector with the a-th entry being 1. We minimize the empirical in-distribution risk via gradient descent. Similar to LoRA, we consider the following minimum norm solution:
S min | s.t. V minimizes R. R (eeuf v)
(6)
Data Generating Process As simplification of the data generating process, we consider multiple linear regression. Assume that the in-distribution data (x ()) E RP+s and duitt---ist data (2r(0) (())) E IRP+9 are generated according
y (k) (k0) (K) + (k), ee {i,
(7)
where ((A) E Rexp, and (h)) E R is the error term satisfying (2(())2(())) 0. Assume that ((k) == ( E Rox exists and (x) = 0. The signal covariance matrix is denoted by
x/ = (k) (K)) gPxP.
We define the in-distribution and out-of-distribution risks of f R Rd as: (x) (xx))))))((())) E (i
(() = (k)ly(k)
For notational brevity, we can write Wppre Whor E ROxp, Let x = (x ( In RPx"
()
( (v1)) .....Ba Ba
(1) _2xx,, and E (E ,.... e = y - B(0 x())xx)) E g Denote the
18

---

in-distribution sample covariance matrices by E. = (1/n)x x (i) E: = (1/n) EOT,
2 = (l/m))x g +0 -..e Define E:(k) = (X E À = (W
:-1 S:لك W )2/2,
A = (W - W 1 1/2 '' == (W 141); 1 = Φ. (W S ) D - B WMc
δ == B Wie + Σ Also define M == T' DE W At and M7 == T'T D W At.
Let V :- vr (A), and G( = 2W (012 W " (o)1/2 be matrix that captures the covariate shift at the ε-th layer.
We consider fine-tuning the f-th (L [L]) layer of the pre- pre-trained deep linear network (x) WL'W 2-1 W x using in-distribution observations (x." y) )
To measure the performance of models, we define the excess risks of f for the task {i. as (f) == Elly (*) - f(x(*) || - inf
y' E - ( (1(
where the infinum is taken over all square integrable functions.
F,3 Assumptions
We assume that W De W.e. * O, since otherwise W xx()) = 0 almost surely and fine-tuning the f-th layer does not improve the performance of the pre-trained model. Define the in-distribution
prediction residuals for the pre- pre-trained model frre by E/ E (0) W x0) (0))) (B@2))
WP* T]. Note that E (SPT)) = ir (x We also assume that |
[lop > O, since otherwise (fPr)) = | I = 0 and there is no room for improvement from the pre-trained model.
Next, we introduce several assumptions.
Assumption F.1 SSub-aausianntt). Assume that there exist some constants C1 C2 E (0, oo) such that (x()) c()) ) in the model 7 satisfies
"00 7 2 ei |1> To and 7' TT 2 call?'T 2 1
for any E R' and r E R?, where || lvz is the sub- Gabb-aussian norm defined as
||ll inf > 0: E[exp( (y²/v²) / s 2}
for random variable y taking values in R. Assumption F22 (Sufficiently Many Observations). Assume that +r DJ)) log + +pt q).
+ + n (A)r()) x2
| 1lop (rc() re log² (n +p+t q).
n | llop
and
(ra() re (=x0 log² (n + ppppp 4)-
Te (A2)
Assumption F.3 (Eigengap Condition). Assume that there exists some constant Cg > 0 such that
X₂ ( D W At)
C3
\ DEW (1) WPeT AT) ( DE (1 W 1 AT)
T
holds.
Assumption F.3 is necessary to identify the rank-r approximation of M, which is used derive the risk of LoRA.
Assumption F.4 (Approximate Sparsity of Channels). Assume that there exists some So C [dg]] with |Sol < 8 and > 0 such that
Σ
|leT (Were (rr11*1(00 (BO)
WPO ). s 82 8 (1 " (BO) WPrr Worres ),
holds.
19

---

Assumption F5 (Distribution Shift), Assume that ΣÚ
Σ/ Eg for some Eg E Raxd and that 2+1 U (B 2 S ( for some O.
| (Wee
Assumption F6 (Condition Number). Assume that (M) - 1, k.
S 1, k1,,, (x.)) S and (+1) k.
t-1 5 w -
Note that Assumption F.6 is not essential our analysis.
F.4 Main Results
We first demonstrate that LoRA and FT exhibit comparable memorization abilities. Next, we present formal restatement of 4.2 that combine Theorems F.10, F.1,, F.15, and Lemma F.14. Theorem F.7. Suppose that Assumptions F.I. F.2, F.3, F.4, and F.6 hold. Choose such that S 3 So holds. Let ULORA, VLORA be the LoRA adaptation matrices defined in (5). Let VS be the F adaptation matrices given U defined in (6). Then, for all sufficiently large n, the following holds with probability 1 - exp(- (Spp-- + +p- + q))): for any >> O,
E (f 1,090r vs 1 (1+ (7 FFT + (1 + 17 (I )2
hias
variance
(0) (fe,tiua vluca S 7) (T + (1 + n ')(())
(TLORA
variance )2
where
0 < (THARAA)2 -8 (f(ull) ~ (T - - (ffell) S (fme),
( FT variance
)2 2 (1>(0) llop + || llap)
sde-1 log (n + p
71 r(de + de- 1) log (n + 4)
(T LoRA
2 S ( lop + / llap)
variincee
Theorem F.8 (Restatement of Theorem 4.2). Consider the limit - co. Suppose that Assumption F.5 holds. Let ULORA, VLaRA be the LoRA adaptation matrices defined in (15). Let V be the FT adaptation matrices given U defined in (25). B() Wite BWTr, holds for some B(0 E Bddexe-11, and s, r S rank(E then,
( S (1 (1+3ee 3e (( (fPo))
g00) (fe.uuaa, 2 (B()) B D
Intuition of the proof of Theorem F.8. LoRA forgets pre-trained tasks due to its model complexity. Consider the simplest low-rank adaptation to a single-layer linear network:
A1 E arg min El - (Wfe + 11)x 12]
A x 44
rank(A()=r
Assume that x I. then we can show that the solution is A, = SVD,(BB) - W Under the condition that the rank of B(i) - WoPe is smaller than, or comparable to LoRA fine-tuned model can learn the in-distribution best regressor in & sense, since (W/Pre + A )a) Rs B = Ely (0()) x]
Hence makes LoRA fine-tuned model vulunerable to distribution shift.
On the other hand, we model FT as fine-tuning only a few channels:
A
arg min Af= Car, 2..3.
Eilyy (W + 11)00
Although SFFT is special case of LoRA, the constraint on the direction of low-rank matrix prevents overfitting the in-distribution task. To see this, note that a sparse fine-tuned model can be written as
(WPe + A wP x + Σeat eaee (B WP
WPR + > eee B(0,,
eal
aES
BESc
aES
where C [dj] is a set of channels with cardinality 8. Since SPFT keeps most of parameters from the pre-trained model, except for rows specified by S, the model forget less pre-training tasks. D
20

---

E55 Proofs for LoRA
E5.1
Excess Risk of LoRA
Lemma E9 (Excess Risk). Consider the minimum norm solution
(UAORA, VloRA) E arg min
((U, s.1. (U,V) minimizes R (Jeuv).
(U.V) xB
Then, the low-rank adaptation matrix satisfies
ULaRA =
e41 e+ E ( DSO W A')A",
and
(fa tr ((B(L) W - SSVD, W
EXA
Pre
(+1) )1 ( W A A' -2/))
E+1
(B(A) - WPP- - SW (+)++) ) 1US W At) A ))
for kE E (i,
Proof of Lemma F.9. The empirical risk of fuuuv for the in-distribution task can be written as
R (f(u,v) 1 IlB" )x
(1) |x!" + +c.0 W UV T WTT ---1=
(1)
iE|m]
tr (B(() WPe WiT UV W 1:1(00 (B() Wme
UV T W )7)
+ 2r (L(() - WP - W
W.F.P W )i: ) + tr (1(00
tr V WPrc
2 W
5:0 pro
pre 6+1 W Pre (+1U)
-2tr (Wite UV W
50 (Bi) WPnnOTT T + - })
+ir( W 5 (B() T T)
2tr (B() - WP 2,1) ) tr
(2(0
(8)
$050 Since (01) 5:01 5:00 (X(T))EET = LI Li...
R (fe,u.)) tr (ÁVU WHRT we
C+I c+1 UVTA) A) 2 (Whre UV AAt - (00DT
22tr UV - AÁ' W. $5000 DT)
+1((DS00) DT) 2 DSS0 2,0) Lr (101
- -IWer A - 1 MM
At1 - 1DS0 8-1 At
+ tr (DSS)) DT 2
DVO + tr 10
where we used (1- AÅ' WT minimizing the norm:
//111/2 21
0. From (9), minimizing R. (fe,,,) is equivalent to
[We,UVT UV À - D&S T
1 | UV Å Wit (Weee DSOW T
Atl
&+1 E+1 1-1
+ +I - Wet-WiriWA ( )
(1) Pre A A
This is minimized by (U", V') satisfying
U'V'T_
(W pte
(W (W 15ر00 ( W. At)
+ (T -
+ A² $'¹/T).
(10)
21

---

where A1, A2 E RRdex are arbitrary matrices. Since we particularly consider the minimum norm solution, we must have A = and A2 0. Hence
W y - SVD,WWEL 6+1 (W T A
Therefore, the excess risk for (i, becomes
g(0) (Seeeuaa, = I2 ((()) Wet (WW + ULORA W x(() (
tr (B(b)) W - SVD, THe
6+1 ( (+1 ) DE W ) W 201
((()) W0ne SVV, pre
e+1 ( (W و W At At )")
This concludes the proof.
E.5.2 In-distribution Excess Risk of LoRA
Let g() ((((iil))) (ffill) denote the excess risk of frr after fine-tuning all the parameters of the E-th layer under population an-distribution risk.
Theorem F.10 (Restatement of Theorem F.7: LoRA Part). Suppose that Assumptions F.I, F.2 and E3 hold. Then, the following holds with probability 1 exp(- (-22(l00 (n + p q))) For any n >
( (fe, (fe yLuRA) & (1 7) LoLRA)) + (1 + 1 LTLA
where
V (rank( DEA DJ) r) 7)
(TIoRA)3
DJ x: ( )
+ (I(
(11)
rank (DD: DT)
2 0 (M)) ( lap (A) ((r (T/ x Φp) + +r(((2) log (n + +
(TLORA
n
4') + C *C* D Tlop (A)r ( DE D + k (A)r. log (n + +p 9)
n
Note that the first term the right hand side of (11) depends on the rank of residual matrix x0 DD DT. becomes zero when rank(= S r and small when r/ rank ~ 1.
Proof of Theorem F.0.. Let WIORA_ =
Wre e+1
LOORA From Lemma F.9, we have
80 (fEULRss = ir(((( -W
W!ARA
LORA
1 1D W LoRA
w
= = LORA
AA AA'W D)
where we used (I AAt)Wer 1-1 (1)11/2 (()pppeee(()1/
0. From Lemma E99
WLORA A = SVD,
DES WoreT At) À'A A.
This gives
(W LORA
MATWPe - D) |F < |I LORA
SV-SV ( Pre
8+1 ( (+) DM Waree A A WPPeT 1-
11111/2111
+ [ISV,,WW'' e+1 (Whr
64 DE Pre A+)AttPP , 100pppee)11// _011/2 0112111
TLORA + TLoRA
We bound I LORA
variance and 7 LoRA
separately.
For the term
LORA varianes> since A A (1)
W WeeT
pre T At AF APAt.
ToORA variance
[SVD, 4+1 W DS T At At)
SVD, (W F 1 (1) W. AT)A
22

---

Therefore,
T
Whe 1 SVD, E+1 2+1) ) DE T A - SVD,, 241 (4 W
2-1 A')AA'Al Alle
2-1
S W pee (Wer) E-1) )t D W (AtA A AtAll
ToLAR variance, 1
+ T va 2
We first bound TLORA vartance, 1-
From Lemma G.1 and Assumption F.3, we have
T variance,1
2 (M) SVD,(M)l
(M)
S (M) (M) - A++1(M) VFIIMM Mlop
I < (M)CV
Mlpp:
where M
Lemma G.3, e+1 (W pe
Wht pe
(+1) ) D W T Al and M wat (W pre (+1) DE W A!. From
|| Mllpp S | DSO W T DI W 1l 1lpp
+ || 1IDD W llp Alllop
(Te( DI (1) + +re (A()) log (n + p q)
2 | ( 1og 2 (a)V
(re( DM DJ D')') + tr((4))) (n + + p g)
+ + DJ 1b6 (e.2kxx())) (A)1
2-1 llop (A)
(r((A2) (n + p + 4)
+ IDD WP
(A)
(re (Te DO ((()))))))) + tree (A2)) log 2 + +
x 1 11//2 Nie (1)l
(R3(A)r ( DI DT +
(n + p g)
| DJ 1
holds on the event F, where we used
|IDE (1) W llop I ||D 11//
lo lop- Hence
((ra(0 xI x') + r. rr())) log (n + +pp+ 4)
TLLORA variance, 1
S Cgm2(MM) 11 2 (A)
(x(((A)r DE DT) + (C))) log" (n + p p+0 4)
1/2 + (MO) DT
lop
Next we bound LORA variance.2-
Again from Lemma G.3,
TLaRA
varunce,, 2 I /r DSS0 w.preT I
lpp A |A4op
(A)
re (A?) (n + p g)
x |DE011/ op
1 1112
T 1lop
(A)
7
(.ra(A2) log (n + p 9)
| 10ph (A))
n
holds on the event F. Therefore,
(Te((TT Φ2) + (A3)) log (n + p q)
TLLRA Variance
C,R3(M)) 1/2 ke, (A)1
n
r(x3(A)r: DE DT Φh) r((AArr log (n p + g)
+ +CE DJ 11
n
(12)
23

---

hold with high probability.
Bound 7 LoRRA
Note that
bias
bias
(M) 5 - 1
E/ ,
'' DE W W Eq1/2
DE - 8-1 (A²)
W 2(11/ l
=
+ +I T'
DE W (AP) 1-1 1 0112
where the second equality follows from the fact that cross terms are zero, i.c..
(T( tr( tr(TaT)) since v, (W f-1
( //11 V.
(W M 001/2
E 1/1/// (A2)twee,E!! 001//2 and
V' Db. (SVD, = 0, W W-1
001/2(1 v, (W f-1 201/2
(W 8-1 x(1/2)) = 0
hold. Thus from Lemma F.17,
(TLoRA)2 = [SVD, (' DI WTeT At) - "T D WPeT A + E00 (ffell)) (13)
(Tius
Notice that
SVD, W Al) - क' '' DM W T A 1E
1-1
< {0 v (rank( TT DI Wrre
(-1 A!) - -r) T DE preT
2-1 1
S (0 V (rank( 2' "T DM WPeTT 2-1 AT) - 1>
OV (rank(DE)) DJ -
k R (T (
(14)
rank Bank(DD11/2
where the last inequality follows since
|D 021011222 = |A.. (DE01))) ( = rank Pank(DD.j) (22//)/22)
<2( (11// |D (1)1/21 12p
Summary Note that for any 7) > 0, (TLORA TLORAA2 < (1 (1+ (Tss (TooRA) (1 + 1/ holds. Therefore,
(T LORA
80 (fe,cunnn,yaa I (1 +1+ 1) LoRA2) + (1+ 11
)2.
variance
Combined with (12), (13), and (14), this concludes the proof.
F,5.3 Out-of-distribution Excess Risk of LoRA
We define the low-rank matrix obtained by LoRA under population in-distribution risk as
LLRA
VLLRA) aarg min [|(7, V) s.t. (U,V) minimizes RR (fe,,,))
(15)
0c
U.V
Theorem F.11 (Restatement of Theorem F.8: LoRA Part). For (ULORA, V.LOR), defined in (15)
g(o) (fe.uuaa ViARa) S [I(T Φ' [B()) 1/21 | + [l(B()) - Bdi) -
+ |[(B(o) - WPn (E()112 _001
0 (rank(DE DJ) -)) (DE D || e (1
rank(D DT)
24

---

Furthermore, for any 1y E (0,1),
(Je,uaa ViSaAa 2 (1-1) (1- BC)
/0) 2
|F -3(1 - ''TTT) BB B x
-3(1 1)]((B() w (E - -
-311-- 1) 1)00
OV (rank DT) -))
ne RD DJ (00 (src).
rank (D52 D)
(16)
Proof of Theorem F.I. With slight modification to the proof of Lemma F.9, it follows that
g(0) (fe,ouma Vilaa) tr (B W SVD, ( (W 2+1
W DI
A
R-1
(B(0) WWe - SVD, (W1+1)
= =I -V00))002--SS.00000 DM WPpe At)
A -
At
(0)11/2 2
(17)
Recall that M = 'T ( W At. Then,
(B - WP - ( D WPRT A A - A 0(001/2
$ (B - WP '' DE W
W 0/01/
+ ||MA W ( (M
= (B()) - / - 1' DEC 02)11/222111 "
00)1/22 i wor 1-1 2
(6)1
| - -/
< II(I 4' B |
| 1 (B - B(i) 201 (1.0)
+ ||
+ ||MA W
pre
D (0) SVD
< ||(T F' 
[1
n |
+ [[(Bio)
wM ) 1/2 1,0 (1)) I|
|M - (MD,)))AA /20)1 11
where we used gp /T' Wpe = Wrr From (14), we have
(fe,nnaa < [|(I - D' ( 2())) /2 + (I BO)
|
+ (1(((()) W ) (0)1122 2011/22 (2 ((
O (rank( ( DT) r) g(0 (fme), rank (ain((D)(11/)
+ (1.) [o (D>()) DT),
where we used Wfe e- 21
x- -
llo =I|G 6-1 llop- This gives the first claim.
25

---

for any 1) > 0 and any matrices A, B of the same shape, Using 2 (tt(A 2 - - (1/n)IBB
(17) can be rewritten as
glo) (fe,Uuaa
(Bll) - (1 - 2'0 )B - WPrresi
=I
D'D'T (BO) W ( 2011/
77
MA MA 2001/22 SVD, W (0)1/2
(B B 1
1F + +2r (BLO) -BO) (T T2 + T))
1|27 + +72 +
2 (1 n| (B Biy BO) 10/
+ (1- (1- + T2 + Tsl
(18)
Choose E (0, By similar argument as above, and using '' WrP Wrr we can show that
|T + T2 + Tylli s 1 + 31 + 3 Tzll
s 3l((1 - 4 5 B > (0)1/2 |/ (B( - WPy ( 23
(()))((i))
v (rank (D2 ( DJ DT)-r) k (DE! D ||]|G (lpp ()r0) (
3
rank (ank(DEF (112)
where we used (14) again. This concludes the proof.
F.6
Proofs for Structured Sparse Fine-tuning
E6.1
Excess Risk of Structured Sparse Fine-tuning
Lemma F.12 (Excess Risk). Given S C [de], consider the minimum norm solution
SS'FT E arg min
||V s.t. V minimizes RQQ ( (e,rzny y))
VeRte-1xa
Then, the structured spurse adaptation matrix satisfies
S =US -USPF (W (eriuu S SonpDDSWws (At)2,
(19)
and
glh) (Fe.ugmr tr ((B(1)) W - W U FT ( W US Sryi D W. (A 22(,0)
(At) ))
Bii)) - Wpe W US (
US
D
(1) preT
1-1
for k E (i.
Proof. Since (ilk)
F(O)T and (k)
(1/1)X(4)X(1)) have S:0))
{(()) $.0)) (X()) E(K)T= =: (1(k) ((()) Similar to (9), we have
R (Feggor)))) |W e+ US U Ā D&S0 Át 1
W AVI
tr D 22tr (DE + tr 1:00
Thus minimizing R.dl (f e/u.g y is equivalent to minimizing the norm
| FT V T Á
pre ATI T =IlW e+1 U. F A
(20)
US (W USFFTT ( e+1 ) "
WS E- A
i+1
+ +I[(1 - ( 1 U FT)
W Tre A 1..
26

---

Using the same argument as in the proof of Lemma F.9, the minimum norm solution V is obtained by
VS'F -
FTT W
e+1
The excess risk for & E {i becomes
(0 (l E
[ (k) (4) ((() - Wet (W? + UT S FTT jW i- x (2)) {
tr (B(R) - W - With UŠ
(W pre
US FT
DiS W (AT) 20100
)²)
( Wre Whe FT ( U
2+1 US
U
D
E+
This concludes the proof.
F.6.2 In-distribution Excess Risk of Structured Sparse Fine-tuning
Theorem F.13 (Restatement of Theorem F.7: F Part). Suppose that Assumptions F.I and F.2 hold. Fix S C [de] with [S] = s. Then, the following holds with probability eexxp(=S((lo + p 9))) For any n 0,
1 V (.uggm S (1 (T
(1 + 17 "1 (I
(Tariance)
where
(TS*PTT3 < 11 $5 )db (DS g(0ي (SPT)) + 80 (J!
(21)
(rre 45 TT 3 + tre (A3)) log (n + -
(1) lop (A)
s (A) ( DE 43) + Kº (A)re log (n + p g)
+ +I D llop
Note that the term (T - ""4""T (DE. (112 opp in (21) measures the distance between subspaces spanned by क' and &y in label space, weighted by t. (E())) In high level, this quantity shows the closeness between the ê-th layer full fine-tuning and F2FT. It takes small values when the important channels for residual prediction are sparsely distributed among all channels. This aligns with the intuition that F2 only selectively fine-tunes small number of coordinates, and thus relying on the information contained in those coordinates.
Proof of Theorem F.13. Using the same argument in the proof of Theorem F.10 combined with Lemma F.12, we have
50 (Fegtt = [[ U
US e+1
AA WHE 2-1 - D)
and
(W USFFT AAtWor f-1 D)
< || S
U S ( ( U S ( pre (A3) 1 - DE pre (Al)
+ the e U hhe
E+1 US F D W
1-1 dd)twrrr,>000 2011/2 D2
7 SPF STT
We bound 7 variance and SFTT separately. bins
27

---

Bound FT
Note that
variancs*
T variance
Wee s+1 US (W w+t F D T (At) W
US UT (W c+1 UU FT D A
W e+1 UŠ FT (W U 6+1 US DE
At
WW (W US DES w A'Ile
I (+1 US U
PT (Wet th
S e+1
FT D W T
TTST variance, 1 +
7 FT variance, 2-
For the term 7 Variance, 1+ using Lemma G.3,
T variance,
S 2 DEU W V5 D W |lpp |a 1lap
((r x 2 (3) - re(())) log (n + +p 9)
S 1 lp K4 (A)
s(re(""] D DJ (3) + +r(()))) log (n + pppp + 4)
II D kl (A)
holds on the event F, where the first inequality follows since the term inside the norm is at most rank-2s. Again from Lemma G.3,
T variance,2 2 I vs|l DEOW
L-1 (ooppp (At)) - ( |looplAAllpp
(A)
STc (Tc( (re(AA)log' log" Jo d + p)
D lop 1201/
W.
e-1 llop
X. (A) V
71
sre(A2) log 2 (n + p)
1D lopt
n
holds on the event F. Therefore,
8 ( (1)) (b)) + (A2)) log" (1 + + + ②)
TSSIT variance
1 2 (A)1
s(n2(A)r, DU DT xx)(A)r log² p ②)
D D 11/2//
n
(22)
Bound FT
By the same argument as in the proof of Theorem F.10.
(TS 1 DEC WT At D'
DE W AT 11 + 80 (ffill)
(23)
< 1!
ob (DE())) 113 DI WTeT A || + g00 (ffal))
2T '' 1 (DE
1a g0) (1 + g0) (f(al))
(24)
where we used | WT At 1 < |D 01/2 = ()(P) (f(r). We hypothesize that F ~ TLRRA by comparing (13) and (23), Here, SVD. (Q'0' DΣ W A!) is the best rank-s approxi- bias
mation of d'Ø'T D W At and D D W A benefits from rank-r approximation,
where r 8.
Summary Note that for any 1j > 0, (TS'FT holds. Thus
S < (1 + (T ST + (1 + 1/m) (TS2F bius
variance
bias
variance
80 { (fau < (1 (T (1 (ZZ varianee )2
Combined with (22) and (24), this concludes the proof.
Next we characterize the bias terms bius
and
S FF biss
under sparsity assumption.
28

---

Lemma F.14. Suppose that Assumption F.4 holds. Then, for sparse fine-tuned network with the
choice S 3 So. it follows that
U ( s (TbIRA)² < (I S (JE ) (E
Note that D is projection into a subspace, which is contained in subspace projected by TT Thus
Proof.
1 DE W Al L' DE (il WTT A
DE
1| 1) T DE W-pre
L-1 A
=1(+"'0's I)W &+1 W+t)) + P At
- =110"000 I)W ++1 - U FT
UŠ
U FIT (W D
W T A
=1(+""0" I) 4+1 -1 FU (W Me
2+1)
DE W A
where the last equality follows since ( $. (W Thus
U+ by definition of D'g
10'g DE TIPT AT 4'4' T/ DE AT
< li Elp (T US F
(W ) DE (012/ 10/0/// A l1
= |i 2+1 leg (01
1l
Σ
leed D
aE(d:](S
< & t+1
(W
1+1
S 8 e+1
||
D> (112//1122
where the second inequality follows from |>
11/1/2 /0''"pp Atllop S 1, Assumption F.4 and S 3 So-
The conclusion follows from (13) and (23).
F.6.3 Out-of-distribution Excess Risk of Structured Sparse Fine-tuning
Given S C [de) with |S| = 8, we define the structured sparse adaptation matrix obtained by FT under population in-distribution risk
SSFF arg min |VI| s.t. V minimizes RD (S (Se,ugmr
(25)
Theorem E.15 (Restatement of Theorem F8: S Part). Fix S C [de] with |S| = s. For S
defined in (25).
glo) (uug""" ( S (( + + 2/ (B B 100
+ 3/1 B (2/0)1122 ي (111 1
+ 3/ "h uee 1 | 0/0)122 w 20) (
Remark F.16. If there is no covariate shift, i.e., x (0)) Eg for some Σ., Theorem F.15 further gives the bound
glo) (Lo (ee e.ggeer vette) S g(0) ((0))por)))) +3 +310"00 (B()) B |
+ اا 1/22 (. - 1 "","",," w
29

---

Proof of Theorem F.15. With slight modification to Lemma F.12, we obtain
E Lee = tr (B() W We
(B
we110 US
(We+1 Whe
e+1 U+
P i DE W ( 20(0)
(BO) W W UU US
SPT FT (W 8+1 S
(At)
( WPrre 43 D W
(A "
- -100000 (B()) - w !
+ +103003 {18() W
DV
W
where we used 1211 W (Ai)2 D/
(0)11/22 (W /
(o)11//2 W 21
Note that
|T - 100 (0)
( 20 (011/2 H 1 1|:
+ 110000 Wet Wa0 -
10001/22 1
whe 001
2-1 1 1
1
2011122
}
l|r
|| W' (B(0) B(0) |
+ |0 B (E
1011
+ $ We (W (W 20)1/22
- G-1
w -4 1/
Therefore.
(o.33 = 11 10 4 (B())) - W 1 |T|
I g00) ( + 3 35 TT (B()) B&0) )0)) 120
+ 3 8 (I 1/011/2
01) (1) (i.0) )
n
+ 3 W tre 1 | 0/0)1/2 W , 20011222
1
where we used x + + = 3x + 3y + 3 This concludes the proof.
F.7 Proofs for Full Fine-tuning
x as fine-tuned network with full fine-tuning applied to the L-th layer, evaluated under the population in-distribution risk, where A is obtained by
Define (x) = Wire (W:P + A W
A full
E
arg min
E (BO20
- Whe (WWT" + A )
27
4ede
Lemma F.17 (In-distribution Excess Risk), For ffull it holds that
e (ffull)) | > (1)1 W
(A2)
+ - -1(- DE 11 Pre (A2) pre
Proof of Lemma F.17. Similar to the proof Theorem F.10, we have
g(i) 80( (!!
min AER", xdy-1
E
BBOz Pre (W
AM (00) 27
)
min A Xdc-)
|15001/2
Wi Pee Pre
1/122
30

---

and
|D
W 6+1
Aw
1-1 -σ
1/
W pre
e+1 AW 1 2011/2 p' DE Wr A l (26)
+ D - W (A9) W /Q1/3
+ (T - D' D (1) W (AP) 20112 1|:
(-1
T3
where we used the fact that the inner products (r( = tr (T2T7)) tr(TTT]) 0. By
choosing A = (W2+1)) DE w T Al for example, the term 7i becomes 0. Thus
( (1! | 11111
(A3) - |1
+ |1(I 4' DE 1 W (A ( 2
/2 1
This gives the desired result.
We obtain the following corollary as direct consequence of Lemma F.17. Corollary F.18. For ftall, it holds that
(ffll - | (DE + [I( - 2 p' ). (DE(11)))) llppp (fro))
6-1 (AR) /211 Mopp (fme)
(27)
The first term on the right hand side of (27) measures the distance between two subspaces spanned Intuitively, this quantifies the information coded at the č-th layer, and the necessary information to predict residuals. Thus, it bounds the maximum improvement by the l-th layer fine-tuning. The second term measures the subspace distance between the subspace where prediction residuals reside, and the subspace predictable by the (-th layer fine-tuning.
by V. (DE (01/2 and V (WW" =01/3) (WTr,_,01/)
G Auxiliary Results for Proofs
Lemma G.I. Fix s, di, d2 N+. For any A, B E Rdixds, Xds, if Allop S ||Allop and Xs
A (A) hold, then,
(A(A)
|SVD, SVD, (A))I (A) (A - \ (A+1(A) (Ve| Allop A Allrr.
Proof. By triangle inequality,
[B)) SVD,( = (B) (A)) : (A)
$ <B) (B) A)II: + (B) (B) - (A) Al) Alop + (B) (B) (A) (A)]I=I|A||g)
B
Using Kavis-Kahanaaa theorem (Theorem 4 from [73]). and Lemma 2.6 from [11].
up (s| 6/
Allup ^ ||B AlE
[B,( (B) - (B) ( (A)Il: <
x2(A) 2² (A)
Thus
||A|33 XA(A)
SVD, - SVD.(A))le <
(Vs) Allop A B AAlp)
X²(A) (² - 22 (A)
||
X₂ (A)
(Vs||B- - Allopp A B Allr).
(3) \a(A) - \0+1 (A)
This concludes the proof.
31

---

We cite the concentration inequality for cross covariacce matrices from [47].
Lemma G.2 (Proposition 9.1 from [47]) Let Z and Z be mean zero random vectors taking values in Red and Rda, respectively. Denote covariance matrices of 7 and 7. by Ez and Σ, respectively. Fix any > Assume that there exist constants C1, C2 > such that
7 Ezy 2 ei 1 Zl and n Ezn's 2 02 l 21
(28)
holds for any 7 Rdd. and 7 E Rdd, Choose > (re(Ez) A (r (22) + log(d1 + d2)). Let (Zi, Zi) be independent copies of (Z,Z). Then, there exists constant C C(ci, 02) such that with probability least 1 e
S | 1 1/2 (rc(Ez) + (e log(d, (1 + d2))
-
n (E(M)
n
hold.
Note that if random variable Z taking values in IR satisfies 7 "'zyy 2 cl7>' 2 for any y Rd with some constant > 0, AZ also satisfies " ΣAzy' 2 cl>" T AZ| 2 bb matrix A E IRdd xd and arbitrary d' E N+, where ΣaZ = AΣ
for any 7 gd and any
We then prove the following lemma show the existance of a 'good' high probability event bound multiple inequalities.
Lemma G.3. Suppose that Assumptions F.l and F.2 hold, Fix any [de]. Then, there exists an event F with (FF) = 1 (pS--22(lo' (n+ + q))) such that on the event F. E { V')).
| D w lop S | 11pp A| |lAt S ||All
(29)
and
1(A3))-(A3))) - (Ap) 2 (A)
(e(A)) log" (n + p + q)
(30)
X2(A)
n
re(A2) log (n p 9)
|| Allop S (A))lllll
(31)
(A) || A - (A)
re(A2) log (n + p + 9)
(32)
n
hold. Furthermore,
|| (DE01/2 DE(11122
/ 1lap
(re( TT( + rr()))) (n + +p+ 9)
1 Top
/ llp
(re(2 DV T) + +r(()))) log (n + +
+
+
D T |1 llpp
(33)
holds on the event F.
Proof. We only prove for + = n6 without loss of generality. Before proving Lemma G.3, we first derive several concentration inequalities. Assumption F.2 implies
n > r>?) log" (++
n> r. (E(() log" 2 (n +
> (ra(2())
(n
n > (re(4) y 4) ^ Aee (og
> (r.(0T T D D )) A (1 log (n + g).
32

---

Using Lemma G.2, we obtain
|| A2llop
1 W p-1 W M W
1lpp
||Allp re()) log (n + q)
(34)
n
and
S ||>20 " " 1bp2 (re(E!)) (r(eD( log2 +
- lop
(35)
1
D lpp
x | 1l
Te
log" (n pon++p+00 +
(36)
(ro( + rr(A))) log oogg(n + +
W
(37)
op
④ D(S
W
x |
D
(ra( DE
②) (r())))) log (n + p g)
n
(38)
with high probability. Hereafter we only focus on the event F where these inequalities hold. We divide the proof into parts.
Part 1. In this part derive (30), (31) and (32). Note that | A S (A7)/2 holds on the event F since n >> x² (A)r. (A2) log (n + d + p) by Assumption F.2, and hence rank(A?) = rank( A?). Using Theorem 5.2 from [62].
|[(A)] - (A lo S
R. (A -A
n. (A7))|AA -Allaal--lla
A
|AA
||A||
Again from Assumption F2, (34) gives
Ki (A²)
Ta (A2) log2 (n + p +
II(AP) - (AP)* S
X. (A2)
This yields (30). Proposition 3.2 from [67] and (34) yield,
"T (A2 - A²) 1lop
|| 1
re(AA)) log" (n + + ②)
"T 12 "T /33 I
/ A245"")
X.
where T (AP). and used AT APT"") "*"' 12 and A1/2 """TT "" /22"0///200"" we obtain (31)
(A3).
Since Á
ra(A2) log (n + + + g)
|| - Allop (A)
(39)
n
Again using Theorem 5.2 from [62] combined with Assumption F22, we obtain (32) as
x²((A) Tc(AA))og') + p) A AA--Allopp S (A)
n
This yields || I ||
Part 2. Next we derive (33). By similar argument as Part (36) and Assumption F.2,
1>
Te (2(())
log" (n + d + p)
|(
op
(40)
x
n
33

---

Since D - -D- = 500 50 10 )
( T (DS..) D W op
④ (Ō- D W T
② T D(D - 1
+
1 TT D) - >
op
op
1 I (1:)) T
4 T D'S L.Th | W TreT
+ Φ 10 (5 (S 1
+
Φ 2Q
W
Φ T (DD)) (5:))
W
+ Φ T D(C Σ W T
1 0 ( (50))
21 )
op
=: Q1 + R + Q2 + R2.
We bound Q1, 22, Rí and Ro separately. For the terms Q and Q2. (37) and (38) give
(To(DTEE D Φ) raa)) log 2 (n + p q)
Q x (
1 1/2 |lap
(41)
n
(re(TDDE)) DT 4) + re (A3)) log" (n + + q)
Q2 x DE (i) D 11 || lop
(42)
n
For the term R1, using (35) and (40),
R1 S |0 c.a llp
(1
(
lop | |,/
1-1 1lpp
1//1 lop
(ra(I!") + re (- log (n + rc((_ log + 9)
12
n
rr(E2) (r (ra( + re D )) log (n +
S <N. 1 / 1 oo
n
For the term Rg, using (35) and (36),
Rs < 100 op |(_0) 1lop -
>110/ 1 12 W W
pre
1lop
(re(): + re
log" (n + p q)
re(2
log" (n + p g)
< >|(
A
1p
3/2
71
n
Vr( (re())) + tZ!l log On p
$ Ke 2/
1/2 | 11pp
llop
n
where we used [1(:(0) llopp S
by Assumption F.2 combined with (40). Again from Assumption F.2, Ri + Ri is bounded by the right hand side of (41). Therefore,
[ T (DSS - DΣC W llop
TT x) ra(A))) log (n + p ②)
< ( 11ap 11/211/Allp
(re DV DJ 4) + re (e()) log 2 (n + + 9)
+
D 1/1///1 |
"
Finally, from Assumption F.2, we obtain This concludes the proof.
11 DS00 W llp x | |1 11 (1)1 W
011/2
(-1 op-
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions of this work.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in the Conclusion Section (section 8).
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The main assumptions and theorems provided in Section 4, while additional details and complete proofs can be found in Appendix F.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: The paper has disclosed all the information in the method and experiment part.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA].
Justification: We have the code required to reproduce our experimental results and are working towards making our code available in a public GitHub repository.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: The experimental setting is clearly described in Section 2, Section 5 and Section 6, and we will make our code available in public GitHub repository.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: All statistics and results included the paper are accompanied by confidence intervals.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Answer: [Yes].
35

---

Information for the resources required to reproduce the experiments are included the oaoer. All experiments are run with A100 (80G). For the efficiency analysis, a single A100 GPU was used.
Justification:
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://eeriiie http://p......olocccccccciiiiie.
Answer: [Yes].
Justification: The research conducted in the paper fully conforms with the NeurlPS Code of Ethics in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We discuss the broader impacts of our work in Appendix.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our paper does not introduce any assets that have high risk for misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We have explicitly mentioned the citations for the datasets and have ensured that all conditions are fully respected.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: Upon acceptance, we will make our codebase publicly available and complete documentation for our assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: We do not include any experiments with human subjects or crowdsourcing.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: We do not include any experiments with human subjects crowdsourcing.
36


Reviews:
[{"review_id": "mcjoSX3zr3", "reviewer": "Reviewer_CFWk", "summary": "This paper introduces a new family of methods called Structured Sparse Fine-Tuning (S$^2$FT) for large language models (LLMs). S$^2$FT aims to achieve state-of-the-art fine-tuning performance, training efficiency, and inference scalability simultaneously. The method selects a few heads in the multi-head attention (MHA) module and a few channels in the feed-forward network (FFN) module for each Transformer block, then co-permutes the weight matrices to connect these selected components. This results in multiple compact, dense, and trainable weight submatrices that are updated during fine-tuning. The approach prevents overfitting and forgetting, delivers superior performance on benchmarks, and improves memory and throughput efficiency compared to full fine-tuning and existing parameter-efficient fine-tuning (PEFT) methods.", "questions": "Could the authors elaborate on the scaling of trainable parameters, as well as the space and time complexity, compared to other SFT-based or LoRA-based methods?\n\nCould the authors elaborate on how the hyperparameters are chosen, and provide a more comprehensive list of hyperparameters used in this paper, the procedure to choose them, and the discarded hyperparameters?\n\nRight now, the paper mainly discusses how to apply S$^2$FT to attention-based LLMs. The reviewer wonders how to identify the important weights and apply this method to other models.\n\nMost of the experiments were performed on LLaMA and LLaMA2. The reviewer wonders if there is any difficulty in applying the method to LLaMA3.\n\nIs it possible that different datasets/tasks could result in a different set of \"important weights\"? The reviewer wonders if the authors explored this possibility.\n\nThe reviewer wonders if the author could release the source code into an anonymous repository for review purposes.\n\nIn the NeurIPS checklist, the author claims that their results include confidence intervals. Could the authors make some comments on the confidence intervals for the tables in the paper?", "limitations": "The limitations and broader impacts are addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a method combining structured sparsity with fine-tuning, enhancing both efficiency and performance.\nThe method proposed can reduce memory costs and improve throughput.\nThe method demonstrates strong generalization capabilities.\nThe approach allows for scalable batched serving of multiple fine-tuned models without additional inference overhead.\nThe experiments are comprehensive.", "weaknesses": "While the results look impressive, the reviewer is not fully convinced by the theoretical motivations. For example, the paper relies on the assumption that structured sparsity can effectively represent the necessary model adaptations, which is hard to verify in realistic settings. \n\nAlthough the benchmarks are comprehensive, most of the benchmarks do not include SFT or other SFT-based methods. Since S$^2$FT is in some sense an SFT-based method, it would be necessary to include such comparisons to understand if the performance improvements come from the \"structure\" (i.e. the $^2$) or SFT itself. Based on the current benchmarks, it is difficult to make such conclusions.\n\nWhile the paper includes some implementation details, aiding reproducibility, the code is not provided, making it hard to verify or reproduce the results, especially for a paper with mostly empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the insightful and valuable comments! They are very helpful for further improving the clarity and quality of our paper. We'll revise our manuscript in the camera-ready version to address all of your concerns.\n\n**Q1**: How can we verify the model adaptation ability in real-world settings?\n\n**A1**: To verify the model's adaptation ability in real-world settings, we evaluate its performance in an out-of-distribution (OOD) scenario, as shown in Table 2 in the original paper. Our method, S$^2$FT, significantly outperforms both LoRA and Full FT by a large margin in this task, demonstrating its superior model adaptation ability. Additionally, we conduct an experiment on arithmetic reasoning tasks to demonstrate that S$^2$FT significantly outperforms LoRA in OOD settings (See **Empirical results for generalization task** part from rebuttal to all reviewers).\n\n---\n\n**Q2**: Ablation study between SFT and S$^2$FT.\n\n**A2**: Thank you for your suggestion. Our results in Table R2 show that SFT primarily leads to performance improvement, while our structure enhances efficiency and scalability, as discussed in Section 6 in the original paper.\n\n**Table R2**: Performance comparison between S$^2$FT and SFT on the commonsense reasoning tasks.\n\n| Model        | SFT | S$^2$FT |\n| - | - | - |\n| LLaMA-7B | 81.2 | 81.8 |\n| LLaMA2-7B | 83.7 | 83.0 |\n| LLaMA3-8B | 87.2 | 86.6 |\n\n---\n \n**Q3**: Reproduction of the paper.\n\n**A3:** We strongly agree that open-sourcing the code is critical for reproducibility and supporting future research in this area. We therefore will definitely make our code publically available and results easy to reproduce upon acceptance.\n\n---\n\n**Q4**: How are the hyperparameters chosen?\n\n**A4**: Thank you for highlighting the importance of detailing our hyperparameter configuration. We have included a more comprehensive list of the hyperparameters used in this paper in Tables R4 and R5. Most hyperparameters follow previous work [2, 3], and we only tune the learning rates.\n\n**Table R4**: Hyperparameter configurations of S$^2$FT for LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B on the commonsense reasoning tasks.\n| Hyperparameters (S$^2$FT) | LLaMA-7B | LLaMA-13B | LLaMA2-7B | LLaMA3-8B |\n| - | - | - | - | - |\n| LR  | 1e-6         | 1e-6         | 1e-6         | 1e-6         |\n| LR Scheduler  | Linear       | Linear       | Linear       | Linear       |\n| Optimizer| AdamW| AdamW        | AdamW        | AdamW        |\n| Batch size| 16 | 16           | 16           | 16           |\n| Warmup Steps | 100          | 100          | 100          | 100          |\n| Epochs | 3 | 3 | 3   | 3   |\n| Where | Up,Down,Gate | Up,Down,Gate | Up,Down,Gate | Up,Down,Gate |\n\n\n**Table R5**: Hyperparameter configurations of S$^2$FT for LLaMA2-7B and Mistral-7B on the instruction-following task.\n\n| Hyperparameters (S$^2$FT) | Mistral-7B | LLaMA2-7B |\n| - | - | - |\n| LR | 2e-5         | 1e-5         |\n| LR Scheduler | Cosine | Cosine |\n| Optimizer|AdamW| AdamW|\n| Batch size|4|4|\n| Warmup Steps|100|100|\n| Epochs  | 1 | 1 |\n| Where  | Up,Down,Gate | Up,Down,Gate |\n\n---\n \n**Q5**: How to identify the important weights?\n\n**A5**: In Section 5.2, we discuss our weight selection strategy. For a standard transformer architecture, we first uniformly allocate the trainable parameters across different transformer layers. Next, we freeze all attention modules and evenly assign parameters to the Up, Gate, and Down projection modules. Since these three modules represent a coupled structure within the FFN module, we randomly select the same channels to update for each module. In Table 3, we also introduce an alternative activation-based selection strategy, which resulted in inferior performance. We leave the exploration of more advanced metrics to identify important weights as a topic for future research.\n\n---\n \n**Q6**: How to apply S$^2$FT to different models?\n\n**A6**: Thank you for your interest in the application of S$^2$FT across different model architectures. It is indeed versatile and can be applied to Transformers, CNNs, RNNs, and GNNs.\n\nAs detailed in Section 3.2 of our work, S$^2$FT can be utilized in the Multi-Head Attention and Feed-Forward Network modules of standard Transformer architecture, which is sufficient for most LLMs and diffusion models currently. Additionally, such coupled structures in Figure 3(a) also exist in CNNs, RNNs, and GNNs as shown in [1]. Therefore, S$^2$FT is effective across various model architectures.\n\n---\n \n**Q7**: Most of the experiments were performed on LLaMA and LLaMA2. The reviewer wonders if there is any difficulty in applying the method to LLaMA3.\n\n**A7**: In Table 1, we present the results of LLaMA3-8B on commonsense reasoning tasks following DoRA and ReFT. S$^2$FT achieves an average improvement of 1.4%, demonstrating its applicability to LLaMA3. For other experiments, we primarily focus on LLaMA and LLaMA2 to ensure a fair comparison, as these base models are commonly used in our baseline methods.\n\n---\n\n**Q8**: Could the authors make some comments on the confidence intervals for the tables in the paper?\n\n**A8**:  Thank you for your suggestion. We have added the full results with confidence intervals in the Appendix, and included comments in the tables in the main paper, which will be visible in the camera-ready version. \n\n---\n \n**Q9**: Could different datasets/tasks result in a different set of \"important weights\"?\n\n**A9:** See **The affect of selecting different trainable parameters on different datasets/tasks** part from rebuttal to all reviewers.\n\n---\n\n**References**:\n\n[1] Fang G, Ma X, Song M, et al. Depgraph: Towards any structural pruning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 16091-16101.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J2ewo1ydQ8", "reviewer": "Reviewer_8C1L", "summary": "Current PEFT methods for LLMs fail to achieve high quality, efficient training, and scalable serving simultaneously. To overcome this, the authors developed Structured Sparse Fine-Tuning (S²FT), which excels in all three areas. S²FT improves generalization by selecting a few heads in the Multi-Head Attention (MHA) and channels in the Feed-Forward Network (FFN) modules for each Transformer block. It forms dense, trainable submatrices by co-permuting weight matrices, preventing overfitting and forgetting. S²FT achieves state-of-the-art performance and reduces fine-tuning memory usage by up to 3 times and increases throughput by 1.5-2.7 times.", "questions": "1. The author has some ablations on the trainable allocation choice within a block, what about the allocation in different layers?\n2. Can this paper be combined with quantization just like QLoRA.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and organized. The proposed method is technically sound. \n2. Efficient tuning of LLMs is an important topic.\n3. The performance of the proposed method is very promising.", "weaknesses": "1. The author claimed that S2FT prevent overfitting and forgetting. However except the toy experiments in 2nd section, I did not see any experimental proof. E.g. The author only gives MTBench results after the finetuning on AlpacaGPT4, what about the overfitting and forgetting issues?\n2. Experiments only conducted on relative small models.\n3. The name SFT may incur some confusions as it already has the meaning of supervised fine-tuning.\n4. The experiments parts are not sufficient to proof the effectiveness of the proposed method especially on some downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we will revise our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.\n\n**Q1**: Empirical Results about the overfitting and forgetting issues?\n\n**A1**: To verify that S$^2$FT can prevent overfitting and forgetting issues, we evaluate its performance in an out-of-distribution (OOD) scenario using MT-Bench, as shown in Table 2 of the original paper. Additionally, we conduct an experiment on arithmetic reasoning tasks, demonstrating that S$^2$FT significantly outperforms LoRA in both near OOD and far OOD settings.\n\nIn Table 2, we present the results of training the model on the Alpaca GPT-4 dataset and evaluating it on the MT-Bench benchmark, representing an OOD setting. Our method significantly outperforms both LoRA and Full FT in this task, demonstrating that it leads to much less forgetting and better generalization to new datasets.\n\nTo further address your concern, we trained LLaMA-7B/13B on the Math10K dataset and evaluated its performance on three in-distribution (ID) tasks (GSM8K, AQuA, MAWPS), four near OOD tasks (SVAMP, MultiArith, AddSub, SingleEq), and one far OOD task (MMLU). The results are shown in Table R1.\n\n**Table R1**: Performance comparison between S$^2$FT and LoRA for LLaMA-7B/13B on the arithmetic reasoning tasks and MMLU benchmark.\n\n| Method | Average (ID) | Average (Near OOD) | MMLU (Far OOD) |\n| - | - | - | - |\n| LoRA (LLaMA-7B)  | 45.1 | 78.7     | 27.8 |\n| S$^2$FT (LLaMA-7B)  | 46.6 | 81.7     | 33.1 |\n| LoRA (LLaMA-13B)  | 49.9 | 81.6     | 36.3 |\n| S$^2$FT (LLaMA-13B)   | 50.5 | 84.2     | 42.2 |\n\nAccording to the results, S$^2$FT led to improvements of 4.3%, 3.0%, and 1.4% for LLaMA-7B in far OOD, near OOD, and ID settings, respectively, compared to LoRA. For LLaMA-13B, the corresponding improvements are 5.9%, 2.6%, and 0.6%. As the distribution difference between the training data and test data increases, the performance gap between S$^2$FT and LoRA enlarges, demonstrating that our method is effective in preventing overfitting and forgetting. We will include these results in the camera-ready version of our paper if accepted.\n\n---\n\n**Q2**: Experiments on larger models.\n\n**A2**:  Thank you for your suggestion. Following LISA, we have added experiment results for LLaMA2-70B on MT-Bench and GSM8K in Table R2. The results show that S$^2$FT outperforms other PEFT methods for larger models,  providing strong evidence to support S$^2$FT’s scalability under large-scale training scenarios. We will include these results in the camera-ready version of our paper if accepted.\n\n**Table R2**: Performance comparison between different methods for LLaMA2-70B on MT-Bench and GSM8K.\n\n| Method  | MT-Bench | GSM8K |\n|-|-|-|\n| Vanilla | 5.19  | 54.8   |\n| LoRA    | 6.10 | 59.4   |\n| LISA    | 6.72 | 61.1   |\n| Full FT      | 6.25   | **67.1** |\n| S$^2$FT |**6.91**|64.7|\n\n---\n\n**Q3**: The name SFT may incur some confusions as it already has the meaning of supervised fine-tuning.\n\n**A3**: Thank you for your suggestion. We will replace \"SFT\" with \"sparse FT\" in the camera-ready version to avoid any confusion.\n\n---\n\n**Q4**: The experiment parts are not sufficient.\n\n**A4**: In the original paper, we included experiments with five different base models and more than ten tasks, covering both commonsense reasoning and instruction-following tasks. We have also added some new experiments in Table R1 and R2. We hope these findings provide more comprehensive results of our method, and we will include them in the camera-ready version of our paper if accepted.\n\nIn Table R1, we further present experimental results for arithmetic reasoning tasks, demonstrating the effectiveness of our method across various downstream tasks. The results in near OOD and far OOD settings further highlight its ability to address overfitting and forgetting issues. Additionally, in Table R2, our experiments on LLaMA2-70B showcase S$^2$FT's scalability in large-scale training scenarios.  \n\n---\n\n**Q5**: What about the allocation in different layers?\n\n**A5**: Thank you for pointing out this issue. In Section 5.3, we maintain a uniform allocation of parameters across different layers. To further address your question, we have added an ablation study in Table R3 concerning layer-wise allocation. This study includes the following design patterns: (i) Increasing ($n_{i+1} > n_i$): the number of trainable parameters in every layer gradually increases (or remains the same); (ii) Uniform ($n_{i+1} = n_i$: the number of trainable parameters in every layer is the same; and (iii) Decreasing ($n_{i+1} < n_i$): the number of trainable parameters in every layer gradually decreases. \nThe results in Table R3 indicate that maintaining a uniform distribution of trainable parameters across different layers leads to the best performance. A more detailed analysis in this direction will be left for future research.\n\n**Table R3**: Performance of different allocation strategies on commonsense reasoning tasks for LLaMA-7B.\n\n| Allocation Strategy    | Average Accuracy | \n|-|-| \n| Increasing | 81.2     | \n| Uniform | **81.8** | \n| Decreasing  | 80.4 |\n\n---\n\n**Q6**: Can this paper be combined with quantization just like QLoRA\n\n**A6**: Thank you for highlighting the potential of combining S$^2$FT with quantization. Our method can indeed be integrated with quantization, similar to QLoRA, by using mixed precision storage. Once the trainable parameters are determined, we retain these parameters in their original precision while quantizing the other parameters to low bits. This approach enables quantized PEFT, similar to QLoRA. By maintaining the trainable parameters as small, dense submatrices after permutation, our storage remains relatively hardware-efficient, even with mixed precision. We plan to conduct more experiments in this direction in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IhSGluv6ay", "reviewer": "Reviewer_d9g6", "summary": "The paper introduces a structured pruning method for LLMs. The main idea is to permute the rows and columns of the weight matrices and select a submatrix during the fine-tuning process. The authors show that the proposed technique outperform previous techniques in terms of accuracy and efficiency.", "questions": "Will the authors open-source the code?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written. The authors gave a good summary of existing parameter-efficient fine-tuning (PEFT) methods. While the proposed technique itself is not entirely new (it can be considered as a special form of structured sparse training), the authors described the technique clearly. \n2. Theoretical analysis is provided for generalization performance. \n3. Experiments are comprehensive. The authors tested their technique for two types of tasks (commonsense reasoning and instruction-following) with five fine-tuning categories on more than 10 datasets.", "weaknesses": "Novelty in the pruning technique itself is limited. It is basically a structured sparse training method. Similar structured pruning methods have been proposed on non-LLM models previously (e.g., [27, 24]).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.\n\n**Q1**: Novelty in the pruning technique itself.\n\n**A1**: Thank you for your comment. We acknowledge that structure sparsity is commonly used in model pruning. However, we are the first to adopt structure sparsity for the parameter-efficient fine-tuning (PEFT) of large language models (LLMs). Therefore, our method is a gradient selection technique instead of a pruning technique. Motivated by the challenges of practical efficiency and scalability in previous SFT-based methods, we use coupled structures for flexible and fine-grained gradient selection, introducing a completely new gradient selection strategy. This idea is both novel and effective in this research line. \n\nBefore the era of LLMs, methods like Diff pruning [1] and Fish Mask [2] mainly focused on unstructured selective fine-tuning, as model sizes were not very large. These methods used a binary mask to enable sparse gradient updates during training, which led to large memory footprints and time costs. In the era of LLMs, researchers additionally prioritize the memory efficiency of PEFT methods during training and scalable serving ability during inference, leading to the popularity of LoRA. SFT-based methods like LISA [3] only enable layer-wise selection and have limitations in serving scalability. In comparison, our method addresses these efficiency bottlenecks and revitalizes SFT-based approaches, surpassing LoRA in both performance and efficiency. Given the current research trends in PEFT methods, our approach is novel in this area.\n\n---\n\n**Q2**: Will the authors open-source the code?\n\n**A2**:  We strongly agree that open-sourcing the code is critical for reproducibility and supporting future research in this area. We therefore will definitely make our code publically available and results easy to reproduce upon acceptance.\n\n\n---\n\n**References**:\n\n[1] Guo D, Rush A M, Kim Y. Parameter-efficient transfer learning with diff pruning[J]. arXiv preprint arXiv:2012.07463, 2020.\n\n[2] Sung Y L, Nair V, Raffel C A. Training neural networks with fixed sparse masks[J]. Advances in Neural Information Processing Systems, 2021, 34: 24193-24205.\n\n[3] Pan R, Liu X, Diao S, et al. LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning[J]. arXiv preprint arXiv:2403.17919, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAIdcDenlw", "reviewer": "Reviewer_Cfys", "summary": "This paper aims to address the fact that the current parameter efficient fine-tuning methods cannot simultaneously achieve high-quality, efficient training or scalable LLM services. Therefore, a series of structured sparse fine-tuning methods for LLM are proposed, which simultaneously achieve state-of-the-art fine-tuning performance, training efficiency and reasoning scalability. S2FT achieves this through \"sparse selection and dense computation\". It only selects a few heads in the MHA module and a few channels in the FFN module in each Transformer block. Then the weight matrix of the coupling structure in the LLM is rearranged so that the selected parts are interconnected to generate multiple compact and dense trainable weight sub-matrices. S2FT only needs to update these sub-matrices for efficient parameter fine-tuning. The paper focuses on the fact that current methods such as LoRA, DoRA, etc., although they can reduce memory, are not as good as fine-tuning in large language models. The unstructured characteristics of SFT require matrix sparse operations, which makes service scalability and training efficiency impossible to guarantee. This is the main reason and motivation for the proposal of S2FT in this paper. In addition, this paper establishes an interface between S2FT and LoRA to support joint or non-joint computing paradigms.", "questions": "1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless. \n2. There are few figures for the method in this paper, only the right picture of Figure 2 is shown, and the overall picture is not clear enough. In addition, does this method extract the same layer for each different model? Or is it better to extract specific layers for each model? \n3. This paper locks the U of the low-rank decomposition matrix, which is a good attempt, but for different data sets, will the distribution of data affect the data form of the U matrix and the form of the U matrix basis? \n4. In the experimental results, there are many phenomena that the method used in this paper is lower than the previous methods. Can the author analyze each task more fully, because it can be seen from the table that some tasks are more obviously degraded?", "limitations": "This paper mentions some limitations of this paper in the limitations. For example, although various model architectures show coupled structures, this paper does not explore the opportunity to extend S2FT to other architectures. At the same time, in terms of model deployment, although this paper verifies the feasibility of scalable services in the work, there is still a lack of a practical and scalable actual service system. However, when using this method, is it necessary to consider the fairness or importance of each layer? For example, for a large language model, the first layer may contain more information, and some information in the middle layer can be appropriately eliminated. \nSecondly, whether it is truly quality-oriented needs to be considered because S2FT does not perform well in any task in the experiment of this paper. Finally, this paper also mentions that the subsequent optimization direction may be targeted at the residual dependency network, and how to perform S2FT on this type of network needs further consideration. However, since the work of this paper mainly focuses on PEFT, it leads to a reduction in GPU computing resource consumption. Therefore, this method has the potential to have a positive impact on the computing environment by minimizing the computing resources required for fine-tuning LLM.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of ​​this paper is very novel and valuable. It starts with the three problems that the current LLM model may encounter, namely training quality, efficient training, and service scalability. It proposes a structured sparse fine-tuning method to accelerate the large language model and provides corresponding mathematical proofs. This method is very effective for both in-distribution and out-of-distribution model training. The highlights of this paper are as follows:\n1. The mathematical theoretical proof of this paper is very complete, the proof ideas are very clear, and the proofs of the in-distribution model and the out-of-distribution model are very complete. The conclusions and theoretical analysis of this paper are clearly seen, and the mathematical theory is very solid.\n2. The analysis process of this paper is complete. The advantages and disadvantages of the current research are analyzed in place in the literature review part, the method part is concise and clear, and a complete theoretical proof is provided. The advantages of operating efficiency and service scalability are also analyzed in the experimental part.\n3. The simplicity and clarity of this method is a highlight of this paper. Compared with the complicated fine-tuning methods of other methods, this paper only needs to select and rearrange the matrices of some layers for updating, and develops a partial back-propagation algorithm. The number of lines of code implemented is very small. Simplicity is a big highlight of this paper.", "weaknesses": "The advantages of the S2FT method in this paper are very prominent, and it provides some references for efficient, scalable, and high-quality fine-tuning methods for large language models. However, the author can consider the following areas for improvement or sufficient explanation: \n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless. \n2. There are few figures for the method in this paper, only the right picture of Figure 2 is shown, and the overall picture is not clear enough. In addition, does this method extract the same layer for each different model? Or is it better to extract specific layers for each model? \n3. This paper locks the U of the low-rank decomposition matrix, which is a good attempt, but for different data sets, will the distribution of data affect the data form of the U matrix and the form of the U matrix basis? \n4. In the experimental results, there are many phenomena that the method used in this paper is lower than the previous methods. Can the author analyze each task more fully, because it can be seen from the table that some tasks are more obviously degraded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We will carefully revise our paper based on your comments. Our responses to your questions are detailed below. We would greatly appreciate your input on whether our revisions address your concerns.\n\n**Q1**: The symbols in the proof process of this paper can be slightly organized.\n\n**A1**: Thank you for pointing out this issue. In the camera-ready version of our work, we will simplify the symbols and add additional notations before our proof to improve readability.\n\n---\n\n**Q2**: There are few figures for the method in this paper, and the overall picture is not clear enough.\n\n**A2**: Thank you for your suggestion. We will include more figures to illustrate our selection and permutation strategy specific to transformer architectures in detail. The current Figure 2 serves as an abstract demonstration of each coupled structure in a standard Transformer model. Additionally, we will ensure that our figures are clearer in the camera-ready version.\n\n---\n\n**Q3**: Does this method extract the same layer for each different model? \n\n**A3**:  Our method employs the same extraction strategy for all models discussed in our paper. Specifically, we always extract all feed-forward network (FFN) layers for Transformer models, with the trainable parameters uniformly distributed across the Up, Down, and Gate Projection Layers. This approach is based on our findings in Figure 4, which show that fine-tuning FFN layers results in better performance compared to fine-tuning Attention layers under the same budget. To demonstrate the effectiveness of our uniform allocation strategy among FFN layers, we have included an additional ablation study in Table R1. \n\nIn Table R1, we have added an ablation study regarding layerwise allocation of trainable parameters. This study includes the following design patterns: \n\n 1. Increasing ($n_{i+1} > n_i$): the number of trainable parameters in every layer gradually increases (or remains the same); \n2. Uniform ($n_{i+1} = n_i$: the number of trainable parameters in every layer is the same; \n3. Decreasing ($n_{i+1} < n_i$): the number of trainable parameters in every layer gradually decreases; \n4. Random One: only one randomly selected layer has trainable parameters. \n\nThe results show that maintaining a uniform distribution of trainable parameters among different layers leads to the best performance. We will include these results in the camera-ready version of our paper if accepted. \n\n**Table R1**: Performance of different layer allocation strategies on commonsense reasoning tasks.\n\n| Allocation Strategy | Average Accuracy | \n|-|-| \n| Increasing | 81.2| \n| Uniform | **81.8**| \n| Decreasing  | 80.4 |\n| Random One  | 79.9 |\n\nWhile our allocation strategy is both model- and task-independent, there is great potential to further improve performance by selecting trainable parameters based on model weights or downstream tasks. We leave this problem for future research.\n\n---\n\n**Q4**: Will the distribution of data affect the data form of the U matrix and the form of the U matrix basis?\n\n**A4**: Thank you for your question. Our work employs a random selection strategy for channel selection in Section 5.3, which is task-independent. To demonstrate how different U matrices affect performance on various tasks, we used three different random seeds for channel selection. The results for both commonsense reasoning and math reasoning tasks are presented in Table R2. Our findings indicate that the optimal form of the U matrix varies across different tasks, suggesting that the \"best U matrices\" should be task-specific. We have left this direction for future research.\n\n**Table R2**: Ablation study of channel selection strategies on the commonsense reasoning and math reasoning tasks.\n\n| Selection Strategy | Commonsense | Math |\n|-|-|-|\n| Seed: 0 | **82.2** | 68.6 |\n| Seed: 42 | 81.5 | **70.1** |\n| Seed: 1234 | 81.0 | 69.4 |\n\n**Q5**: Can the author analyze each task more fully?\n\n**A5**: We would like to clarify that in Sections 5.1 and 5.2, we train the base model on a single dataset and evaluate it across several sub-tasks. Therefore, the average performance should be the primary focus, as our method achieves significant improvements in both settings. These results sufficiently demonstrate the superiority and robustness of S$^2$FT. It is expected that our method may not always perform the best in every sub-tasks as different sub-tasks will affect each other. Nonetheless, S$^2$FT consistently achieves the best or second-best performance in sub-tasks, which cannot be characterized as \"obviously degraded.\"\n\nTo further address your concerns, we have provided more in-depth analyses of the results in Table 1 and Table 2 of the original paper. For the commonsense reasoning tasks in Table 1, S$^2$FT achieves the best performance in most sub-tasks, while maintaining second-best performance in the remaining tasks. These results demonstrate the superiority and robustness of S$^2$FT in memorizing common knowledge. \n\nTable 2 further verifies this conclusion, showing that S$^2$FT outperforms LoRA, especially in memorization tasks such as Writing or Humanities within instruction-following scenarios. It also excels in tasks requiring pre-trained knowledge, such as those in STEM. This indicates that S$^2$FT's performance improvements are primarily due to its enhanced ability to memorize new information and retain pre-trained knowledge. For other tasks, such as Reasoning and Math, which focus on multi-hop reasoning with limited knowledge, the performance of LoRA and S$^2$FT is similar. It is common for LoRA to slightly outperform S$^2$FT in some of these tasks. Consequently, SFT-based methods are more effective for knowledge-rich tasks. In comparisons among SFT-based methods, LISA performs slightly better than S$^2$FT in some tasks. This is because LISA updates much more parameters than S$^2$FT, resulting in similar behavior and better performance in certain tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers [R1(CFWk), R2(8C1L), R3(d9g6), R4(Cfys)] for their thoughtful and highly supportive feedback! We were glad that the reviewers found the problem significant and interesting [R2], the observations and theoretical analysis insightful and highly valuable [R3, R4], the methods novel, simple, and effective [R2, R4], the presentation easy to follow [R2, R3], experimental results comprehensive and impressive [R1, R2, R3]. 

We have updated the paper to incorporate constructive suggestions, which will show in the camera-ready version. We summarize the major changes:

* **[ R1, R3] Empirical results for generalization tasks:** In Table R1, we trained LLaMA-7B/13B on the Math10K dataset and evaluated its performance on three in-distribution (ID) tasks (GSM8K, AQuA, MAWPS), four near OOD tasks (SVAMP, MultiArith, AddSub, SingleEq), and one far OOD task (MMLU). As the distribution difference between the training data and test data increases, the performance gap between S$^2$FT and LoRA enlarges, demonstrating that our method is effective in generalization by preventing overfitting and forgetting.

   **Table R1**: Performance comparison between S$^2$FT and LoRA for LLaMA-7B/13B on the arithmetic reasoning tasks and MMLU benchmark.

    | Method                     | GSM8K | AQuA | MAWPS | SVAMP | MultiArith | AddSub | SingleEq | Average (ID) | Average (Near OOD) | MMLU (Far OOD) |
    | -------------------------- | ----- | ---- | ----- | ----- | ---------- | ------ | -------- | ------- | ----------- | ---------- |
    | LoRA (LLaMA-7B)  | 37.5  | 18.9 | 79    | 52.1  | 95         | 83.3   | 84.4     | 45.1 | 78.7     | 27.8 |
    | S$^2$FT (LLaMA-7B)             | 35.8  | 22   | 81.9  | 57.1  | 93.3       | 87.3   | 89.2     | 46.6 | 81.7     | 33.1 |
    | LoRA (LLaMA-13B) | 47.5  | 18.5 | 83.6  | 54.6  | 94.8       | 87.3   | 89.8     | 49.9 | 81.6     | 36.3 |
    | S$^2$FT (LLaMA-13B)           | 45.8  | 21.7 | 84    | 63    | 95         | 87.3   | 91.5     | 50.5 | 84.2     | 42.2 |


* **[ R2 ] Experimental results for large-scale language models**:
    We have added experiment results for LLaMA2-70B on MT-Bench and GSM8K in Table R2. The results show that S$^2$FT outperforms other PEFT methods for larger models,  providing strong evidence to support S$^2$FT’s scalability under large-scale training scenarios. We will include these results in the camera-ready version of our paper if accepted.

    **Table R2**: Performance comparison between different methods for LLaMA2-70B on MT-Bench and GSM8K.

    | Method  | MT-Bench | GSM8K |
    |---------|-----------|--------|
    | Vanilla | 5.19      | 54.8   |
    | LoRA    | 6.10      | 59.4   |
    | LISA    | 6.72  | 61.1   |
    | Full FT      | 6.25      | **67.1** |
    | S$^2$FT |**6.91**|64.7|

* **[ R2, R4 ] Ablation Study for layerwise allocation strategy**: For the allocation among different layers, we maintain a uniform allocation strategy, meaning we fairly assign trainable parameters to all layers. Our study in Table 3 includes four design patterns: (i) Increasing ($n_{i+1} > n_i$): the number of trainable parameters in every layer gradually increases (or remains the same); (ii) Uniform ($n_{i+1} = n_i$: the number of trainable parameters in every layer is the same; and (iii) Decreasing ($n_{i+1} < n_i$): the number of trainable parameters in every layer gradually decreases; (iv) Random One: only one randomly selected layer has trainable parameters. The results show that maintaining a uniform distribution of trainable parameters among different layers leads to the best performance.


    **Table R3**: Performance of different layerwise allocation strategies on commonsense reasoning tasks for LLaMA-7B.

    | Allocation Strategy    | Avg. Accuracy | 
    |-----------------------|---------------| 
    | Increasing            | 81.2     | 
    | Uniform   | **81.8**          | 
    | Decreasing  | 80.4         |
    | Random One  | 79.9         |

* **[R1, R4] The affect of selecting different trainable parameters on different datasets/tasks:**  As detailed In Section 5.3, we use a random selection strategy for channel selection. We apply the same three random seeds to determine the selection strategies for both commonsense reasoning and math reasoning tasks. The results are shown in Table R3. The results indicate that the best channel selection strategy varies across different tasks. Therefore, the "important trainable parameters" are different for each task. 

    **Table R4**: Ablation study of channel selection strategies on different tasks.

    | Selection Strategy    | Commonsense | Math |
    |-----------------------|---------------|-----------------------|
    | Seed: 0            | **82.2**  | 68.6 |
    | Seed: 42 | 81.5    | **70.1** |
    | Seed: 1234 | 81.0       | 69.4 |

Meta-review: The reviewers generally found the paper well written, the proposed method to be technically sound and easy to understand, the work well motivated, and supported by the comprehensive experimental results. Reviewer's concerns during the rebutal centred on the theoretical analysis, confusing naming of the method, novelty of the pruning method itself, complex notation used in theoretical analysis and lack of explanatory figures. Thanks to the significant discussion and efforts of the author's and reviewers, it appears most of these concerns have been addressed and the reviewers are unanimous in accepting the paper.
Decision: Accept (poster)