Title: S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity
Venue: NeurIPS 2024
Accepted: yes
Keywords: PEFT; LLM Efficiency; LLM Post-training
Abstract: Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable serving, but not all three simultaneously.  
To address this limitation, we investigate sparse fine-tuning and observe a remarkable improvement in generalization ability. 
Utilizing this key insight, we propose a family of Structured Sparse Fine-Tuning (S${^2}$FT) methods for LLMs, which concurrently achieve state-of-the-art fine-tuning performance, training efficiency, and inference scalability. S${^2}$FT accomplishes this by "selecting sparsely and computing densely". Based on the coupled structures in LLMs, \model selects a few attention heads and channels in the MHA and FFN modules for each Transformer block, respectively. Next, it co-permutes the weight matrices on both sides of all coupled structures to connect the selected subsets in each layer into a dense submatrix. Finally, S${^2}$FT performs in-place gradient updates on all selected submatrices.
Through theoretical analyses and empirical results, our method prevents forgetting while simplifying optimization, delivers SOTA performance on both commonsense and arithmetic reasoning with 4.6% and 1.3% average improvements compared to LoRA, and surpasses full FT by 11.5% when generalizing to various domains after instruction tuning. 
Using our partial back-propagation algorithm, S${^2}$FT saves training memory up to 3$\times$ and improves latency by 1.5-2.7$\times$ compared to full FT, while achieving an average 10\% improvement over LoRA on both metrics. We further demonstrate that the weight updates in S${^2}$FT can be decoupled into adapters, enabling effective fusion, fast switch, and efficient parallelism when serving multiple fine-tuned models.

S 2 FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity

Abstract

Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable serving, but not all three simultaneously. To address this limitation, we investigate sparse fine-tuning and observe a remarkable improvement in generalization ability. Utilizing this key insight, we propose a family of Structured Sparse Fine-Tuning ( S 2 FT ) methods for LLMs, which concurrently achieve state-of-theart fine-tuning performance, training efficiency, and inference scalability . S 2 FT accomplishes this by 'selecting sparsely and computing densely'. Based on the coupled structures in LLMs, S 2 FT selects a few attention heads and channels in the MHA and FFN modules for each Transformer block, respectively. Next, it co-permutes the weight matrices on both sides of all coupled structures to connect the selected subsets in each layer into a dense submatrix. Finally, S 2 FT performs in-place gradient updates on all selected submatrices. Through theoretical analyses and empirical results, our method prevents forgetting while simplifying optimization, delivers SOTA performance on both commonsense and arithmetic reasoning with 4.6 % and 1.3 % average improvements compared to LoRA, and surpasses full FT by 11.5 % when generalizing to various domains after instruction tuning. Using our partial back-propagation algorithm, S 2 FT saves training memory up to 3 × and improves latency by 1.5-2.7 × compared to full FT, while achieving an average 10% improvement over LoRA on both metrics. We further demonstrate that the weight updates in S 2 FT can be decoupled into adapters, enabling effective fusion, fast switch, and efficient parallelism when serving multiple fine-tuned models.

1 Introduction

Recently, Large Language Models (LLMs) have achieved significant success [16, 1, 66]. With these models being applied in diverse domains, full fine-tuning (FT) is commonly employed to enhance their downstream capabilities [56, 6, 74]. However, retraining all parameters comes with three drawbacks: (i) Full FT suffers from catastrophic forgetting, where a model forgets pre-trained knowledge while acquiring new information [44, 8]. (ii) As the model and dataset sizes grow at scale, full FT becomes increasingly computation-demanding and memory-intensive [70]. (iii) It is impractical to store and serve thousands of fine-tuned LLMs on modern GPUs if each requires full parameter storage [81, 60].

Parameter-efficient fine-tuning (PEFT) methods propose to address these bottlenecks by updating a small fraction of parameters [21]. Rather than merely reducing the number of learnable parameters, an ideal PEFT method should possess three key properties to be practically effective and efficient:

High Quality : It should exhibit both memorization and generalization capabilities, balancing the acquisition of new information from fine-tuning tasks with the retention of pre-trained knowledge.

Efficient Training : It should minimize the memory footprint for model gradient and optimization states, and further translate such memory efficiency into less computation and fine-tuning speedup.

Scalable Serving : It should avoid adding inference overhead when serving a single PEFT model. For multiple models, new parameters should be partially stored as adapters to save memory, and allows for effective fusion [78], fast switch [33], and efficient parallelism [60] among thousands of adapters.

𝑊

!

𝑊

"

𝑊

#

Query

Key

Value

Split

Split

Key Heads

Split

Value Heads

Attention

𝑊

$

MHAOutput

MHAInput

𝑊

%

𝑊

&

Up

Gate

MHAInput

𝑊

"

!

Value

′

Split

Value Heads

′

A

ttention′

𝑊

$

!

MHAOutput

Step 2: Compute densely after co-permutation

𝑊

'

Activation

FFN Output

Multi-head Attention

Feed-forward Network

Activation:

Parameter:

Trainable:

Frozen:

Modified:

Saved:

𝑊

!

Up

′

𝑊

'

!

Activation

′

FFN Output

&

Figure 1: An Overview of the S 2 FT Family for LLMs : First, we perform sparse selection of specific attention heads and channels within the coupled structures of the MHA and FFN modules. Next, we apply co-permutation to the weight matrices on both sides of these structures, enabling dense gradient computation only for the selected components. While we demonstrate S 2 FT by selecting the same heads/channels on both sides for clarity, our approach also supports asymmetric selection strategies.

However, achieving all the aforementioned goals simultaneously is challenging. Common PEFT approaches, such as LoRA [27], DoRA [38], and Galore [80], project the model's weights or gradients onto a low-rank subspace. While this significantly reduces memory footprint, their performance lags behind full fine-tuning in most large-scale scenarios. Recent state-of-the-art PEFT methods have aimed to improve performance but at the cost of serving efficiency. ReFT operates on a frozen base model and learns task-specific interventions on hidden representations that cannot be merged into the original model, leading to a 2 . 2 × increase in inference latency. LISA [48] employs a coarse-grained selective method by randomly freezing most Transformer blocks during optimization, which requires significantly more trainable parameters. Consequently, in scaled serving settings like S-LoRA [60], LISA can only serve at most 1 10 as many fine-tuned models as LoRA under the same memory budget.

Prior to the era of LLMs, PEFT methods based on unstructured sparse fine-tuning (SpFT) have shown a strong trade-off between low number of parameters and high model performance without sacrificing serving efficiency [63, 3, 71]. We hypothesize that SpFT, which selectively updates a small subset of model parameters, can outperform LoRA and its variants in generalization capabilities. In Figure 2, our findings across various generalization tasks support this hypothesis. However, the unstructured nature of SpFT necessitates sparse operations in computation, hindering its efficient training and scalable serving on modern hardware. This makes SpFT less practical for adapting LLMs at scale.

In this work, we propose a family of Structured Sparse Fine-Tuning ( S 2 FT ) methods to 'select sparsely and compute densely' (See Figure 1), thereby closing the efficiency gap in SpFT. Inspired by structured weight pruning techniques [45, 42], we first identify several coupled structures inherent in LLMs that are connected by intermediate activations. For example, in the multi-head attention (MHA) module, each attention head in the query, key, and value projections is linked to only a few rows in the output projection. Similarly, in the feed-forward network (FFN) module, each column in the up and gate projections corresponds to a single row in the down projection. By co-permuting the matrices on both sides of these coupled structures, we can preserve the original output of these structures, with only the order of the intermediate activations changed. Exploiting this property, our S 2 FT strategically selects a subset of attention heads for the MHA module and a subset of channels for the FFN module. We then permute the coupled structures to connect the selected components within each linear layer into a dense submatrix. Finally, through our partial back-propagation algorithm with only two-line code modification, S 2 FT performs in-place gradient updates exclusively for all selected submatrices, boosting training efficiency by eliminating redundant forward activations and backward calculation.

Through our theoretical analysis, S 2 FT mitigates forgetting under distribution shifts while simplifying optimization. Empirically, S 2 FT outperforms other PEFT methods on LLaMA and Mistral family models, improving 1.2-4.1% on commonsense reasoning tasks and 0.6-1.9% on arithmetic reasoning ones. It also surpasses full FT by 11.5% when generalize to various domains after instruction tuning.

Finally, we conduct a comprehensive analysis to verify the training efficiency and serving scalability of S 2 FT. Compared to existing PEFT methods, S 2 FT not only saves 1.4-3.0 × memory, but also increases latency by 1.5 to 2.7 × , making LLM fine-tuning more accessible. Additionally, S 2 FT's parameter updates can be decomposed into adapters, enabling adapter fusion with smaller performance drop than LoRA. Our method also results in more scalable and efficient adapter switch and parallelism through reduced matrix multiplications, showcasing strong potential for large-scale LLM serving scenarios.

2 Memorization or Generalization?

In this section, we evaluate the memorization and generalization capabilities of various fine-tuning methods, including full FT, LoRA, and SpFT. We hypothesize that SpFT can generalize better to downstream tasks. To support this hypothesis, we present detailed observations and analyses. Further theoretical analysis about the generalization capabilities of the S 2 FT family can be found in Section 4.

Hypothesis. We hypothesize that SpFT offers superior generalization than both full FT and LoRA, while maintaining comparable memorization to LoRA with the same number of trainable parameters.

Experimental Setup. We fine-tune the Llama3-8B on the Math10K data [28] using SpFT, LoRA, and full FT. In addition to training losses, accuracies are measured on downstream tasks in LLM-Adapters, including near out-of-distribution (OOD) generalization on both easy (i.e, MultiArith, AddSub, SingleEq, MAWPS) and hard (i.e, GSM8K, AQuA, SVAMP) arithmetic reasoning tasks, and far OOD generalization on commonsense reasoning ones. For PEFT methods, we set three ratios of trainable parameters ( p = 10% , 1% , 0 . 1% ) and search for the optimal hyperparameters on the valid set. In SpFT, trainable parameters are selected randomly with given ratios. See details in Appendix C.

10

3

10

2

10

1

(a) Training Loss

0.03

0.07

0.11

0.15

0.60

0.67

LLaMA3-8B

10

3

10

2

10

1

(b) Near OOD Acc. (Easy)

23.5

24

94

95

96

10

3

10

2

10

1

(c) Near OOD Acc. (Hard)

19.5

20

53

54

55

56

57

10

3

10

2

10

1

(d) Far OOD Acc.

25

30

35

40

45

50

SpFT

LoRA

Full FT

Vanilla

Observations. Figure 2 indicates several key findings. First, SpFT achieves lower training losses than LoRA when using the same ratio of trainable parameters, especially at very small ratios. This gap arises from the more complex optimization process in LoRA, which requires the simultaneous updating of two matrices [23]. Second, we observe both elevated training loss and reduced average accuracy on easier math tasks as the ratio decreases, suggesting a positive correlation between memorization abilities and trainable parameters. Notably, with only 10% of the parameters updated, PEFT methods learn comparable memorization abilities to full FT when trained on a 10k-sample dataset.

When generalizing to complex mathematical problems or commonsense reasoning tasks, the performance ranking emerges as: SpFT > Full FT > LoRA. SpFT effectively transfers reasoning abilities to commonsense domains, while LoRA exhibits significant performance drops in far OOD generalization. This indicates (i) freezing a larger fraction of the parameters can retain more pre-trained abilities, and (ii) approximating high-dimensional gradients with low-rank decomposition may overfit fine-tuned data and hinder the model from generalization. Since LLMs are pre-trained on high-quality data, SpFT emerges as the preferred choice for fine-tuning on task-specific data of varying quality.

3 The S 2 FT family of methods

While SpFT demonstrates strong generalization ability and good overall performance in Section 2, its unstructured nature poses challenges for efficient training and scalable serving on modern hardware (e.g., GPU). This is because of the need for sparse operations when storing and computing weights, gradients, and optimization states, which are significantly slower than their dense variants on GPU. This motivates our investigation into structured sparsity approaches that utilize only dense operations: Can structured sparsity improve hardware efficiency while preserving performance by selecting sparsely but computing densely? If so, how far can the flexibility of selection be pushed in this context? To answer this question, we design a family of Structured Sparse Fine-Tuning ( S 2 FT ) methods with dense-only computations, making PEFT effective, efficient and scalable. We begin by discovering the coupled structure in LLMs in Section 3.1. Leveraging this property, Section 3.2 introduce the selection and permutation strategies of S 2 FT, with overall pipeline illustrated in Figure 1b. In Section 3.3, we present our partial back-propagation algorithm that enables end-to-end training latency reduction.

3.1 Discover Coupled Structures in LLMs

We initiate our pursuit of flexible structured sparsity by examining the coupled structures in LLMs.

Residual Connection

W

1

W

2

W′

1

W′

2

Permute

W

1

W

2

Residual Connection

W′

1

W′

2

+ Permutation

Permute

(a) Basic Structure

W

1

W

2

W′

1

W′

2

Permute

W

1

W

2

Residual Connection

W′

1

W′

2

Residual Connection

+ Permutation

Permute

(b) Residual Structure

O

O

O

O

O

O

O

O

O

O

Structure Dependency in LLMs. Inspired by prior work on structured pruning [45, 17], our study start by building the dependencies between activations and weights for LLMs. Let A denote an activation and W denote a weight in the model. We define In( A ) as the set of parameters that directly contribute to the computation of A , and Out( A ) as the set of parameters that depend on A in the computation of subsequent activations. The dependency between structures can be defined as follows:

where Deg + ( W 1 ) represents the out-degree of weight W 1 , and Deg -( W 2 ) represents the in-degree of weight W 2 . Each equation represents a unqiue directional dependency between activations and weights. When both equations hold simultaneously, a coupled structure exists between W 1 and W 2 . In Figure 3, we employ deep linear networks to illustrate two types of coupled structures in LLMs:

Basic Structures : In Figure 3a, these structures exist in both the multi-head attention (MHA) and feed-forward network (FFN) modules. Taking LLaMA as an example, in the MHA module, we consider the Query ( Q ), Key ( K ), and Value ( V ) projections as W 1 , and the Output ( O ) projection as W 2 , while Softmax ( QK ⊤ ) V ( x ) acting as the activation between weight matrices. Similarly, in the FFN module, the Up ( U ) and Gate ( G ) projections function as W 1 , with the Down ( D ) projection corresponding to W 2 . Here, U ( x ) · SwiGLU ( G ( x )) serves as the activations connecting W 1 and W 2 .

Residual Structures : In Figure 3b, this type of coupled structures exists between the MHA and FFN modules. We further consider how residual connections influence the activations in these structures.

Permutation Invariance of Coupled Structures. Figure 3 demonstrates that W 1 and W 2 can be co-permuted using the same order, which only affects the order of activations between them while preserving the original output from the coupled structure. Since residual dependencies require an additional run-time step to permute the residuals, we will focus on basic dependencies in our method.

3.2 Sparse Selection and Permutation

At this point, all coupled structures within the model have been identified. The subsequent sparse selection and permutation processes are straightforward, with overall pipeline illustrated in Figure 1b.

MHAModule : There are four linear layers in a MHA module: Q,K,V,O ∈ R d × d . For a model with h attention heads, each head i ∈ [ h ] has its own projections denoted as Q i ∈ R d × d h , K i ∈ R d × d h , V i ∈ R d × d h , and O i ∈ R d h × d , where d h = d/h is the dimension per head. Let S MHA ⊆ [ h ] denote a small subset of attention heads. By permuting S MHA to the beginning of each weight matrix, we are able to update these selected heads using dense-only operations, while keeping the other ones frozen.

FFN Module : There are three linear layers in an FFN module: U, G ∈ R k × d and D ∈ R d × k . In S 2 FT, only a few channels require gradient updates. Let S FFN ⊆ [ d ] denote the selected channels. We can permute S FFN to the beginning of each weight matrix and only fine-tune this compact subset.

Next, we provide several strategies for identifying and selecting important subsets in each module.

S 2 FT-R (S 2 FT) : In this strategy, a subset of channels is randomly selected and set to be trainable.

S 2 FT-W : This variant selects subsets based on the magnitude of the weights for linear layers.

S 2 FT-A : This variant selects subsets based on the magnitude of activations on a calibration set.

S 2 FT-S : Top-K subsets are ranked and selected by the product of weight and activation magnitudes.

S 2 FT-G : This variant selects subsets based on the magnitude of gradients on a calibration set.

Here, 1 and 2 can be applied directly without pre-processing. 3 and 4 only require a forward pass on a small calibration dataset. While 5 necessitates a backward pass on this dataset, it does not store optimization states and can mitigate memory footprints for activations through gradient checkpointing [18]. By default, we use S 2 FT-R for a fair comparison and discuss other variants in Section 5.4.

3.3 Partial Back-propagation Algorithm

Finally, we introduce our partial back-propagation algorithm with only two line modifications in PyTorch. our algorithm stores trainable channels based on their start and end positions, thereby improving training efficiency by eliminating redundant forward activations and backward calculations.

4 Theoretical Analysis

In this section, we theoretically explain why S 2 FT demonstrates stronger generalization capabilities compared to LoRA. Following previous work [23, 79, 53, 52], we further show that S 2 FT is simple and efficient in optimization by maintaining stability in both the magnitude and direction of updates.

4.1 Stronger Generalization Capability

First, we theoretically explore why S 2 FT demonstrates stronger generalization capabilities compared to LoRA. We consider a pre-trained L -layer deep linear network, which has been widely used to facilitate the theoretical analysis of complex DNNs [59, 30, 43, 22, 34, 5]. Let f pre ( x ) := W pre L W pre L -1 . . . W pre 1 x be the pre-trained deep linear network, where W pre ℓ ∈ R d ℓ × d ℓ -1 , with d 0 = p and d L = q . We fine-tune the ℓ -th layer with low-rankness level r ≤ min { d ℓ , d ℓ -1 } or sparsity level s = ⌊ r · d ℓ + d ℓ -1 d ℓ -1 ⌋ . Denote a class of adaptation with parameters U ∈ R d ℓ × d and V ∈ R d ℓ -1 × d as f ( x ) := W pre ( W pre + UV ⊤ ) W pre x,

where W pre ℓ := W pre L W pre L -1 . . . W pre ℓ ∈ R d L × d ℓ -1 and W pre ℓ := W pre ℓ W pre ℓ -1 . . . W pre 1 ∈ R d ℓ × d 0 with W pre 0 = I p and W pre L = I q . In a transformer-based LLM, each row of W ℓ can represent the parameters in a single attention head for the MHA module or in a single channel for the FFN module.

Given n observations ( x ( i ) i , y ( i ) i ) ⊂ R p × R q , we fine-tune f pre by minimizing the empirical risk R ( i ) n ( f ℓ,U,V ) := (1 /n ) ∑ i ∈ [ n ] ∥ y ( i ) i -f ℓ,U,V ( x ( i ) i ) ∥ 2 via gradient descent. For LoRA, we train both low-rank matrices ( U, V ) in Equation (3) with d ← r . For S 2 FT, we train only V in Equation (3) with d ← s and fixed U ← U S 2 FT S := [ e a 1 ; e a 2 ; . . . ; e a s ] , where S = { a 1 , . . . , a s } ⊂ [ d ℓ ] and e a is the a -th standard basis. Similar conclusions hold when we fine-tune only U . Motivated by the implicit regularization in gradient descent [77, 19, 5], we directly consider minimum norm solutions.

We consider a multiple linear regression setting. Assume that the in-distribution training data ( x ( i ) , y ( i ) ) ∈ R p + q and out-of-distribution test data ( x ( o ) , y ( o ) ) ∈ R p + q are generated i.i.d. according to

where B ( k ) ∈ R q × p is the coefficient matrix, x ( k ) and ϵ ( k ) are mean zero sub-Gaussian signal and noise with covariance matrices Σ ( k ) x and Σ ( k ) ϵ , respectively. The generalization capacity is measured by the fine-tuned model's excess risk E ( f ) := E [ ∥ y ( o ) -f ( x ( o ) ) ∥ 2 ] -inf f ′ E [ ∥ y ( o ) -f ′ ( x ( o ) ) ∥ 2 ] .

For these OOD data, LoRA suffers from forgetting, while S 2 FT can maintain pre-training knowledge.

Assumption 4.1 (Distribution Shift) . Assume that Σ ( i ) x = Σ ( o ) x = Σ x for some Σ x ∈ R p × p , and ∥ ( W pre ℓ +1 U S 2 FT S )( W pre ℓ +1 U S 2 FT S ) † ( B ( o ) -B ( i ) )Σ 1 / 2 x ∥ 2 F ≤ ε 2 E ( o ) ( f pre ) for some ε > 0 .

Assumption 4.1 states that while the covariate distribution remains unchanged, the label distribution conditioned on covariates may shift, but not exceeding a factor of ϵ 2 of the OOD risk of f pre . This holds for fine-tuning with proper channel selection, where primarily the output distribution is changed.

Theorem 4.2 (Out-of-distribution Excess Risk, Informal) . Suppose Assumption 4.1 holds. Consider n →∞ . If B ( i ) = W pre ℓ +1 ˜ B ( i ) W pre ℓ -1 holds for some ˜ B ( i ) ∈ R d ℓ × d ℓ -1 , and s ≤ rank(Σ ( i ) f ) , then,

Theorem 4.2 indicates that the OOD risk of S 2 FT is bounded above by that of f pre , while that of LoRA is bounded below by the label shift magnitude. If f pre already has a low risk for OOD tasks, and the label shift is significant, S 2 FT is expected to outperform LoRA. Essentially, when the OOD task deviates significantly from the FT distribution, LoRA may forget pre-trained knowledge and overfit to the FT data, compromising its generalization capabilities. See formal statements in Theorem F.8.

4.2 Simple and Efficient Optimization

Next, we explain why S 2 FT is a simple and efficient optimization method. In Equation (3), S 2 FT can be viewed as a LoRA variant that fixes U S 2 FT S as a combination of multiple orthogonal standard basis vectors while optimizing V S 2 FT with zero initialization. The gradient is given by ∂ L S 2 FT =

∂V

Since U S 2 FT S is an orthogonal matrix, the update simplifies to ∆ f ℓ,t ( x ) = -ηG ⊤ || x || 2 . Following LoRA+ [23], assuming that x = Θ n (1) , where n is the width of the layers in LLMs, we expect ∆ f ℓ,t ( x ) = Θ(1) to ensure stability and feature learning in the infinite-width limit [72]. S 2 FT can achieve this when η = Θ( n -1 ) while LoRA requires η U = Θ(1) and η V = Θ( n -1 ) for optimal performance. These rates become impractical for modern LLMs with very large n . Therefore, S 2 FT aligns with LoRA variants that fix one matrix [52, 79], offering more stable and efficient optimization.

Furthermore, under a given sparsity level as regularization, our model simplifies optimization when approximating the full fine-tuning gradients at non-zero positions. Similar to LoRA-SB [53], let G V denote the gradient of V S 2 FT . The equivalent gradient ˜ G , which describes the virtual gradient of the pretrained weight matrices, can be expressed as U S 2 FT S G ⊤ V . Then, the gradient with respect to V S 2 FT can be expressed in terms of the gradient of the pretrained weight W pre as: G O V = U S 2 FT ⊤ S G . Using this relationship, our objective is to minimize the distance between the equivalent gradient and the full gradient as min G V ∥ ˜ G -G ∥ 2 F , where the optimal solution is given by G V = ( U S 2 FT ⊤ S U S 2 FT S ) -1 G O V . Since U S 2 FT S is orthogonal, we have G V = G O V . This shows that S 2 FT can keep the optimal update directions throughout the training process, establishing it as an efficient sparse optimization method.

5 Experiments

In this section, we conduct a series of experiments across three diverse benchmarks covering more than 20 datasets. Our goal is to provide a rich picture of how S 2 FT performs in different scenarios. Here, we compare our method with different fine-tuning strategies and categories including: (i) Full fine-tuning (FT), (ii) reparameterized fine-tuning : LoRA [27], DoRA [38], and Galore [80], (iii) adapter-based fine-tuning : Series Adapter [26], Parallel Adapter [24], and LoReFT [69], (iv) promptbased fine-tuning : Prefix-Tuning [36], (v) sparse fine-tuning : LISA [48]. For a fair comparison, we keep a comparable number of trainable parameters in S 2 FT to that of LoRA. The design choices for trainable parameter allocations in S 2 FT will be detailed in Section 5.4. All other hyperparameters are selected via cross-validation. Detailed setups and dataset descriptions are provided in Appendix E.

5.1 Commonsense Reasoning

The results of eight common sense reasoning tasks in Table 1 show that S 2 FT consistently outperforms existing PEFT methods in the LLaMA-7B / 13B , LLaMA2-7B and LLaMA3-8B models. Compared to LoRAand DoRA, it achieves average performance gains of 4.6% and 2.8%, respectively. Furthermore, S 2 FT also shows superior performance against recent approaches, including Galore, LoReFT, and LISA, with improvements of at least 1.0%. Remarkably, despite using less than 1% of trainable parameters, our method surpasses full FT by 0.5%. The 3.0% improvement on the LLaMA3-8B suggests that keeping most pre-trained parameters frozen enables better generalization to test distributions.

5.2 Arithmetic Reasoning

As showcased in Table 2, S 2 FT consistently outperforms other PEFT methods for different base models. On average, it achieves improvements of 1.3% and 0.9% over LoRA and DoRA, respectively. These results highlight the versatility and effectiveness of our approach across a diverse range of tasks. Additionally, we observe substantial improvements even when compared to Full FT for the LLaMA3-8B model, particularly on complex tasks such as GSM8K and AQuA. This suggests that S 2 FT better preserves the original reasoning capabilities of this stronger model while acquiring new skills from the fine-tuning data, thereby validating the enhanced generalization ability of our method.

5.3 Instruction Following

Table 3 comprehensively compares various methods on eight tasks in the MT-Bench dataset [82]. It is observed that S 2 FT > LISA > Full FT > LoRA/Galore ≥ Vanilla for both the Mistral-7B and LLama2-7B model. This is because sparse FT methods like S 2 FT and LISA retain more pre-trained knowledge while acquiring new skills on the FT dataset, thereby generalizing better to diverse tasks in the MT-Bench dataset. Moreover, our method outperforms LISA due to its fine-grained and flexible selection strategy, enabling all layers to learn to follow instructions on the full fine-tuning set.

5.4 Design Choices for Trainable Parameter Allocations

Finally, we detail how S 2 FT distribute trainable parameters across layers, modules, and channels.

Uniform across Layers : Following Chen et al. [10], we allocate parameters to each layer uniformly. Fine-tune Important Modules : Figure 4 analyzes the effectiveness of different components in a LLaMA-like Transformer Block for fine-tuning, including Query, Key, Value, Output, Up, Gate, and Down projections. To ensure a fair comparison, we maintain a fixed number of trainable parameters when fine-tuning each component. The results show that the effectiveness of components in finetuning follows the order: Query/Key ≪ Value/Up/Gate < Output/Down. This is because Query/Key are only used to measure token similarities, while others serve as persistent memories of training data. Based on this finding, we allocate our parameter budget fairly to the Output and Down projections. For the LLama3-8B and Mistral-7B models, we only fine-tune the Down projection due to the inflexible selection in multi-query attention. Further analysis of this setting is left for future research.

BoolQ

60

80

100

PIQA

SIQA

WinoGrande

ARC-e

ARC-c

QBQA

Query

Key

Value

Output

Up

Gate

Down

Selection across Channels : In Section 3.2, we discuss several strategies for channel selection. In our main experiments, we employ random selection to ensure fair comparisons with baseline methods, as these approaches treat all channels with equal importance. However, the sparse structure of S 2 FT offers controllability during fine-tuning, allowing us to prioritize important channels in the selection process to further boost performance. Table 4 compared nine different strategies, incorporating five varying selection metrics (i.e., random, weight, activation, weight-activation product, and gradient), each choosing either the largest or smallest values. For S 2 FT-A, S 2 FT-S, and S 2 FT-G, we employ 1% of the fine-tuning data as a calibration set, introducing only negligible overhead during inference.

Our results demonstrate that random selection serves as a strong baseline due to its unbiased nature. Among heuristic metrics, selecting channels with the smallest activations (i.e., S 2 FT-A and S 2 FT-S) outperforms random selection. This indicates that these channels contain less task-specific information, enabling us to inject new knowledge through fine-tuning while preserving pre-trained capabilities in other channels. In contrast, other strategies introduce bias that compromises model performance. Notably, the counterintuitive accuracy decrease in S 2 FT-G (Large) suggests that channels with large gradients contain task-related pre-trained knowledge, and modifying them will disrupt these abilities.

6 Analysis

Having demonstrated the strong generalization capability and overall performance of S 2 FT, we now further explore its training efficiency and serving scalability compared to other fine-tuning techniques.

6.1 Training Efficiency

To evaluate training efficiency, we examine two crucial metrics: peak memory footprint and average training latency. These numbers are measured on a single Nvidia A100 (80G) SXM GPU. We keep a comparable number of parameters for all methods. To obtain the average latency, we fine-tune the model for 50 runs, each run including 200 iterations, with 10 warmup runs excluded in measurement.

bs=1

bs=2

Memory (GB)

18

23

20

26

22

30

19

24

20

26

19

25

54

54

bs=1

bs=2

26

39

29

45

32

52

26

40

28

42

27

42

54

54

bs=1

bs=2

33

41

36

46

39

52

34

42

38

46

34

43

OOM

OOM

bs=1

bs=2

45

67

50

75

56

OOM

46

67

50

71

47

70

OOM

OOM

LLaMA2-7B, length 512

bs=1

bs=2

Latency (ms)

118

231

138

260

234

398

135

250

485

579

610

717

314

439

LLaMA2-7B, length 1024

bs=1

bs=2

258

476

288

529

418

742

279

504

606

822

733

1010

461

713

LLaMA2-13B, length 512

bs=1

bs=2

190

369

214

411

429

669

215

398

815

987

1144

1312

OOM

OOM

LLaMA2-13B, length 1024

bs=1

bs=2

411

779

453

865

702

OOM

437

814

1029

1391

1355

1763

OOM

OOM

S 2 FT(Ours)

LoRA

DoRA

LISA

LoReFT

Galore

Full FT

In Figure 5, we thoughtfully profile S 2 FT on various model sizes, sequence lengths, and batch sizes. Compared to Full FT, S 2 FT saves 1.4-3.0 × memory, and speedups fine-tuning by 1.5-2.7 times. When benchmarking against other PEFT methods, S 2 FT establishes new standards for efficiency, offering average reductions of 2% in memory usage and 9% in latency. Notably, S 2 FT outperforms the widely adopted LoRA, achieving about 10% improvement in both metrics by avoiding the need to store new parameters and perform additional calculations. Our partial back-propagation algorithm further improves efficiency by saving unnecessary forward activations and backward calculations.

6.2 Serving Scalability

While S 2 FT avoids additional inference overhead for a single fine-tuned model through in-place gradient updates, we will now discuss its scalability for serving thousands of fine-tuned models. To begin, we introduce the unmerged computation paradigm of S 2 FT: Given a pre-trained weight matrix W pre ∈ R d × k and its corresponding fine-tuned weight matrix W with sparsity level s , we define the weight difference as ∆ W = W -W pre . Similar to Section 4, ∆ W can be decomposed into the product of a weight matrix V ∈ R k × s and a permutation matrix U ∈ R d × s . This decomposition allows us to 'unmerge' an adapter ∆ W = UV ⊤ from W , thereby sharing similarities with other adapters during inference. Following Zhong et al. [83], we consider three different adapter composition scenarios:

Adapter Fusion. To combine knowledge from multiple trained adapters, we employ weighted fusion when fine-tuning is impractical due to limited data access or computational resources. However, this approach degrades performance. In Table 5, we compare the effectiveness of LoRA and S 2 FT when combining adapters trained separately on commonsense and arithmetic reasoning tasks, where we consider both fine-tuning overlapped and non-overlapped parameters for different adapters in S 2 FT. Our results show that S 2 FT with non-overlapped parameters achieves the best performance, while the overlapped variant shows inferior results. This is because S 2 FT (non-overlap) modifies orthogonal low-rank spaces for different tasks. Similarly, LoRA largely retains task-specific capabilities during adapter fusion by optimizing low-rank projection matrices to create separate spaces for each adapter.

1024

2048

4096

8192

16384

32768

Base Weight Dimension

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Latency (ms)

S

2

FT:

scatter_add

LoRA:

+

add

(a) Switch Time on GPU

1024

2048

4096

8192

16384

32768

Base Weight Dimension

0

20

40

60

80

100

120

Latency (ms)

S

2

FT:

scatter_add

LoRA:

add

(b) Switch Time on CPU

1

10

100

1000

10000

100000

Number of Adapters

0

5

10

15

20

25

30

Latency (ms)

S

2

FT(W

1

):

+

scatter

+

add

S

2

FT(W

2

):

+

gather

+

add

LoRA:

2×

+

add

(c) Parallelism Time on GPU

Adapter Switch. Another way to leveraging multiple adapters is to dynamically switch between them. This process involves four steps: unfusing the old adapter, unloading it from memory, loading the new adapter, and fusing it into the model. In such setting, LoRA needs two matrix multiplications ( matmul ) and two additions ( add ) on GPU whereas S 2 FT only requires two sparse addition ( scatter add ). In Figure 6a, we increase the base weight dimension while maintaining a sparsity of 32 for S 2 FT and a low-rankness of 16 for LoRA. Notably, we observe that LoRA's switching time scales quadratically, while S 2 FT remains nearly constant. Moreover, in I/O-constrained scenarios such as deployment on CPU, S 2 FT further accelerates adapter switch by only updating a small fraction of the original weights, reducing the volume of I/O transfers, as time compared between scatter add and add in Figure 6b.

Adapter Parallelism. To serve thousands of adapters in parallel, we decompose the computation into separate batched computations for W pre and ∆ W following S-LoRA [60]. While LoRA requires two matmul and one add on GPU, S 2 FT reduces this to a matmul , an add , and either a scatter or gather for W 1 and W 2 in Section 3.1. Figure 6c shows that S 2 FT achieves up to 22% faster inference than LoRA under the same memory constraints, with more speedup as the number of adapters scales.

7 Related Work

PEFT methods reduce the fine-tuning cost for large models, which can be categorized into 4 groups:

Adapter-based Fine-tuning introduces additional trainable module into the original model. Series Adapters insert components between MHA or FFN layers [51, 26], while parallel adapters add modules alongside existing components [24]. Recently, ReFT [69] was introduced to directly learn interventions on hidden representations. However, they introduce additional latency during inference.

Prompt-based Fine-tuning adds randomly-initialized soft tokens to the input (usually as a prefix) and train their embeddings while freezing the model weights [36, 40, 35]. These approaches result in poor performance compared to other groups, while come at the cost of significant inference overhead.

Reparameterized Fine-tuning utilizes low-rank projections to reduce trainable parameters while allowing operations with high-dimensional matrices. LoRA[27] and its recent variants like DoRA[38], AsyLoRA [84], and FLoRA [61], use low-rank matrices to approximate additive weight updates during training. To alleviate the limitations of low-rank structure, other work also add or multiply orthogonal matrices to enable high-rank updating, including MoRA [29], OFT [54], and BOFT [39]. These methods require no additional inference cost as the weight updates can be merged into models.

Sparse Fine-tuning aims to reduce the number of fine-tuned parameters by selecting a subset of pre-trained parameters that are critical to downstream tasks while discarding unimportant ones. This kind of methods are commonly used in the pre-LLM era [20, 75, 64]. However, they cannot reduce the memory footprints due to their unstructured nature. Recent approaches address this limitation through three directions: (1) developing structured variants that sacrifice selection flexibility for better hardware efficiency [48, 85], (2) incorporating sparsity into LoRA [68, 15, 41] but yield limited efficiency gains, or (3) using sparse operators for lower memory cost but slow down training [4, 49, 7].

Our work is based on the last category but achieving better performance and efficiency simultaneously. Additionally, we focus on scalable inference of PEFT methods, with S 2 FT being the only approach that enables effective fusion, rapid switching, and efficient parallelism when serving multiple adapters.

8 Conclusion

This paper introduces S 2 FT, a novel PEFT family that simultaneously achieves high quality, efficient training, and scalable serving for LLM fine-tuning. S 2 FT accomplishes this by selecting sparsely and compute densely. It selects a subset of heads and channels to be trainable for the MHA and FFN modules, respectively. The weight matrices from the two sides of the coupled structures in LLMs are co-permuted to connect the selected components into dense matrices, and only these parameters are updated using dense operations. We hope S 2 FT can be considered as a successor to LoRA for PEFT.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. 1

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. 17

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vuli´ c. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.07560 , 2021. 2

Alan Ansell, Ivan Vuli´ c, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405 , 2024. 10

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019. 5, 18

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631 , 2023. 1

Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Rafael Esteves, Shreya Kadambi, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, et al. Rapid switching and multi-adapter fusion via sparse high rank adapters. arXiv preprint arXiv:2407.16712 , 2024. 10

Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673 , 2024. 1

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020. 16, 17

Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces. arXiv preprint arXiv:2301.01821 , 2023. 8

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends® in Machine Learning , 14(5):566-806, 2021. 31

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 , 2019. 16, 17

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018. 16, 17

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. 16, 17

Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11696 , 2023. 10

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. 1

Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16091-16101, 2023. 4

Jianwei Feng and Dong Huang. Optimal gradient checkpoint search for arbitrary computation graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1143311442, 2021. 4

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems , 30, 2017. 5, 18

Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463 , 2020. 10

Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 , 2024. 1

Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint arXiv:1611.04231 , 2016. 5, 18

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354 , 2024. 3, 5, 6

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 , 2021. 6, 7, 10

Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 523-533, 2014. 16, 17

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International conference on machine learning , pages 2790-2799. PMLR, 2019. 6, 7, 10

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. 2, 6, 7, 10

Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 , 2023. 3, 16, 17

Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130 , 2024. 10

Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems , 29, 2016. 5, 18

Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics , 3:585-597, 2015. 16, 17

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies , pages 1152-1157, 2016. 16, 17

Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, and Yunxin Liu. Lora-switch: Boosting the efficiency of dynamic llm adapters via system-algorithm co-design. arXiv preprint arXiv:2405.17741 , 2024. 1

Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning , pages 2902-2907. PMLR, 2018. 5, 18

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021. 10

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021. 6, 7, 10

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 , 2017. 16, 17

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, KwangTing Cheng, and Min-Hung Chen. DoRA: Weight-Decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 , 2024. 2, 6, 7, 10

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243 , 2023. 10

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. AI Open , 2023. 10

Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403.16187 , 2024. 10

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning , pages 22137-22176. PMLR, 2023. 2

Haihao Lu and Kenji Kawaguchi. Depth creates no bad local minima. arXiv preprint arXiv:1702.08580 , 2017. 5, 18

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747 , 2023. 1

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems , 36:21702-21720, 2023. 2, 4

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018. 16, 17

Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. arXiv preprint arXiv:2302.06232 , 2023. 32

Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. LISA: Layerwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403.17919 , 2024. 2, 6, 7, 10

Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in llms. arXiv preprint arXiv:2406.16797 , 2024. 10

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2080-2094, Online, June 2021. Association for Computational Linguistics. 16, 17

Jonas Pfeiffer, Ivan Vuli´ c, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052 , 2020. 10

Lai-Man Po, Yuyang Liu, Haoxuan Wu, Tianqi Zhang, Wing-Yin Yu, Zhuohan Wang, Zeyu Jiang, and Kun Li. Sbora: Low-rank adaptation with regional weight updates. arXiv preprint arXiv:2407.05413 , 2024. 5, 6

Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint arXiv:2411.19557 , 2024. 5, 6

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Sch¨ olkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems , 36:79320-79362, 2023. 10

Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413 , 2016. 16, 17

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J´ er´ emy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023. 1

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021. 16, 17

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728 , 2019. 16, 17

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013. 5, 18

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 , 2023. 1, 2, 10

Chongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xiaokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405.14739 , 2024. 10

GW Stewart. On the continuity of the generalized inverse. SIAM Journal on Applied Mathematics , 17(1):33-45, 1969. 33

Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems , 34:24193-24205, 2021. 2

Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems , 34:24193-24205, 2021. 10

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model. https://github. com/tatsu-lab/stanford alpaca , 2023. 17

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. 1

J Leo van Hemmen and Tsuneya Ando. An inequality for trace ideals. Communications in Mathematical Physics , 76:143-148, 1980. 33

Haoyu Wang, Tianci Liu, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. arXiv preprint arXiv:2406.10777 , 2024. 10

Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. ReFT: Representation finetuning for language models. arXiv preprint arXiv:2404.03592 , 2024. 6, 7, 10

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient finetuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148 , 2023. 1

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXiv:2109.05687 , 2021. 2

Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems , 34:17084-17097, 2021. 6

Yi Yu, Tengyao Wang, and Richard J Samworth. A useful variant of the davis-kahan theorem for statisticians. Biometrika , 102(2):315-323, 2015. 31

Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge. arXiv preprint arXiv:2303.14070 , 2023. 1

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021. 10

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019. 16, 17

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021. 5, 18

Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems , 36:12589-12610, 2023. 1

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303 , 2023. 5, 6

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507 , 2024. 2, 6, 7, 16

Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.00732 , 2024. 1

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024. 7, 17

Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402.16843 , 2024. 9

Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz S´ aez de Oc´ ariz Borde, Rickard Br¨ uel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. arXiv preprint arXiv:2402.16842 , 2024. 10

Ligeng Zhu, Lanxiang Hu, Ji Lin, and Song Han. Lift: Efficient layer-wise fine-tuning for large model models. arXiv preprint arXiv:2410.11772 , 2023. 10

A Limitations

While our work demonstrates the effectiveness of S 2 FT for LLM fine-tuning, several promising directions remain unexplored. First, extending S 2 FT to other architectures with coupled structures, such as CNNs and RNNs, can broaden its applicability. Second, verifying our approach beyond language tasks, particularly in large vision/multi-modal models, will enhance its versatility. Third, exploring more selection strategies can provide deeper insights into optimal fine-tuning protocols due to the controllability in S 2 FT. Fourth, scaling our method to larger models requires further experiments. Finally, although our work confirms the feasibility of scalable and efficient deployment during inference, developing a practical serving system for S 2 FT remains an important next step.

B Broader Impacts

Since our work focuses on PEFT, it leads to a reduction in hardware resource and energy consumption. Given the growing adoption of LLMs across diverse domains and the corresponding surge in finetuning demands, S 2 FT should represent an important step toward more sustainable AI development.

C Detailed Experimental Setups for Section 2

In this study, we used SpFT, LoRA, and Full FT to fine-tune the LLaMA-3-8B model on the Math10K dataset [28]. The Math10K dataset combines training sets from GSM8K [14], MAWPS [32], and AQuA [37], augmented with chain-of-thought steps generated by language models. We conducted training for 3 epochs with a batch size of 64. For both PEFT methods-SpFT and LoRA-we fine-tune with three ratios of trainable parameters for all linear layers: p = 10% , 1% , 0 . 1% . The model's performance is evaluated on both arithmetic and commonsense reasoning tasks, representing near outof-distribution (OOD) and far OOD generalization scenarios, respectively. The arithmetic reasoning dataset comprises seven subtasks: MultiArith [55], GSM8K, AddSub [25], AQuA, SingleEq [31], SVAMP[50], and MAWPS. The commonsense reasoning dataset includes eight subtasks: BoolQ [12], PIQA [9], SocialQA [58], HellaSwag [76], WinoGrande [57], ARC-challenge [13], ARC-easy [13], and OpenbookQA [46]. Based on task complexity within arithmetic reasoning (accuracy ≥ 90%), we group MultiArith, AddSub, SingleEq, and MAWPS as easy subtasks, while the remaining ones are classified as hard subtasks. This stratification enables us to evaluate whether the model develops advanced reasoning abilities beyond memorizing basic arithmetic operations from the training data.

D Detailed Selection Strategies in Section 3

For the five selection strategies described in Section 3.2, we will detail the methods for identifying and selecting important subsets within each linear layer of both MHA and FFN modules in LLMs.

S 2 FT-R (S 2 FT) : In this strategy, we will randomly select some heads for the MHA modules and select a few channels for the FFN modules. For the output projection, all channels in the selected heads will be included to enable dense-only computation. In the up and gate projections, we will select a subset of columns, while for the down projection, a few trainable rows will be chosen.

S 2 FT-W : This variant selects subsets based on the weight magnitudes (i.e., ∥ W ∥ 2 ) in the MHA and FFN modules. We will test subsets corresponding to both the largest and smallest weights.

S 2 FT-A : This variant selects subsets based on the magnitude of activations (i.e., ∥ A ∥ 2 ) on a calibration set, using 1% of the fine-tuning data. Since collecting activations requires only forward passes, this approach maintains the same memory footprint as inference and incurs a negligible increase in training time. Similarly, we evaluate both the largest and smallest activation variants.

S 2 FT-S : The Top-K subsets are ranked and selected by the product of the weight and activation magnitudes (i.e, ∥ W ∥ 2 · ∥ A ∥ 2 ). The activation values are collected in a manner similar to S 2 FT-A.

S 2 FT-G : This variant selects subsets based on the magnitude of gradients on the calibration set. Since gradients are collected without updating the model, we calculate and discard gradients layer by layer during back-propagation similar to Galore [80], requiring minimal additional memory.

E Detailed Experimental Setups for Section 5

Detailed selection strategies and number of trainable parameters are presented in Section 5.

E.1 Dataset Description

Commonsense Reasoning. The commonsense reasoning dataset comprise eight subsets: BoolQ [12], PIQA [9], SocialQA [58], HellaSwag [76], WinoGrande [57], ARC-challenge [13], ARC-easy [13], and OpenbookQA [46]. Following the experimental setup of LLM-Adapters [28], we split each dataset into training and test sets. Subsequently, we combine the training data from all eight tasks into a single fine-tuning dataset and evaluate performance on the individual test dataset for each task.

Arithmetic Reasoning. Wefollowed Hu et al. [28] and evaluated S 2 FT on seven math reasoning tasks, including MultiArith [55], GSM8K [14], AddSub [25], AQuA [37], SingleEq [31], SVAMP [50] and MAWPS [32]. Our fine-tuning employed the Math10K dataset [28], which combines training sets from GSM8K, MAWPS, and AQuA, augmented with LM-generated chain-of-thought steps. Therefore, these three tasks are considered ID, while the remaining four are classified as OOD tasks.

Instruction Following. To further showcase S 2 FT's superior generalization ability, we employ the instruction-following fine-tuning task with Alpaca GPT-4 dataset, which comprises 52k samples generated by GPT-4 [2] based on inputs from Alpaca [65]. Performance is measured on MT-Bench [82], featuring 80 high-quality, multi-turn questions designed to assess LLMs on eight different aspects.

E.2 Hyperparameter Description

Additional hyperparameter configurations for all tasks are provided in Table 6. We maintain the same hyperparameter settings across the LLaMA-7/13B , LLaMA2-7B , LLaMA3-8B , and Mistral-7B models.

F Proofs for Theoretical Results in Section 4

Here we provide proofs for the results in Section 4.

F.1 Notation

For a vector a , let ∥ a ∥ be the ℓ 2 norm of a . For d 1 ≥ d 2 , denote a set of orthogonal matrices by O d 1 ,d 2 := { R ∈ R d 1 × d 2 : R ⊤ R = I d 2 } . For a matrix A ∈ R d 1 × d 2 , let ∥ A ∥ F and ∥ A ∥ op be the Frobenius norm and spectral norm of A , respectively. Denote the condition number of A by κ ∗ ( A ) := ∥ A ∥ op /λ ∗ ( A ) . Let A † be Moore-Penrose inverse of A . For a symmetric matrix A , denote its effective rank by r e ( A ) := tr( A ) / ∥ A ∥ op. Note that r e ( A ) ≤ rank( A ) always holds. For a, b ∈ R , we let a ∨ b := max( a, b ) and a ∧ b := min( a, b ) . For a matrix A ∈ R d 1 × d 2 , let SVD r ( A ) := Φ r ( A )Λ r ( A )Ψ ⊤ r ( A ) be the topr singular value decomposition of A , where Φ r ( A ) ∈ O d 1 ,r and Ψ r ( A ) ∈ O d 2 ,r are topr left and right singular vectors of A , respectively, and Λ r ( A ) = diag( λ 1 ( A ) , . . . , λ r ( A )) ∈ R r × r is a diagonal matrix of singular values of A , where λ j ( A ) denotes the j -th largest singular value of A . Define Φ ∗ ( A ) := Φ rank( A ) ( A ) and Ψ ∗ ( A ) := Ψ rank( A ) ( A ) as the left and right singular vectors of A corresponding to non-zero singular values, respectively. Define the smallest positive singular value of A as λ ∗ ( A ) = λ rank( A ) ( A ) and let Λ ∗ ( A ) = Λ rank( A ) ( A ) . For a deep learning model fine-tuned on n i.i.d. samples ( x ( i ) i , y ( i ) i ) ⊂ R p × R q , we say an event F occurs with high probability when P ( F ) = 1 -exp ( -Ω(log 2 ( n + p + q )) ) .

F.2 Setup

We consider multivariate regression task. Using n i.i.d. samples ( x ( i ) i , y ( i ) i ) ⊂ R p × R q from in-distribution task, we fine-tune a pre-trained network f pre : R p → R q for better prediction.

Deep Linear Networks We consider deep linear networks of the form x ↦→ W L W L -1 . . . W 1 x : R d → R p , where W ℓ ∈ R d ℓ × d ℓ -1 , with d L = q and d 0 = p . In comparison to multi-head attention transformers, each row of W ℓ can be viewed as corresponding to the parameters in a single head. Let f pre ( x ) = W pre L W pre L -1 . . . W pre 1 x : R p → R q represent a pre-trained neural network. We denote W pre ℓ := W pre L W pre L -1 . . . W pre ℓ ∈ R d L × d ℓ -1 as the weights up to the ℓ -th layer, and W pre ℓ := W pre ℓ W pre ℓ -1 . . . W pre 1 ∈ R d ℓ × d 0 as the weights above the ℓ -th layer, with the promise that W pre 0 = I . Deep linear networks have been widely used to facilitate the theoretical analysis of modern complex deep neural networks [59, 30, 43, 22, 34, 5].

Fine-Tuning We employ ℓ 2 distance as the error metric. Given a pre-trained network f pre , we finetune its ℓ -th layer by minimizing the empirical in-distribution risk R ( i ) n ( f ) := (1 /n ) ∑ i ∈ [ n ] ∥ y ( i ) i - f ( x ( i ) i ) ∥ 2 , where ( x ( i ) i , y ( i ) i ) ⊂ R p × R q are n i.i.d. observations from in-distribution task. More specifically, we consider a class of rankd adaptation defined as

where U ∈ R d ℓ × d and V ∈ R d ℓ -1 × d are parameters to fine-tune. Note that by regarding multiple consecutive layers as a single layer, our settings can be extended to multi-layer fine-tuning.

We specifically compare two fine-tuning methods: LoRA and S 2 FT.

LoRA. For a fixed ℓ ∈ [ L ] , and low-rankness level 1 ≤ r ≤ min { d ℓ , d ℓ -1 } , we train the lowrank matrices ( U, V ) in (4) by minimizing the empirical in-distribution risk via gradient descent. Motivated from the previous results that gradient descent has implicit regularization [77, 19, 5], we directly consider the minimum norm solutions:

S 2 FT. For a fixed ℓ ∈ [ L ] , and a sparsity level s = ⌊ r · d ℓ + d ℓ -1 d ℓ -1 ⌋ , we train only V in (4) with the fixed choice of U ← U S 2 FT S := [ e a 1 ; e a 2 ; . . . ; e a s ] , which specifies s channels to fine-tune, where S = { a 1 , a 2 , . . . , a s } ⊂ [ d ℓ ] . Here e a is the standard basis vector with the a -th entry being 1 . We minimize the empirical in-distribution risk via gradient descent. Similar to LoRA, we consider the following minimum norm solution:

Data Generating Process As a simplification of the data generating process, we consider multiple linear regression. Assume that the in-distribution data ( x ( i ) , y ( i ) ) ∈ R p + q and out-of-distribution data ( x ( o ) , y ( o ) ) ∈ R p + q are generated according to

where B ( k ) ∈ R q × p , and ϵ ( k ) ∈ R q is the error term satisfying E [ ϵ ( k ) | x ( k ) ] = 0 . Assume that Σ ( k ) ϵ := E [ ϵ ( k ) ϵ ( k ) ⊤ ] ∈ R q × q exists and E [ x ( k ) ] = 0 . The signal covariance matrix is denoted by Σ ( k ) x := E [ x ( k ) x ( k ) ⊤ ] ∈ R p × p .

We define the in-distribution and out-of-distribution risks of f : R p → R q as:

For notational brevity, we can write W pre = W pre L ∈ R q × p . Let X ( i ) := ( x ( i ) 1 , . . . , x ( i ) n ) ∈ R p × n , Y ( i ) := ( y ( i ) 1 , . . . , y ( i ) n ) ∈ R q × n , and E ( i ) = ( ϵ ( i ) 1 , . . . , ϵ ( i ) n ) := Y ( i ) -B ( i ) X ( i ) ∈ R q × n . Denote the

in-distribution sample covariance matrices by ˆ Σ ( i ) x := (1 /n ) X ( i ) X ( i ) ⊤ , ˆ Σ ( i ) ϵ := (1 /n ) E ( i ) E ( i ) ⊤ , ˆ Σ ( i ) x,ϵ := (1 /n ) X ( i ) E ( i ) ⊤ , ˆ Σ ( i ) ϵ,x = ˆ Σ ( i ) ⊤ x,ϵ . Define ˇ Σ ( k ) x,ϵ = ( X ( i ) ⊤ ) † E ( i ) ⊤ , ˆ A := ( W pre ℓ -1 ˆ Σ ( i ) x W pre ⊤ ℓ -1 ) 1 / 2 , A := ( W pre ℓ -1 Σ ( i ) x W pre ⊤ ℓ -1 ) 1 / 2 , Φ ′ := Φ ∗ ( W pre ℓ +1 ) , Φ ′′ S := Φ ∗ ( W pre ℓ +1 U S 2 FT S ) , D = B ( i ) -W pre , ˆ D := B ( i ) -W pre + ˇ Σ ( i ) ϵ,x . Also define M := Φ ′⊤ D Σ ( i ) x W pre ⊤ ℓ -1 A † and ˆ M := Φ ′⊤ ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ˆ A † . Let ˆ Ψ ′ := Ψ ∗ ( ˆ A ) , and G ( i , o ) ℓ := ( W pre ℓ Σ ( i )1 / 2 x ) † W pre ℓ Σ ( o )1 / 2 x be a matrix that captures the covariate shift at the ℓ -th layer.

We consider fine-tuning the ℓ -th ( ℓ ∈ [ L ] ) layer of the pre-trained deep linear network f pre ( x ) = W pre L W pre L -1 . . . W pre 1 x using in-distribution observations ( x ( i ) i , y ( i ) i ) i ∈ [ n ] .

To measure the performance of models, we define the excess risks of f for the task k ∈ { i , o } as

where the infimum is taken over all square integrable functions.

F.3 Assumptions

We assume that W pre ℓ -1 Σ ( i ) x W pre ⊤ ℓ -1 = 0 , since otherwise W pre ℓ -1 x ( i ) = 0 almost surely and fine-tuning the ℓ -th layer does not improve the performance of the pre-trained model. Define the in-distribution prediction residuals for the pre-trained model f pre by Σ ( i ) f := E [( B ( i ) x ( i ) -W pre x ( i ) )( B ( i ) x ( i ) -W pre x ( i ) ) ⊤ ] . Note that E ( i ) ( f pre ) = tr ( Σ ( i ) f ) . We also assume that ∥ Σ ( i ) f ∥ op > 0 , since otherwise E ( i ) ( f pre ) = ∥ Σ ( i ) f ∥ 2 F = 0 and there is no room for improvement from the pre-trained model.

̸

Next, we introduce several assumptions.

Assumption F.1 (Sub-Gaussianity) . Assume that there exist some constants c 1 , c 2 ∈ (0 , ∞ ) such that ( x ( i ) , ϵ ( i ) ) in the model 7 satisfies

for any γ ∈ R p and γ ′ ∈ R q , where ∥ y ∥ ψ 2 is the sub-Gaussian norm defined as

for a random variable y taking values in R .

Assumption F.2 (Sufficiently Many Observations) . Assume that

and

Assumption F.3 (Eigengap Condition) . Assume that there exists some constant C g > 0 such that

holds.

Assumption F.3 is necessary to identify the rankr approximation of M , which is used to derive the risk of LoRA.

Assumption F.4 (Approximate Sparsity of Channels) . Assume that there exists some S 0 ⊂ [ d ℓ ] with | S 0 | ≤ s and δ > 0 such that

holds.

Assumption F.5 (Distribution Shift) . Assume that Σ ( i ) x = Σ ( o ) x = Σ x for some Σ x ∈ R d × d and that ∥ Φ ⊤ ∗ ( W pre ℓ +1 U S 2 FT S )( B ( o ) -B ( i ) )Σ 1 / 2 x ∥ 2 F ≤ ε 2 E ( o ) ( f pre ) for some ε > 0 .

Assumption F.6 (Condition Number) . Assume that κ ∗ ( M ) ≲ 1 , κ ∗ ( W pre ℓ +1 ) ≲ 1 , κ ∗ (Σ ( i ) f ) ≲ 1 and κ ∗ ( W pre ℓ -1 Σ ( i ) x W pre ⊤ ℓ -1 ) ≲ 1 .

Note that Assumption F.6 is not essential to our analysis.

F.4 Main Results

We first demonstrate that LoRA and S 2 FT exhibit comparable memorization abilities. Next, we present a formal restatement of 4.2 that combine Theorems F.10, F.11, F.13, F.15, and Lemma F.14.

Theorem F.7. Suppose that Assumptions F.1, F.2, F .3, F .4, and F .6 hold. Choose S such that S ⊃ S 0 holds. Let U LoRA , V LoRA be the LoRA adaptation matrices defined in (5) . Let V S 2 FT be the S 2 FT adaptation matrices given U S 2 FT S defined in (6) . Then, for all sufficiently large n , the following holds with probability 1 -exp ( -Ω(log 2 ( n + p + q )) ) : for any η > 0 ,

where

Theorem F.8 (Restatement of Theorem 4.2) . Consider the limit n →∞ . Suppose that Assumption F.5 holds. Let U LoRA , V LoRA be the LoRA adaptation matrices defined in (15) . Let V S 2 FT be the S 2 FT adaptation matrices given U S 2 FT S defined in (25) . If B ( i ) = W pre ℓ +1 ˜ BW pre ℓ -1 holds for some ˜ B ( i ) ∈ R d ℓ × d ℓ -1 , and s, r ≤ rank(Σ ( i ) f ) , then,

Intuition of the proof of Theorem F .8. LoRA forgets pre-trained tasks due to its model complexity. Consider the simplest low-rank adaptation to a single-layer linear network:

Assume that Σ ( i ) x = I , then we can show that the solution is ∆ 1 = SVD r ( B ( i ) -W pre 1 ) . Under the condition that the rank of B ( i ) -W pre 1 is smaller than, or comparable to r , LoRA fine-tuned model can learn the in-distribution best regressor in ℓ 2 sense, since ( W pre 1 +∆ 1 ) x ≈ B ( i ) x = E [ y ( i ) | x ( i ) = x ] . Hence it makes LoRA fine-tuned model vulunerable to distribution shift.

On the other hand, we model S 2 FT as fine-tuning only a few channels:

Although S 2 FT is a special case of LoRA, the constraint on the direction of low-rank matrix prevents overfitting to the in-distribution task. To see this, note that a sparse fine-tuned model can be written as

where S ⊂ [ d 1 ] is a set of channels with cardinality s . Since S 2 FT keeps most of parameters from the pre-trained model, except for rows specified by S , the model forget less pre-training tasks.

F.5 Proofs for LoRA

F.5.1 Excess Risk of LoRA

Lemma F.9 (Excess Risk) . Consider the minimum norm solution

Then, the low-rank adaptation matrix satisfies

and

for k ∈ { i , o } .

Proof of Lemma F.9. The empirical risk of f ℓ,U,V for the in-distribution task can be written as

Since

where we used ( I -ˆ A ˆ A † ) W pre ℓ -1 ˆ Σ ( i )1 / 2 x = 0 . From (9), minimizing R ( i ) n ( f ℓ,U,V ) is equivalent to minimizing the norm:

This is minimized by ( U ′ , V ′ ) satisfying

where A 1 , A 2 ∈ R d ℓ × d ℓ -1 are arbitrary matrices. Since we particularly consider the minimum norm solution, we must have A 1 = 0 and A 2 = 0 . Hence

W pre ℓ +1 U LoRA V LoRA ⊤ W pre ℓ -1 = SVD r ( W pre ℓ +1 ( W pre ℓ +1 ) † ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ˆ A † ) ˆ A † W pre ℓ -1 . Therefore, the excess risk for k ∈ { i , o } becomes E ( k ) ( f ℓ,U LoRA ,V LoRA ) = E [ ( B ( k ) x ( k ) -W pre ℓ +1 ( W pre ℓ + U LoRA V LoRA ⊤ ) W pre ℓ -1 x ( k ) ) 2 ] = tr ( ( B ( k ) -W pre -SVD r ( W pre ℓ +1 ( W pre ℓ +1 ) † ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ˆ A † ) ˆ A † W pre ℓ -1 ) Σ ( k ) x · ( B ( k ) -W pre -SVD r ( W pre ℓ +1 ( W pre ℓ +1 ) † ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ˆ A † ) ˆ A † W pre ℓ -1 ) ⊤ ) .

This concludes the proof.

F.5.2 In-distribution Excess Risk of LoRA

Let E ( i ) ( f full ℓ ) denote the excess risk of f pre after fine-tuning all the parameters of the ℓ -th layer under population in-distribution risk.

Theorem F.10 (Restatement of Theorem F.7: LoRA Part) . Suppose that Assumptions F.1, F.2 and F.3 hold. Then, the following holds with probability 1 -exp ( -Ω(log 2 ( n + p + q )) ) . For any η > 0 ,

where

Note that the first term on the right hand side of (11) depends on the rank of residual matrix Σ ( i ) f = D Σ ( i ) x D ⊤ . It becomes zero when rank(Σ ( i ) f ) ≤ r and small when r/ rank(Σ ( i ) f ) ≈ 1 .

Proof of Theorem F.10. Let W LoRA ℓ := W pre ℓ +1 U LoRA V LoRA ⊤ . From Lemma F.9, we have

where we used ( I -AA † ) W pre ℓ -1 Σ ( i )1 / 2 x = 0 . From Lemma F.9

This gives

We bound T LoRA variance and T LoRA bias separately.

For the term T LoRA variance , since A † W pre ℓ -1 Σ ( i ) x W pre ⊤ ℓ -1 A † = A † A 2 A † ,

Therefore,

We first bound T LoRA variance , 1 . From Lemma G.1 and Assumption F.3, we have

where ˆ M = W pre ℓ +1 ( W pre ℓ +1 ) † ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ˆ A † and M = W pre ℓ +1 ( W pre ℓ +1 ) † D Σ ( i ) x W pre ⊤ ℓ -1 A † . From Lemma G.3,

Next we bound T LoRA variance , 2 . Again from Lemma G.3,

holds on the event F . Therefore,

hold with high probability.

Bound T LoRA bias . Note that

where the second equality follows from the fact that cross terms are zero, i.e., tr ( T 1 T ⊤ 2 ) = tr ( T 2 T ⊤ 3 ) = tr ( T 3 T ⊤ 1 ) = 0 since Ψ ∗ ( W pre ℓ -1 Σ ( i )1 / 2 x )Ψ ⊤ ∗ ( W pre ℓ -1 Σ ( i )1 / 2 x ) = Σ ( i )1 / 2 x W pre ⊤ ℓ -1 ( A 2 ) † W pre ℓ -1 Σ ( i )1 / 2 x and

hold. Thus from Lemma F.17,

where the last inequality follows since

Summary Note that for any η > 0 , ( T LoRA variance + T LoRA bias ) 2 ≤ (1 + η )( T LoRA bias ) 2 +(1+1 /η )( T LoRA variance ) 2 holds. Therefore,

Combined with (12), (13), and (14), this concludes the proof.

F.5.3 Out-of-distribution Excess Risk of LoRA

We define the low-rank matrix obtained by LoRA under population in-distribution risk as

Theorem F.11 (Restatement of Theorem F.8: LoRA Part) . For ( U LoRA ∞ , V LoRA ∞ ) , defined in (15)

Furthermore, for any η ∈ (0 , 1) ,

Proof of Theorem F.11. With a slight modification to the proof of Lemma F.9, it follows that

Recall that M := Φ ′ Φ ′⊤ D Σ ( i ) x W pre ⊤ ℓ -1 A † . Then,

where we used Φ ′ Φ ′⊤ W pre = W pre . From (14), we have

where we used ∥ A † W pre ℓ -1 Σ ( o )1 / 2 x ∥ op = ∥ G ( i , o ) ℓ -1 ∥ op. This gives the first claim.

Using 2 tr ( AB ⊤ ) ≥ -η ∥ A ∥ 2 F -(1 /η ) ∥ B ∥ 2 F for any η > 0 and any matrices A,B of the same shape, (17) can be rewritten as

Choose η ∈ (0 , 1) . By a similar argument as above, and using Φ ′ Φ ′⊤ W pre = W pre , we can show that ∥ T 1 + T 2 + T 3 ∥ 2 F ≤ 3 ∥ T 1 ∥ 2 F +3 ∥ T 2 ∥ 2 F +3 ∥ T 3 ∥ 2 F

where we used (14) again. This concludes the proof.

F.6 Proofs for Structured Sparse Fine-tuning

F.6.1 Excess Risk of Structured Sparse Fine-tuning

Lemma F.12 (Excess Risk) . Given S ⊂ [ d ℓ ] , consider the minimum norm solution

Then, the structured sparse adaptation matrix satisfies

and

for k ∈ { i , o } .

Thus minimizing R ( i ) n ( f ℓ,U S 2 FT S ,V ) is equivalent to minimizing the norm

Using the same argument as in the proof of Lemma F.9, the minimum norm solution V S 2 FT is obtained by

The excess risk for k ∈ { i , o } becomes

This concludes the proof.

F.6.2 In-distribution Excess Risk of Structured Sparse Fine-tuning

Theorem F.13 (Restatement of Theorem F.7: S 2 FT Part) . Suppose that Assumptions F.1 and F.2 hold. Fix S ⊂ [ d ℓ ] with | S | = s . Then, the following holds with probability 1 -exp ( -Ω(log 2 ( n + p + q )) ) . For any η > 0 ,

where

Note that the term ∥ (Φ ′ Φ ′⊤ -Φ ′′ S Φ ′′⊤ S )Φ ∗ ( D Σ ( i )1 / 2 x ) ∥ op in (21) measures the distance between subspaces spanned by Φ ′ and Φ ′′ S in a label space, weighted by Φ ∗ (Σ ( i ) f ) . In high level, this quantity shows the closeness between the ℓ -th layer full fine-tuning and S 2 FT. It takes small values when the important channels for residual prediction are sparsely distributed among all channels. This aligns with the intuition that S 2 FT only selectively fine-tunes small number of coordinates, and thus relying on the information contained in those coordinates.

Proof of Theorem F.13. Using the same argument as in the proof of Theorem F.10 combined with Lemma F.12, we have

and

We bound T S 2 FT variance and T S 2 FT bias separately.

Bound T S 2 FT variance . Note that

For the term T S 2 FT variance , 1 , using Lemma G.3,

holds on the event F , where the first inequality follows since the term inside the norm is at most rank2 s . Again from Lemma G.3,

holds on the event F . Therefore,

Bound T S 2 FT bias . By the same argument as in the proof of Theorem F.10,

where we used ∥ D Σ ( i ) x W pre ⊤ ℓ -1 A † ∥ 2 F ≤ ∥ D Σ ( i )1 / 2 x ∥ 2 F = E ( i ) ( f pre ℓ ) . We hypothesize that T S 2 FT bias ≃ T LoRA bias by comparing (13) and (23), Here, SVD s (Φ ′ Φ ′⊤ D Σ ( i ) x W pre ⊤ ℓ -1 A † ) is the best ranks approximation of Φ ′ Φ ′⊤ D Σ ( i ) x W pre ⊤ ℓ -1 A † and Φ ′′ S Φ ′′⊤ S D Σ ( i ) x W pre ⊤ ℓ -1 A † benefits from a rankr approximation, where r > s .

Summary Note that for any η > 0 , ( T S 2 FT variance + T S 2 FT bias ) 2 ≤ (1 + η )( T S 2 FT bias ) 2 +(1+1 /η )( T S 2 FT variance ) 2 holds. Thus

Combined with (22) and (24), this concludes the proof.

Next we characterize the bias terms T LoRA bias and T S 2 FT bias under sparsity assumption.

Lemma F.14. Suppose that Assumption F.4 holds. Then, for a sparse fine-tuned network with the choice S ⊃ S 0 , it follows that

Proof. Note that Φ ′′ S Φ ′′⊤ S is a projection into a subspace, which is contained in a subspace projected by Φ ′ Φ ′⊤ . Thus

where the last equality follows since (Φ ′′ S Φ ′′⊤ S -I ) W pre ℓ +1 U S 2 FT S = 0 by definition of Φ ′′ S = Φ ∗ ( W pre ℓ +1 U S 2 FT S ) . Thus

where the second inequality follows from ∥ Σ ( i )1 / 2 x W pre ⊤ ℓ -1 A † ∥ op ≤ 1 , Assumption F.4 and S ⊃ S 0 . The conclusion follows from (13) and (23).

F.6.3 Out-of-distribution Excess Risk of Structured Sparse Fine-tuning

Given S ⊂ [ d ℓ ] with | S | = s , we define the structured sparse adaptation matrix obtained by S 2 FT under population in-distribution risk as

Theorem F.15 (Restatement of Theorem F.8: S 2 FT Part) . Fix S ⊂ [ d ℓ ] with | S | = s . For V S 2 FT ∞ defined in (25) ,

Remark F.16 . If there is no covariate shift, i.e., Σ ( i ) x = Σ ( o ) x = Σ x for some Σ x , Theorem F.15 further gives the bound

Proof of Theorem F.15. With a slight modification to Lemma F.12, we obtain

where we used Σ ( i )1 / 2 x W pre ⊤ ℓ -1 ( A † ) 2 W pre ℓ -1 Σ ( o )1 / 2 x = ( W pre ℓ -1 Σ ( i )1 / 2 x ) † W pre ℓ -1 Σ ( o )1 / 2 x . Note that

Therefore,

2 2 2

where we used x + y + z ≤ 3 x +3 y +3 z . This concludes the proof.

F.7 Proofs for Full Fine-tuning

Define f full ℓ ( x ) = W pre ℓ +1 ( W pre ℓ +∆ full ℓ ) W pre ℓ -1 x as a fine-tuned network with full fine-tuning applied to the ℓ -th layer, evaluated under the population in-distribution risk, where ∆ full ℓ is obtained by

Lemma F.17 (In-distribution Excess Risk) . For f full ℓ , it holds that

Proof of Lemma F.17. Similar to the proof of Theorem F.10, we have

and

where we used the fact that the inner products tr ( T 1 T ⊤ 2 ) = tr ( T 2 T ⊤ 3 ) = tr ( T 3 T ⊤ 1 ) = 0 . By choosing ∆ = ( W pre ℓ +1 ) † D Σ ( i ) x W pre ⊤ ℓ -1 A † for example, the term T 1 becomes 0 . Thus

This gives the desired result.

We obtain the following corollary as a direct consequence of Lemma F.17.

Corollary F.18. For f full ℓ , it holds that

The first term on the right hand side of (27) measures the distance between two subspaces spanned by Ψ ∗ ( D Σ ( i )1 / 2 x ) and Ψ ∗ ( W pre ℓ -1 Σ ( i )1 / 2 x ) . Intuitively, this quantifies the information coded at the ℓ -th layer, and the necessary information to predict residuals. Thus, it bounds the maximum improvement by the ℓ -th layer fine-tuning. The second term measures the subspace distance between the subspace where prediction residuals reside, and the subspace predictable by the ℓ -th layer fine-tuning.

G Auxiliary Results for Proofs

Lemma G.1. Fix s, d 1 , d 2 ∈ N + . For any A,B ∈ R d 1 × d 2 , if ∥ B -A ∥ op ≤ ∥ A ∥ op and λ s ( A ) > λ s +1 ( A ) hold, then,

Proof. By triangle inequality,

Using Davis-Kahan theorem (Theorem 4 from [73]), and Lemma 2.6 from [11],

Thus

This concludes the proof.

We cite the concentration inequality for cross-covariance matrices from [47].

Lemma G.2 (Proposition 9.1 from [47]) . Let Z and ˜ Z be mean zero random vectors taking values in R d 1 and R d 2 , respectively. Denote covariance matrices of Z and ˜ Z by Σ Z and Σ ˜ Z , respectively. Fix any t > 0 . Assume that there exist constants c 1 , c 2 > 0 such that

holds for any γ ∈ R d 1 and γ ′ ∈ R d 2 . Choose n ≫ ( r e (Σ Z ) ∧ r e (Σ ˜ Z ))( t + log( d 1 + d 2 )) . Let ( Z i , ˜ Z i ) i ∈ [ n ] be n independent copies of ( Z, ˜ Z ) . Then, there exists a constant C = C ( c 1 , c 2 ) > 0 such that with probability at least 1 -e -t ,

hold.

Note that if a random variable Z taking values in R d satisfies γ ⊤ Σ Z γ ≥ c ∥ γ ⊤ Z ∥ 2 ψ 2 for any γ ∈ R d with some constant c > 0 , AZ also satisfies γ ′⊤ Σ AZ γ ′ ≥ c ∥ γ ′⊤ AZ ∥ 2 ψ 2 for any γ ′ ∈ R d ′ and any matrix A ∈ R d ′ × d and arbitrary d ′ ∈ N + , where Σ AZ = A Σ Z A ⊤ .

We then prove the following lemma to show the existance of a 'good' high probability event to bound multiple inequalities.

Lemma G.3. Suppose that Assumptions F.1 and F.2 hold. Fix any S ⊂ [ d ℓ ] . Then, there exists an event F with P ( F ) = 1 -exp ( -Ω(log 2 ( n + p + q )) ) such that on the event F , for Φ ∈ { Φ ′ , Φ ′′ S } ,

and

hold. Furthermore,

holds on the event F .

Proof. We only prove for Φ = Φ ′ without loss of generality. Before proving Lemma G.3, we first derive several concentration inequalities. Assumption F.2 implies

and

with high probability. Hereafter we only focus on the event F where these inequalities hold. We divide the proof into 2 parts.

Part 1. In this part we derive (30), (31) and (32). Note that ∥ ˆ A 2 -A 2 ∥ op ≤ λ ∗ ( A 2 ) / 2 holds on the event F since n ≫ κ 4 ∗ ( A ) r e ( A 2 ) log 2 ( n + d + p ) by Assumption F.2, and hence rank( ˆ A 2 ) = rank( A 2 ) . Using Theorem 5.2 from [62],

This yields (30). Proposition 3.2 from [67] and (34) yield,

where Φ ′′′ := Φ ∗ ( A 2 ) , and we used λ ∗ (Φ ′′′⊤ A 2 Φ ′′′ ) ≥ λ ∗ ( A 2 ) . Since ˆ A = Φ ′′′ (Φ ′′′⊤ ˆ A 2 Φ ′′′ ) 1 / 2 Φ ′′′⊤ and A 1 / 2 = Φ ′′′ (Φ ′′′⊤ A 2 Φ ′′′ ) 1 / 2 Φ ′′′⊤ , we obtain (31) as

Again using Theorem 5.2 from [62] combined with Assumption F.2, we obtain (32) as

This yields ∥ ˆ A † ∥ op ≲ ∥ A † ∥ op.

Part 2. Next we derive (33). By a similar argument as Part 1, (36) and Assumption F.2,

We bound Q 1 , Q 2 , R 1 and R 2 separately. For the terms Q 1 and Q 2 , (37) and (38) give

For the term R 1 , using (35) and (40),

For the term R 2 , using (35) and (36),

where we used ∥ ( ˆ Σ ( i ) x ) † ∥ op ≲ ∥ (Σ ( i ) x ) † ∥ op by Assumption F.2 combined with (40). Again from Assumption F.2, R 1 + R 2 is bounded by the right hand side of (41). Therefore,

Finally, from Assumption F.2, we obtain ∥ Φ ⊤ ˆ D ˆ Σ ( i ) x W pre ⊤ ℓ -1 ∥ op ≲ ∥ D Σ ( i )1 / 2 x ∥ op ∥ Σ ( i )1 / 2 x W pre ⊤ ℓ -1 ∥ op. This concludes the proof.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the contributions of this work.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the Conclusion Section (section 8).

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main assumptions and theorems are provided in Section 4, while additional details and complete proofs can be found in Appendix F.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: The paper has disclosed all the information in the method and experiment part.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA] .

Justification: We have the code required to reproduce our experimental results and are working towards making our code available in a public GitHub repository.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: The experimental setting is clearly described in Section 2, Section 5 and Section 6, and we will make our code available in a public GitHub repository.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: All statistics and results included in the paper are accompanied by confidence intervals.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: Information for the resources required to reproduce the experiments are included in the oaoer. All experiments are run with 4 x A100 (80G). For the efficiency analysis, a single A100 GPU was used.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper fully conforms with the NeurIPS Code of Ethics in every respect.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] .

Justification: We discuss the broader impacts of our work in Appendix.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: Our paper does not introduce any assets that have a high risk for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: We have explicitly mentioned the citations for the datasets and have ensured that all conditions are fully respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: Upon acceptance, we will make our codebase publicly available and complete documentation for our assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: We do not include any experiments with human subjects or crowdsourcing.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: We do not include any experiments with human subjects or crowdsourcing.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the fact that the current parameter efficient fine-tuning methods cannot simultaneously achieve high-quality, efficient training or scalable LLM services. Therefore, a series of structured sparse fine-tuning methods for LLM are proposed, which simultaneously achieve state-of-the-art fine-tuning performance, training efficiency and reasoning scalability. S2FT achieves this through \"sparse selection and dense computation\". It only selects a few heads in the MHA module and a few channels in the FFN module in each Transformer block. Then the weight matrix of the coupling structure in the LLM is rearranged so that the selected parts are interconnected to generate multiple compact and dense trainable weight sub-matrices. S2FT only needs to update these sub-matrices for efficient parameter fine-tuning. The paper focuses on the fact that current methods such as LoRA, DoRA, etc., although they can reduce memory, are not as good as fine-tuning in large language models. The unstructured characteristics of SFT require matrix sparse operations, which makes service scalability and training efficiency impossible to guarantee. This is the main reason and motivation for the proposal of S2FT in this paper. In addition, this paper establishes an interface between S2FT and LoRA to support joint or non-joint computing paradigms.\n\nSTRENGTHS:\nThe idea of ​​this paper is very novel and valuable. It starts with the three problems that the current LLM model may encounter, namely training quality, efficient training, and service scalability. It proposes a structured sparse fine-tuning method to accelerate the large language model and provides corresponding mathematical proofs. This method is very effective for both in-distribution and out-of-distribution model training. The highlights of this paper are as follows:\n1. The mathematical theoretical proof of this paper is very complete, the proof ideas are very clear, and the proofs of the in-distribution model and the out-of-distribution model are very complete. The conclusions and theoretical analysis of this paper are clearly seen, and the mathematical theory is very solid.\n2. The analysis process of this paper is complete. The advantages and disadvantages of the current research are analyzed in place in the literature review part, the method part is concise and clear, and a complete theoretical proof is provided. The advantages of operating efficiency and service scalability are also analyzed in the experimental part.\n3. The simplicity and clarity of this method is a highlight of this paper. Compared with the complicated fine-tuning methods of other methods, this paper only needs to select and rearrange the matrices of some layers for updating, and develops a partial back-propagation algorithm. The number of lines of code implemented is very small. Simplicity is a big highlight of this paper.\n\nWEAKNESSES:\nThe advantages of the S2FT method in this paper are very prominent, and it provides some references for efficient, scalable, and high-quality fine-tuning methods for large language models. However, the author can consider the following areas for improvement or sufficient explanation: \n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless. \n2. There are few figures for the method in this paper, only the right picture of Figure 2 is shown, and the overall picture is not clear enough. In addition, does this method extract the same layer for each different model? Or is it better to extract specific layers for each model? \n3. This paper locks the U of the low-rank decomposition matrix, which is a good attempt, but for different data sets, will the distribution of data affect the data form of the U matrix and the form of the U matrix basis? \n4. In the experimental results, there are many phenomena that the method used in this paper is lower than the previous methods. Can the author analyze each task more fully, because it can be seen from the table that some tasks are more obviously degraded?\n\nQUESTIONS:\n1. The symbols in the proof process of this paper can be slightly organized. Readers may feel that the symbols of the proof are a little messy when reading the entire proof process, but the whole process is complete and flawless. \n2. There are few figures for the method in this paper, only the right picture of Figure 2 is shown, and the overall picture is not clear enough. In addition, does this method extract the same layer for each different model? Or is it better to extract specific layers for each model? \n3. This paper locks the U of the low-rank decomposition matrix, which is a good attempt, but for different data sets, will the distribution of data affect the data form of the U matrix and the form of the U matrix basis? \n4. In the experimental results, there are many phenomena that the method used in this paper is lower than the previous methods. Can the author analyze each task more fully, because it can be seen from the table that some tasks are more obviously degraded?\n\nLIMITATIONS:\nThis paper mentions some limitations of this paper in the limitations. For example, although various model architectures show coupled structures, this paper does not explore the opportunity to extend S2FT to other architectures. At the same time, in terms of model deployment, although this paper verifies the feasibility of scalable services in the work, there is still a lack of a practical and scalable actual service system. However, when using this method, is it necessary to consider the fairness or importance of each layer? For example, for a large language model, the first layer may contain more information, and some information in the middle layer can be appropriately eliminated. \nSecondly, whether it is truly quality-oriented needs to be considered because S2FT does not perform well in any task in the experiment of this paper. Finally, this paper also mentions that the subsequent optimization direction may be targeted at the residual dependency network, and how to perform S2FT on this type of network needs further consideration. However, since the work of this paper mainly focuses on PEFT, it leads to a reduction in GPU computing resource consumption. Therefore, this method has the potential to have a positive impact on the computing environment by minimizing the computing resources required for fine-tuning LLM.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
The paper introduces a structured pruning method for LLMs. The main idea is to permute the rows and columns of the weight matrices and select a submatrix during the fine-tuning process. The authors show that the proposed technique outperform previous techniques in terms of accuracy and efficiency.

STRENGTHS:
1. The paper is well written. The authors gave a good summary of existing parameter-efficient fine-tuning (PEFT) methods. While the proposed technique itself is not entirely new (it can be considered as a special form of structured sparse training), the authors described the technique clearly. 
2. Theoretical analysis is provided for generalization performance. 
3. Experiments are comprehensive. The authors tested their technique for two types of tasks (commonsense reasoning and instruction-following) with five fine-tuning categories on more than 10 datasets.

WEAKNESSES:
Novelty in the pruning technique itself is limited. It is basically a structured sparse training method. Similar structured pruning methods have been proposed on non-LLM models previously (e.g., [27, 24]).

QUESTIONS:
Will the authors open-source the code?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
Current PEFT methods for LLMs fail to achieve high quality, efficient training, and scalable serving simultaneously. To overcome this, the authors developed Structured Sparse Fine-Tuning (S²FT), which excels in all three areas. S²FT improves generalization by selecting a few heads in the Multi-Head Attention (MHA) and channels in the Feed-Forward Network (FFN) modules for each Transformer block. It forms dense, trainable submatrices by co-permuting weight matrices, preventing overfitting and forgetting. S²FT achieves state-of-the-art performance and reduces fine-tuning memory usage by up to 3 times and increases throughput by 1.5-2.7 times.

STRENGTHS:
1. This paper is well-written and organized. The proposed method is technically sound. 
2. Efficient tuning of LLMs is an important topic.
3. The performance of the proposed method is very promising.

WEAKNESSES:
1. The author claimed that S2FT prevent overfitting and forgetting. However except the toy experiments in 2nd section, I did not see any experimental proof. E.g. The author only gives MTBench results after the finetuning on AlpacaGPT4, what about the overfitting and forgetting issues?
2. Experiments only conducted on relative small models.
3. The name SFT may incur some confusions as it already has the meaning of supervised fine-tuning.
4. The experiments parts are not sufficient to proof the effectiveness of the proposed method especially on some downstream tasks.

QUESTIONS:
1. The author has some ablations on the trainable allocation choice within a block, what about the allocation in different layers?
2. Can this paper be combined with quantization just like QLoRA.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new family of methods called Structured Sparse Fine-Tuning (S$^2$FT) for large language models (LLMs). S$^2$FT aims to achieve state-of-the-art fine-tuning performance, training efficiency, and inference scalability simultaneously. The method selects a few heads in the multi-head attention (MHA) module and a few channels in the feed-forward network (FFN) module for each Transformer block, then co-permutes the weight matrices to connect these selected components. This results in multiple compact, dense, and trainable weight submatrices that are updated during fine-tuning. The approach prevents overfitting and forgetting, delivers superior performance on benchmarks, and improves memory and throughput efficiency compared to full fine-tuning and existing parameter-efficient fine-tuning (PEFT) methods.\n\nSTRENGTHS:\nThis paper introduces a method combining structured sparsity with fine-tuning, enhancing both efficiency and performance.\nThe method proposed can reduce memory costs and improve throughput.\nThe method demonstrates strong generalization capabilities.\nThe approach allows for scalable batched serving of multiple fine-tuned models without additional inference overhead.\nThe experiments are comprehensive.\n\nWEAKNESSES:\nWhile the results look impressive, the reviewer is not fully convinced by the theoretical motivations. For example, the paper relies on the assumption that structured sparsity can effectively represent the necessary model adaptations, which is hard to verify in realistic settings. \n\nAlthough the benchmarks are comprehensive, most of the benchmarks do not include SFT or other SFT-based methods. Since S$^2$FT is in some sense an SFT-based method, it would be necessary to include such comparisons to understand if the performance improvements come from the \"structure\" (i.e. the $^2$) or SFT itself. Based on the current benchmarks, it is difficult to make such conclusions.\n\nWhile the paper includes some implementation details, aiding reproducibility, the code is not provided, making it hard to verify or reproduce the results, especially for a paper with mostly empirical results.\n\nQUESTIONS:\nCould the authors elaborate on the scaling of trainable parameters, as well as the space and time complexity, compared to other SFT-based or LoRA-based methods?\n\nCould the authors elaborate on how the hyperparameters are chosen, and provide a more comprehensive list of hyperparameters used in this paper, the procedure to choose them, and the discarded hyperparameters?\n\nRight now, the paper mainly discusses how to apply S$^2$FT to attention-based LLMs. The reviewer wonders how to identify the important weights and apply this method to other models.\n\nMost of the experiments were performed on LLaMA and LLaMA2. The reviewer wonders if there is any difficulty in applying the method to LLaMA3.\n\nIs it possible that different datasets/tasks could result in a different set of \"important weights\"? The reviewer wonders if the authors explored this possibility.\n\nThe reviewer wonders if the author could release the source code into an anonymous repository for review purposes.\n\nIn the NeurIPS checklist, the author claims that their results include confidence intervals. Could the authors make some comments on the confidence intervals for the tables in the paper?\n\nLIMITATIONS:\nThe limitations and broader impacts are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}