Expanding Sparse Tuning for Low Memory Usage
Authors: Shufan Shen
Junshu Sun
Xiangyang Ji
Qingming Huang
Shuhui Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: sparse tuning
parameter-efficient fine-tuning
vision model fine-tuning
Primary area: machine_vision
TL;DR: A PEFT method leverage the high performance of sparse tuning with LoRA-level memory footprints.

Abstract: Parameter-efficient fine-tuning (PEFT) is an effective method for adapting pre-trained vision models to downstream tasks by tuning a small subset of parameters. Among PEFT methods, sparse tuning achieves superior performance by only adjusting the weights most relevant to downstream tasks, rather than densely tuning the whole weight matrix. However, this performance improvement has been accompanied by increases in memory usage, which stems from two factors, i.e., the storage of the whole weight matrix as learnable parameters in the optimizer and the additional storage of tunable weight indexes. In this paper, we propose a method named SNELL (Sparse tuning with kerNELized LoRA) for sparse tuning with low memory usage. To achieve low memory usage, SNELL decomposes the tunable matrix for sparsification into two learnable low-rank matrices, saving from the costly storage of the whole original matrix. A competition-based sparsification mechanism is further proposed to avoid the storage of tunable weight indexes. To maintain the effectiveness of sparse tuning with low-rank matrices, we extend the low-rank decomposition by applying nonlinear kernel functions to the whole-matrix merging. Consequently, we gain an increase in the rank of the merged matrix, enhancing the ability of SNELL in adapting the pre-trained models to downstream tasks. Extensive experiments on multiple downstream tasks show that SNELL achieves state-of-the-art performance with low memory usage, endowing PEFT with sparse tuning to large-scale models. Codes are available at https://github.com/ssfgunner/SNELL.

Expanding Sparse Tuning for Low Memory Usage
Shufan Shen 1,2 Junshu Sun 1.2
Xiangyang Ji
Qingming Huang 1.2,4
Shuhui Wang 1.4+
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS
?University Chinese Academy of Sciences TTinnghua University *Peng Cheng Laboratory (shenshufan222 22222 sunjunshu21s, wangshuhui)@ict...
xyji@tsinghua edu, en
quhuangducas ac.cn
Abstract
Parameter-efficient fine-tuning (PEFT) is an effective method for adapting pre- trained vision models to downstream tasks by tuning a small subset parameters. Among PEFT methods, sparse tuning achieves superior performance by only ad- justing the weights most relevant to downstream tasks, rather than densely tuning the whole weight matrix. However, this performance improvement has been companied by increases in memory usage, which stems from two factors, i.e., the storage of the whole weight matrix learnable parameters in the optimizer and the additional storage of tunable weight indexes. In this paper, we propose a method named SNELL (Sparse tuning with kerNELized LoRA) for sparse tuning with low memory usage. To achieve low memory usage, SNELL decomposes the tunable matrix for sparsification into two learnable low-rank matrices, saving spar-
from the costly storage of the whole original matrix. A competition sification mechanism is further proposed to avoid the storage of tunable weight indexes. To maintain the effectiveness of sparse tuning with low-rank matrices, we extend the low-rank decomposition by applying nonlinear kernel functions to the whole-mann matrix merging. Consequently. we gain an increase in the rank of the merged matrix, enhancing the ability of SNELL in adapting the pre-traue pre-trained models to downstream tasks. Extensive experiments multiple downstream tasks show that SNELL achieves ttate-of-the-an performance with low memory usage, endowing PEFT with sparse tuning to large-scale models. Codes are available at https//guhnbbboo/ssgggmeesr/s.
Introduction
Fine-tuning has become predominant way for adapting large pre-trained models to downstream tasks with limited training samples [13 24 23). Nevertheless, fine-tuning all model parameters requires substantial memory usage and is susceptible to over-fitting, making it costly and infeasible for large- To address these limitations, parameter-efffiient fine-tuning (PEFT) [64] 27 22 has been proposed to tune small subset of parameters while keeping other parameters frozen. PEFT methods can be categorized into addition--asee and reparameterization based methods. The former attaches additional parameters to frozen pre- me-trained backbone, while the latter adjusts the original parameters in the pre-trained backbone.
scale models 58 III
62
30 8
have achieved remarkable performance on vision tasks. However, adopting additional parameters incurs extra computational costs during the inference process. In contrast, reparameterization--aass
Addition-based methods 53 30
methods 7 27 directly fine-tune the original parameters. These methods select specific parameters, involving reduced memory usage compared full-parameter fine-tuning. Based on the granularity of parameter selection, one primary approach focuses on
"Corresponinng author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Matrix in Optimizer
Tunable Weight Indexes
Tunable Parameters
Frozen Parameters
Memory
Fall Finz- tuning Store Thring RSELL
Update
Memorr Usage (GB)
Usage
Full Fine-tuming
Sparse Tuning
Menged by Kernel
Compeiition
Rased
(b)
Low Rank
Spursify
Matrices
600
Model Size (M) (c)
Sparse Tuning with Kernelized LoRA (SNELL)
Figure I: (a) The high memory usage of sparse tuning arises from taking the whole weight matrix as learnable parameters, in addition to the storage of the tunable weight indexes (typically represented as a binary mask). (b) Our framework (SNELL) only stores the learnable low-rank matrices in the optimizer. (c) Memory usage comparison pre-trained models with different depths.
specific parameter matrices. For example, Bitfit 5 only adjusts bias to reduce the volume of tunable parameters while Partial-k [30] fine-tunes the last few layers to avoid back-propage packppropagation through the entire pre-trained backbone. To further reduce memory usage, LoRA [27 optimizes each selected weight matrix using two low-rank matrices to achieve memory-cfficiee
fine-tuning. Although sufficient in reducing memory usage, these methods usually gain inferior performance compared to addition-basee methods 330 Recently, SPT (2 and GPS 63 found that combining existing PEFT methods with sparse tuning, which only adjusts the most task-related weights in matrix, can achieve state-of-ther the-art performance on vision tasks. Concurrently, the effectiveness of sparse tuning has also been observed in NLP tasks (18). By focusing on individual weights in matrix, sparse tuning allows for more precise adjustments, thus achieving good performance and mitigated over-fitting risks [18].
However, the performance gained from sparse tuning has been accompanied by high memory usage, as Figure [Ta) shows. Although sparse tuning only updates part of weights in the pre- traineaned weight matrix, the whole matrix still needs to be stored as learnable parameters in the optimizer and computed for their corresponding gradients in practice. Additionally, sparse tuning necessitates storing the tunable weight indexes, further aggravating the memory demands. The above observation indicates that sparse tuming gains no advantage over full fine-tuning regarding memory usage, especially given the increasing parameter volumes in pre- pre-trained models [58 2 A sparse tuning method with low memory usage is urgently required for applications on large-scale pre-traine trained models.
In this paper, we propose method that conducts Sparse tuning with kerNELized LoRA (SNELL) shown in Figure [b). SNELL can adapt pre-trained models to downstream tasks with both low memory usage and strong performance. To reduce memory usage, decompose the tunable matrix for sparsification into low-rank learnable matrices to store fewer parameters in the optimizer and develop immpetttion-based method to avoid storing the tunable weight indexes. To improve the performance on downstream tasks, we extend LoRA from a kernel perspective and merge low-rank matrices with nonlinear kernel functions obtain matrices with higher ranks.
Specifically, SNELL updates the pre- pre-trained weight matrix using a sparse low-rank adaptation matrix. This adaptation matrix is first merged with two low-rank learnable matrices and then sparsified toward effective fine-tuning. Compared to storing the whole adaptation matrix, storing low-rank matrices in the optimizer results in lower memory usage. For the sparsification process, we propose method inspired by the neuron competition phenomenon in neuroscience [49], avoiding the storage of the tunable weight indexes that incur additional memory usage. The proposed method promotes competition among weights based on their absolute values. Most task-relevant weights are encouraged to have larger absolute values and survive during the fine-tuning process. By setting a sparsity ratio as the hyperparameter and determining tunable weights based on their absolute values in an end-to-end manner,
competition-
eliminate the storage of the tunable weight indexes.
In addition to low memory usage, the performance also critical for model fine-tuning. However, directly merging two low-rank matrices through the inner product leads the low-rank structure of the adaptation matrix, which narrows the optimization scope of tunable matrices and further limits the expressiveness of sparse tuning. To overcome this bottleneck, draw inspiration from DyN 45

---

on weight matrix interpretation based on low-dimensional dynamical systems, and reformulate the merging process with nonlinear kernel functions that increase the rank of the merged adaptation matrix. This new formulation enables more expressive sparse tuning while maintaining a compact representation with low memory.
Extensive experiments are conducted on 24 downstream visual recognition tasks with both plain and hierarchical vision Transformer backbones under supervised and self-supervised pre- pre-training. Results show that SNELL can gain the performance improvement of sparse tuning and the low memory usage of LoRA concurrently. SNELL obtains the -tate-offin of-the-art performance on FGVC (91.8% vs. 90.7%) and VTAB-IK Ik (74.6% vs. 74.1% %) benchmark with LoRA-level memory usage. Moreover, as fi)) shows, the low hemory-usage advantage of SNELL becomes increasingly apparent as the model size grows, enabling sparse tuning on larger models.
2 Related Work
Parameter Sparsity. In early work, the parameter sparsity usually serves an optimization objective in model pruning [21] 42). These pruning methods remove the weights from pre-trained models irrelevant to a specific task, without significantly degrading model performance. The relevance of individual weights can be estimated based on activations (28]. redundancy [88. per-layer second derivatives [15]. and energy efficiency [57]. Except for the post-training pruning strategy, sparse net- works 44 41 17 directly introduce parameter sparsity into the training process, removing redundant weights more precisely [17]. Motivated by the advantage of parameter sparsity in model optimization. recent studies introduce sparsity to the the-ttunng of pre-trained models and achieve enhanced model performance on downstream tasks 21 56 The parameter sparsity gives rise to reduced number of trainable parameters and serves as a regularization constraint during fine-tuning [18]. Among sparse tuning, pre- pruning methods adopt model pruning for fine-ture tuning. These methods sparsify the weight matrix [22] 63 or adapter [1] through pruning metrics 43 to identify learnable parameters for the fine-tuning process. Other methods select trainable parameters during fine-tuning, including learnable mask [64] or diff vectors (20) with sparsity constraints. However, the parameter sparsifica- tion methods need to store the indexes of tunable weights, which incurs additional memory usage. For sparse tuning under low memory budget, competition- based mechanism selects weights relevant to downstream tasks in learnable manner without storing the tunable weight indexes.
Parameter-efffiient Fine-tuning. Fine-tuning is the most predominant approach for adapting pre-traine trainedi model to downstream tasks. However, for large pre-trained models, fine-tuning all parameters is costly and prone to overfit downstream datasets. To tackle these problems, parameter- efficient fine-tuning (PEFT) 8 30 62) which tunes only a tiny portion of parameters, becomes a desirable choice. Following the taxonomy of SPT (22). PEFT methods can be categorized into and reparameterrization--uus
budet
3 26
46
50
27 methods.
Addition-based methods attach additional trainable parameters to a frozen pre-trained backbone. 46 50 (1 adopt a residual pathway and learn a bottleneck layer including two linear projections and a non-linear activation. Prompt-tuning methods [44 31,
Adapters [3] 266
37 34] add trainable
parameters to the input and keep the entire pre-trained model unchanged during training. Recent work 62) attempts to find the optimal configurations to combine multiple addition- based methods. Despite of the popularity and effectiveness of addition -bsed methods, the additional trainable parameters incur excess computational costs during the inference process
33
Reparrametization--bree methods adjust the inherent parameters in the pre-trained backbone to avoid excess computational costs during inference. Farly work directly selects parameters with low memory usage for fine-tuning, such as the bias terms [5] and the final few layers of the pre-trained model (7]. To further reduce the memory usage of the selected matrices, LoRA [27] optimizes low-rank matrices that can be reparameterized into the pre-trained weight matrices to reduce memory usage. Exploring aner-erained parameter selection, some studies 220 64) propose sparse tuning, which involves selecting and tuning individual weights sparsely within the weight matrices. Recently, SPT [22] combines sparse tuning and LoRA in hybrid framework that achieves state-of- ofte--of-the-art performances on visual PEFT tasks. SPT has revealed that optimizing the weights most relevant to the downstream task through sparse tuning can significantly enhance the performance, which is also supported by SAM [18 and GPS [X]. However, existing sparse tuning framework still faces the challenge of high memory usage brought by sparse tuning. Unlike existing methods, our SNELL
3

---

inherits high performance and low memory usage concurrently by sparsifying the adaptation matrix merged with low-rank matrices through nonlinear kernels.
Methodology
27 and kernel trick 322 (Section 3.1). Then we propose SNELL. sparse tuning method including kernliized LoRA that enables high-performance tuning with low-rank learnable matrices (Section 3.2) and competition- mechanism that sparsifies weights without additional memory usage (Section 3.3)
We first introduce the definitions of sparse tuning 64
20
LoRA
3.1 Preliminaries
Sparse Tuning. Given downstream training set D {a("), " is to minimize the model's empirical risk on the downstream task, with the sparsity constraints on the volume of tunable weights weight matrix W E RR=x,, The sparsification is usually achieved through a binary mask M E {0, 13mxn. The objective function can be formulated as
Jn=1 the objective of sparse tuning
N
Σ c (faal)) W). N
1
min M
where -(::-) is a parameterized function over the input (c.g., a neural network), L(:,-) -) loss function (e.g., cross-entropy), and @ denotes element-wise multiplication. The binary mask M can be either a fixed hyperparameter, pre-computed with heuristics such as pre-pruning [22], or a learnable parameter obtained through end-to-end fine-tuning [64]. All these methods require storing M to determine the tunable weights, which results in additional memory usage. More importantly, the tunable parameters W @ M occupy the same amount of memory as the weight matrix W in practice. As result, the memory usage sparse tuning even higher than that full fine-tuning.
LoRA. Given a pre-trained weight matrix Wo. LoRA (27 optimizes two low-rank matrices B E I Xr A E gaxr to reduce the memory usage during fine-tuning. The low-rank matrices A and B can be reparameterized into the pre-trained weight Wo.
W Wo AW Wo BAT
(2)
With < min(m, n), LoRA achieve high training efficiency and low memory usage by only optimizing the smaller low-rank matrices.
Kernel Trick (32. In many machine learning tasks, mapping the vectors into higher dimensions is frequently used to achieve lincar separability [5]]. However, the explicit mapping process incurs significant computational costs. To address this problem, the kernel trick is proposed to efficiently model data relationships in high-dimensional spaces, without the need to explicitly formulate the space. According to Mercer's 's theorem (6), kernel function IR" R + R express inner product in some space as x(x, x') (xx) d(x'), if and only if x. is positive semi-definite (Appendix B). x, E IR", and á R' + Fd is an implicit feature map. By selecting an appropriate kernel function K, obtain the inner product of two vectors in digher-dimensional space Id (d r) without explicitly formulating the feature map Φ.
3.2 Kernelized LoRA
We leverage LoRA reduce the memory usage of sparse tuning in light of its low memory usage. An intuitive solution is to sparsify the adaptation matrix W composed of the two low-rank matrices. However, the low-rank property of AW can lead to the performance degradation of sparse tuning. For the original sparse tuning, the weight matrix W free of the rank constraint, and weights are independent of each other. Therefore, we can independently select and optimize weights most relevant to the downstream task. For sparse tuning with LoRA, the adaptation matrix AW with rank is constrained R" (m-+n-7), a subspace of mm When r & min(m, n), the weight optimization scope of sparse tuning contracts, hindering its performance on downstream tasks.
To achieve sparse tuning with both strong performance and low memory usage, we propose to construct a high-rank matrix using low-rank matrices. Inspired by DyN (45) that fits a high-rank

---

A
R Raxr
B
VER
AW
K( A. Amms
Sparsify
W
Prediction
Awwl
Bi,
Ay.
Equivalent
B, AW
B
ABB,
Absolute y Ranking
[aw,
<+>
Lince Meriig
Vesturs
A1
Noolinaar Merging
((A)))
Sparsity Ratio E (0, 1]
Tunble Weight
AER R
B maae
A. gund B. Rmdd
Threshold Acquirement
Presing Weights
Kernelized LoRA
Competition-bee Sparsification
Figure 2: Overview of our SNELL stratey.. Given two learnable low-rank matrices, we merge them using a non-linear kernel function (left). This merging process is equivalent to mapping the matrices to higher-rank matrices and then performing matrix multiplication. Then we sparsified this merged adaptation matrix using a competition- based sparsification mechanism (right). This mechanism zeros out weights with small absolute values based on the specified percentage of
matrix using the distance matrix of low-dimension dynamical system, we extend the distance function to general kernel functions and investigate LoRA in the kernel perspective. Given two vectors æ, x E RR, the kernel function k(x, x²) be formulated with an implicit feature map & R" + Rd The merging process of LoRA can be seen as applying linear kernel function ki((:;) on the rows of the learnable parameters A and B,
inner product Φ(x) "(x')
AW, = (Ai,,, Bj..) = (B,,))r(A,)) = A,
where A,,. Bj. RT, 0i R' HT denotes the identity mapping. By replacing Ki(--)) with more complex non-linear kernel functions, we can approximate relations in higher-dimensional spaces Rd and obtain matrices with rank larger than r. The merged adaptation matrix in SNELL can be represented by
AW = (I,(A,,,,,) B,n)) = (B..).) of
(( =
Note that in practice, explicit computation of As E Raxd and Bg E Rmmx is unnecessary. AW can be directly derived based on A and B with the kernel function By extending LoRA in kernel perspective, SNELL can build high-rank adaptation matrices based on low-rank learnable matrices, empowering strong sparse tuning with low memory usage. We utilize the piecewise linear kernel introduced in Appendix B without a specific statement.
3.3
Competition- Sparsification Mechanism
Existing methods store tunable weight indexes M E m) for sparsifying the update of the weight matrix W R The storage of M leads to additional memory usage. Inspired by the neuron competition phenomenon in neuroscience [49], we design competition- parameter sparsification mechanism to avoid this additional storage. Instead of determining the learnable weights in the optimization process based on M, our objective is to encourage the weights to compete based on their contributions to performance improvement. Weights with stronger contributions survive in the sparsification while the remaining cowccontributed weights zeroed out. The weight contribution is reflected in their absolute values during the end-to-end optimization. During optimization, weights contributing more to the loss reduction are encouraged to have more significant values, while weights contributing less approach zero. By retaining higher importance to significant weights and zeroing out the less impactful weights, relying on the absolute values of weights, avoiding the storage of M.
achieve end-to-end tunable parameter selection by solely
Specifically, given merged adaptation matrix AW and a sparsity ratio 8 E [0, 1], we sparsify weights with soft-threshold function. induce weight competition during end-to-end fine-tuning, propose dynamic threshold Aws, i.e., the weight having the [smn]] -th smallest absolute value in AW. This threshold ensures that only a fixed proportion (s x 100%) of weights remain non-zero. Therefore, the weights have to compete with each other be selected instead of just having larger absolute value than a fixed threshold.
A AW; AW max AW, g Aw,) 0)),
(5)
5

---

where AW* = (AW;
3j mxn denotes the sparse matrix with sparsity ratio s. In practice, the sparsity ratio s is manually determined regarding specific downstream tasks. Given sparsity ratio s, the training objective in Equation | can be reformulated as
, Wo Wo Wo++AW") AWW))) Σ & (s(o A.B N
min
(6)
This objective encourages weights that are most relevant to the downstream task gain more significant values for survival. Adjusting the sparsity ratio & allows us to control the sparsification process precisely and identify the optimal number of tunable parameters for different tasks.
Experiments
4.1 Experimental Setup
Datasets and Metrics. We evaluate our methods on 24 downstream tasks categorized into two groups following SPT [22] (i) FGVC (30) is a benchmark for fine-grained image classification. This benchmark includes 5 downstream tasks, which are CUB-200-2011 [55], NABirds [25). Oxford
Flowers [4), Stanford Dogs [9] and Stanford Cars [12] We follow the validation splits in [22 if the official validation set is unavailable. (ii) VTAB-Ik (59] is large-scale transfer learning benchmark
consisting of 19 visual classification tasks. VTAB- 1k can be further divided into three groups, i.e., natural tasks with natural images, specialized tasks with images captured by specialized equipment, and structured tasks with images mostly generated from synthetic environments. We use top-1 accuracy averaged within each group as our main metric following 22
Pre-trained Backbones. We conduct experiments on the plain vision Transformer backbone ViT- B/16 [16] that is pre-trained on ImageNet (47) with different pre-training strategies following 22
including supervised pre-training and self-supervised pre-training with MAE [23] and MoCo v3 [00
We also conduct experiments on the representative hierarchical vision Transformer backbone Swin-
(8 and CNN backbone ConyNeXt-Base (39 under supervised pre-training. In addition, we
fine-tune the supervised pre-trained large-scale models (ViT-L/16 [16), ViT-H/14 [16]) on VTAB-
to demonstrate the memory -ffnorr-ffficienyy and
of SNELL.
Competitors. We compare our methods with bddeti methods including MLP-k, VPT- Shallow 30, VPT-Deep (30), Adapter-r (26). and SPT--dapter (2 For reparameterrizationbaass
methods, we compare with Linear, Partial-1, Bias [5) LoRA-r [27.. SSF (35), and SPT-LoRA [22).
Here represents the number of bottleneck dimensions in Adapter-r and the value of rank in LoRA-r and our proposed SNELL-r. Details of the competitors are presented in Appendix A.I We also
provide additional comparisons with other approaches [63 53) in Appendix C.]
Implementation Details. Following SPT 22), we use the Adam optimizer 40 with cosine learning rate decay. The batch size. learning rate, and weight decay are 32, 1e 3, and 1e 4, respectively. We also follow SPT [22) to implement the standard data augmentation pipeline for "TAB-IK and follow SSF [35] for FGVC as well. SNELL is applied the pre- pre-trained weight matrix of all linear layers. For each task, we fine-tune the model with different sparsity ratios 8 to search the optimal volume of tunable parameters for this task. Without specific stating, adopt the piecewise linear kernel (introduced in Appendix BB as the kernel function for SNELL. Ablation studies on different kernel functions are presented in Figure
4.2 Performance on Downstream Tasks
Performance on Different Benchmarks. Experiments on FGVC and VTAB-IK benchmarks indicate that SNELL achieves the best performance with supervised pre-trained ViT-B/16 backbone as shown in Table I SNELL gains large performance improvements over LoRA variants, e.g., SNELL-8 surpasses LoRA-8 significantly by 5.5% terms of mean accuracy on the FGVC benchmark. Moreover, SNELL outperforms the state-of-the method SPT-LoRA by clear margin of 0.5% in terms of mean top-1 accuracy on the VTAB-Ik benchmark. This stems from the fact that SPT-LoRA only performs sparse tuning a portion of the weight matrices while employing LoRA for the remaining part. contrast, the low memory property of SNELL empowers sparse tuning on all the weight matrices, allowing for more precise adjustments and giving rise to superior performance.

---

Table 1: Top-1 accuracy (%) on FGVC and VTAB- benchmarks using ViT-B/16 pre-trained on ImageNet-2lk supervisedly. The best result is in bold, and the second best result is underlined.
FGVC
VTAB-IK
Method
Oxford
Stanford
Stanford Cars
CUB 200
NABirds
Mean Acc.
Natural
Specialized
Structured
Mcan Acc.
Flowers
Dogs
Full
87.3
82.7
98.8
89.4
84.5
88.5
75.9
83.4
47.6
Additional-baset methods
MI.-33
30
85.1
77.3
97.9
84.9
53.8
79.8
67.8
72.8
30.6
57.1
VPT-Shallw
80
86.7
78.8
98.4
90.7
63.7
84.6
76.8
79.7
47.0
67.8
VPT-Deep
84.2
83.6
89.1
78.5
82.4
55.0
Adapter-
26
87.3
84.3
98.4
88.8
68.4
85.5
79.0
84.1
58.5
73.9
Adapter-32 32
84.0
58.3
SPT-Adapter
89.1
83.3
99.2
91.1
86.2
89.8
82.0
85.8
61.4
76.4
MoSA 60
89.3
85.7
99.2
91.9
83.4
89.9
79.9
84.0
50.3
71.4
Reparameter-basee methods
M0
75.9
79.3
77.2
Partial-1
30
85.6
77.8
98.2
85.5
66.2
82.6
69.4
78.5
34.2
60.7
Bias
88.4
84.2
98.8
91.2
79.4
88.4
73.3
78.3
44.1
65.2
LoRA-8
84.9
79.0
98.1
88.1
79.8
86.0
79.5
84.6
60.5
74.9
LoRA-16
27
85.6
79.8
98.9
87.6
72,0
84.8
79.8
84.9
60.2
75.0
SPT-LoRA
83.4
99.5
87.3
85.9
SSF 35
89.5
85.7
99.6
89.6
89.2
90.7
81.6
86.6
$9.0
75.7
SNELL-8 (ours)
89.6
86.8
99.3
92.1
89.9
91.5
82.0
85.7
61.6
76.4
SNELL (ours)
90.3
91.7
61.7
76.7
SNELL-32 (ours)
89.9
87.0
99.4
92.0
90.5
91.8
82.7
86.1
61.8
76.9
Table 2: Top-1 accuracy (%)
VTAB-1k benchmarks using ViT-B/16 backbone pre-trained ImageNet using MAE and MoCo v3 strategies. The best result is in bold.
VTAB-Ik MAE
VTAB-IK MoCo v3
Methods
Natural
Specialized
Structured
Mean Acc.
Natural
Specialized
Structured
Mean Acc.
59.3
79.7
64.3
72.0
84.7
42.0
69.6
Additional methods
Adapter-8
$7.2
78.4
63.4
48.4
49.0
Adapter-33 32
26
55.3
78.8
53.3
62.5
74.2
82.7
47.7
68.2
VPT-Shallow
40.0
69.7
27.5
45.7
67.3
37.6
62.4
VPT-Deep
30
36.0
60.6
26.6
41.1
70.3
83,0
42.4
65.2
SPT-Adapter
22
65.6
82.7
60.7
69.7
76.6
85.0
61.7
74.4
Reparameterizztion based methods
Linear
52.7
30.3
59.6
Partial-1
50
58.4
78.3
47.6
61.5
72.3
84,6
47.9
68.3
Bias
54.6
75.7
47.7
59.3
72.9
81.1
53.4
69.2
LoRA-8
23
57.5
77.7
57.7
64.3
45.1
44.3
LoRA-16
27
$7.3
77.1
$9.9
64.8
16.0
64.0
48.7
42,9
SPT-LoRA
65.4
82.4
75.3
SNELL-8 (ours)
68.3
83.8
63.5
71.8
76.8
63.7
75.5
Performance on Different Pre-training Strategies. Experimental results on models pre-trained us- ing different strategies presented in Table SNELL outperforms the state-of-the-arr performances on models pre-trained with MAE (71.8% vs. 69.8%) and MoCo v3 (75.5% vs. 75.3%). Furthermore, SNELL consistenlyy outperforms other PEFT methods every group of downstream datasets. This demonstrates the general effectiveness of SNELL under different pre-training strategies.
outperform
Performance Different Architectures. Following VPT 30 and SPT 22 apply SNELL to hierarchal vision transformer Swin-B and the CNN architecture Nex-NeX, Experimental results are shown in Table 3 Results on Swin-B demonstrate that SNELL-8 outperforms existing reparameterization based PEFT methods by 0.3% and achieves comparable performance to the state- af-the-art addition-baue method SPT-Adapter For ConvNeXt-Base, SNELL achieves a performance improvement of 0.4% compared to the best-reported result. These results obtained on different architectures further validate the versatility and effectiveness of our SNELL approach.
Memory Usage Comparison. We illustrate the effectiveness of SNELL in terms of memory usage by comparing with various PEFT methods. Figure 3Ja) shows the accuracy and memory usage of different methods on ViT-B/16. Although some methods achieve satisfactory performance, their memory usage is excessively large, even surpassing that of full fine-tuning (e.g. SPT-Adapter and VPT-Deep). comparison, SNELL achieves superior performance on downstream tasks with memory usage comparable to memory methods, including LoRA and Adapter.

---

Table 3: Comparisons on VTAB- benchmark with supervised pre-trained Swin- and ConvNeett-.. Top-1 accuracy (%) is reported. The best result is in bold.
VTAB- B Swin.B
VTAB Ik ConvNeXi-B
Methods
Natural
Specialized
Structured
Mcan Acc.
Natural
Specialized
Structured
Mcan Acc.
Full
79.1
86.2
39.7
75.0
78.0
83.7
60.4
74.0
Additional based methods
3LP-3
73.6
75.2
35.7
61.5
73.8
81.4
35.7
VPT-Deep
30
76.8
84.5
53.4
71.6
78.5
83.0
44.6
68.7
Adapter-8
26
81.7
87.3
61.2
76.7
83.1
84,9
64.6
77.5
SPT-Adapper
83.0
87.3
77.5
83.7
86.2
Reparaacceriizatio--bsss methods
Linear
80
73.5
80.8
33.5
62.6
74.5
81,5
34.8
63.6
Partial-1
30
73.1
81.7
35.0
63.3
73.8
81.6
39.6
65.0
LoRA-8
81.7
87.2
76.3
82.2
84.7
LPT-LORA
22
83.1
87.4
60.4
77.2
83.4
86.7
65.9
78.7
SNELL-8 (ours)
83.3
87.7
61.4
77.5
84.5
87.4
65.6
79.1
17.3
Full Millhacrainn sNcLLiratt SW
Lab4-
SNLLL-4
11.00
Nean
Feod
Bockward
Momory Usage (GB)
Intialiae
Input
Foroord Propagation
(a)
(b)
(c)
Figure 3: (a) Accuracy vs. memory usage (batchsize=64) with supervised pre-trained ViT-B/16 on
VTAB-1k. (b) Memory usage evolutions of full fine-tuning, SNELL, and SNELL storing the merged
adaptation matrix (SNELL storing AW) on ViT-H/14 during fine-tuning (batchsize=8). (c) Model parameter volumes memory usage (batchsize=8). As the model gets larger, SNELL's advantage of low memory usage over full fine-tuning becomes more obvious.
Additionally, we present the memory usage evolutions during the fine-tuning process in Figure 3 to provide a detailed explanation of how SNELL can save memory. In the model initialization stage, SNELL exhibits significantly smaller memory usage compared to full fine fnn-tuning. This is because full fine-tuning stores all weight matrices as learnable parameters in the optimizer, whereas SNELL only stores low-rank matrices with smaller parameter volumes. In the feed-forwart phase, the memory usage increases with the storage of intermediate variables for backpropagation. Unlike other intermediate variables, the adaptation matrix A W in SNELL solely relies the low-rank parameter matrices, which are already stored in the optimizer. Therefore, it can be dumped in the feed-forward phase and recovered in backpropagation immediately, saving from large amount of memory usage (SNELL vs. SNELL storing AW).
Scaling to Larger Models. To investigate the scalability of SNELL to large models, apply it to ViT models varying sizes (VIT-B/16, ViT-L/16, and ViT-H/16 pre-trained on ImageNet21 We follow the experimental setup presented in Section 4.1 except for modifying the batch size for experiments on ViT-H/14 to 8 and changing the search scope of sparsity ratios E (0, 0.9).
As depicted in Figure c), the memory usage of full fine-tuning increases rapidly as the model size grows. This observation highlights that existing PEFT methods like VPT and SPT, despite their advanced performances, incur substantial memory costs when applied to large-scale models due to even higher memory usage than full fine-tuning. contrast, SNELL exhibits a notable advantage in terms of memory usage for larger models (similar to LoRA-8). When applied to ViT- H/14, the memory usage SNELL only approximately 50% that required for full fine-tuning, exemplifying its significant memory -avoin capability large models.
Regarding the performance, as shown Table 4 SNELL-8 outperforms L-RA-8 on dataset groups (Natural, Specialized, and Structured) as well the mean accuracy for both VIT-L and ViT-H on the VTAB-Ik benchmark. This demonstrates the effectiveness of SNELL for adapting large pre-trained models to downstream tasks.

---

Table Comparisons on VTAB-IK 1k benchmark with supervised pre-trained ViT-L/16 and ViT-H/16. Top-1 accuracy is reported. The best result is in bold.
VT ViT-L/16
VTAB-IK ViT-H/14
Methods
Natural
Specialized
Structured
Mean Ace.
Natural
Specialized
Structured
Mean Acc.
LoRA-8
81.2
86.6
53.4
73.7
77.9
84.8
55.9
72.9
SNELL-8
82.3
86.9
56.6
75.3
79.5
85.1
56.9
73.8
Table 5: (a) Performance on VTAB-IK Ik of sparsifying full-rank matrix, the merged adaptation matrix of LoRA-8 and kernelized LoRA-8 (KLoRA-8) with sparsity ratio s 0.9. (b) The mean accuracy on 1k of kernelized LoRA (KLoRA) and SNELL (KL.R -aarssssssiiyiinnng with different ranks of learnable matrices. Perf. Imp. denotes the performance improvement of SNELL over KloRA. (h)
(a)
Matrix
Natural
Specialized
Structured
Mean Acc.
Method
r 8
16
T
32
Full-Rank
80.5
85.1
57.6
74.4
KLoRA SNELL
73.2
73.0
72.7
74.2
74.4
74.6
LoRA-8
61.1
81.2
54.7
65.7
KLoRA-8
79.4
84.5
57.9
73.9
Perf. Imp.
+1.0
+1.4
+1.9
4.3
Ablation Studies
Effect of Kernelized LoRA. We explore the effectiveness of kernelized LoRA by comparing the performance of sparsifying a full-rank matrix, the merged adaptation matrix of LoRA, and the merged adaptation matrix of kernelized LoRA. Experimental results are presented in Table a). We can see that sparsifying the merged adaptation matrix of LoRA significantly underperforms a full-rank matrix. This reveals that the low-rank property of the merged adaptation matrix LoRA greatly compromises the weight selection scope, leading to performance degradation for sparse tuning. However, when we replace LoRA with kernelized LoRA, the performance becomes notably comparable to that of the full-rank matrix under the strong sparsity constraint (s 0.9). This indicates that kernelized LoRA can effectively leverage sparse tuning while maintaining a low memory usage.
Effect of Sparse Tuning. Table b) shows the performance comparison between SNELL and kernelized LoRA to explore the effectiveness of sparse tuning. For kernelized LoRA with different ranks, applying sparse tuning can consistently improve their performance. Moreover, as the rank of the learnable matrix increases, the performance of kernelized LoRA decreases while that of SNELL increases. This difference stems from the model regularization. Similar to sparse regularization, the low-rank property of LoRA that constrains the dependence between individual weights, can also be taken as a form of regularization. As the rank of the learnable matrix increases, the effect of low-rank regularization diminishes. Consequently, kernelized LoRA becomes more susceptible to over-fittinn werr-itting and encounters performance degradation. In contrast, SNELL employs both low-rank and sparse regularization. Higher ranks enable better sparsification towards downstream tasks, boosting sparse regularization that counteracts the diminished low-rank regularization. Therefore, higher rank may lead to over-fitting in kernelized LoRA, but it can further enhance performance with sparse tuning.
Effect of Different Kernel Function. We investigate the effectiveness of different kernel functions in kernelized LORA. First, we explore the ability of different kernel functions fit randomly generated full-rank matrices based on low-rank matrices using the gradient descent algorithm (introduced in Appendix A.3). As shown Figure Ja), a)), we explored four kinds of kernel functions. Compared with the linear kernel function, nonlinear kernel functions can reconstruct the full-rank target matrix more accurately based on the low-rank matrices. Subsequently, we explore the performance of kernelized LoRA with different kernel functions for pre-trained model fine-tuning in Figure 4b... We find that using piccewise linear distance as the kernel function can achieve better results compared to linear kernel function (LoRA), while using Sigmoid and RBF kernels leads to severe performance degradation. This is because the complex non-linear kernel functions such as the exponential function increase the optimization difficulty in deep networks shown in Figure 4fc). More comparisons between LoRA and kernelized LoRA (with piecewise linear kernel) are presented in Appendix C33.2
Optimal Sparsity Ratio for Different Downstream Tasks. We provide the optimal sparsity ratio of SNELL-8 on tasks from VTAB- benchmarks in Figure 5 The optimal sparsity ratio varies

---

0.36
Litoe (LoRA)
100
Linn (AAA))
XB8 Parovin Linoe
Linor (LoRA)
0.30
Sqaaod
Meau Acc. (96)
signest
Traniig Loos
MSB Lose
RBP
RBF
0.15
Piscentia Linser
Plassrro Linser
0.30
0000
20
60
Natural
Speciallad
Streotzzed
30
90
02
90
Hank
(Rank=8)
Epoch
(b)
(c)
Figure 4: (a) The fitting ability of different kernel functions. We fit random sparse matrices by merging two learnable low-rank matrices with different kernel functions and compute the MSE loss. (b) Performance comparison groups of datasets in V.AB-1k. (c) Training loss on CIFAR-100
dataset VTAB- 1 benchmark of kernelized LoRA with different kernel functions.
Nataal
0.6
Spucialad
0.8
Sparsit Ratio
Spansity Ratoo
0.6
0.4
0.2
0.
0.0
Briesdedy
Clene
Chee
age
wiReo de
Natural
Spenialinn
Staettre
Figure 5: The optimal sparsity ratio of SNELL-8 different downstream tasks (left) and the average optimal sparsity ratio within each group (right) in VTAB-IK benchmark. The pre-trained model is the ConvNeXt-B pre-trained ImageNet-21k.
significantly across different downstream tasks within the group (e.g., Cifar Sun397, dSpr- loc vs. Clevr-Dist). Furthermore, we can observe that the Natural task group exhibits higher average optimal sparsity ratio compared the Specialized group, while the Structured group demonstrates the lowest ratio. This observation aligns with the example illustrated in Figure A6] where cross-domain adaptation from a model pre-trained on natural images (ImageNet) images of Specialized and Structured groups necessitates a larger number of tunable parameters.
Conclusion
In this work, we proposed PEFT method named SNELL (Sparse tuning with kerNELized LoRA) to conduct regfoperformance sparse tuning with low memory usage. To reduce memory usage, sparsified the adaptation matrix merged with low-rank matrices rather than the pre-trained weight matrix to reduce the volume of learnable parameters stored in the optimizer. Then designed competition-batt sparsification mechanism to avoid the additional memory usage of storing the tunable weight indexes. To reveal the effectiveness of sparse tuning, utilize nonlinear kernel functions to merge the adaptation matrix, increasing the rank the merged matrix to maintain a compact representation suitable for sparse tuning with low memory usage. Extensive experiments demonstrated the ability of SNELL to leverage the high performance of sparse tuning and the low memory usage of LoRA. For future work, will apply SNELL larger models such LLMs and improve its training efficiency. For limitations discussion, please refer to Appendix E.2
Acknowledgement
This work was supported in part by the National Key R&D Program of China under Grant 2023YFC2508704 in part by the National Natural Science Foundation of China: 62236008, U21B2038. and 61931008, and in part by the Fundamental Research Funds for the Central Universi- ties. The authors would like to thank Zhengqi Pei, Yue Wu, and the anonymous reviewers for their constructive comments and suggestions that improved this manuscript.
10

---

References
[1] Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages Association for Computational Linguisties.
Dublin, Ireland, 2022.
[2] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. ArXiv preprint, 12.0/233..00785, 2023.
[3] Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing pages 1538- 1538-1548 1548, Hong Kong, China, 2019. Association for Computational Linguistics.
[4] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert A. Legenstein. Deep rewiring: Training very sparse deep networks. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[5] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficte fine-tuming for transformer-basee masked eeguage-mmdels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, 2022. Association for Computational Linguistics.
[6] Alain Berlinet and Christine Thomas- Aggnan Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011.
[7] Sergi Caelles, Kevis- KokitsK Maninis, Jordi Tuset, Laura Taixe Daniel Cremers, and Lue Van Gool. One-shot video object segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2329. IEEE Computer Society, 2017.
[8] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 15:16664-16678, 2022.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceedings of Muchine Learning Reseurch, pages 1597-1607. PMLR, 2020.
[10] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self supe-supe vision transforrmrs. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9620-9629. IEEF, 2021.
(11) Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Process- ing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 3965-3977,205
[12] Dataset. Novel datasets for -ne-grained image categorization. In First Workshop on Fine Grained Visual Categorization, CVPR. Citeseer. Citeseer. Citeseer, 2011.
[13] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota, 2019. Association for Computational Linguisties.
11

---

[14] Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. OpenPrompt: An open-source framework for prompt-learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 105-113, Dublin, Ireland, 2022. Association for Computational Linguisties.
[15] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4857-4867, 2017.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[17] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[18] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In P Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 22799-12807 2023.
[19] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In Satinder P. Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February AAAI Press, 2017.
4-9, 2017, San Francisco, California, USA, pages
[20] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4884 Online, 2021. Association for Computational Linguistics.
[21] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William Dally. Eie: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243 254, 2016.
[22] Haoyu Hc, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11825-18355, 2023.
[23) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979- 15988. IEEE, 2022.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. IEEE, 2020.
[25] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge J. Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 595-604. IEEE Computer Society, 2015.
[26] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th
12

---

International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 2019.
[27] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[28] Hengyuan Hu, Rui Peng. Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. ArXiv preprint, abs/1607. 1600003270,, 2016.
[29] Piotr Indyk and Sandeep Silwal. Faster linear algebra for distance matrices. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2022.
Curran Associates, Inc.,
[30] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[31] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference Computer Vision, pages 105-124. Springer, 2022.
[32] Konstantinos Koutroumbas and Sergios Theodoridis. Pattern recognition. Academic Press, 2008.
[33] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross- few-shot learning with speciseciiicc adapters. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 7151-7160. IEEE, 2022.
[34] Xiang Lisa Li and Percy Liang. Prefix Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11 International Join! Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, 2021. Association for Computational Linguistics.
[35] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109-123, 2022.
[36] Shih-Yang Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. ArXiv preprint, abs/2402.09353, 2024.
[37] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. ArXiv preprint, abs/21 10.07602, 2021.
[38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992-10002. IEEE, 2021.
[39] Zhuang Liu, Hanzi Mao, Chao-Yual Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022. New Orleans, LA, USA, June 18-24, 2022, pages 11966-11976. IEEE, 2022.
[40] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
13

---

[41] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 0 regularization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April M0y May 3, 2018, Conference Track Proceedings.
OpenReview.net, 2018.
[42] Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2498-2507. PMLR, 2017.
[43] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1124--12222 Computer Vision Foundation/ / IEEE, 2019.
[44] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[45] Zhengqi Pei and Shuhui Wang. Dynamics-inspired neuromorphic visual representation learning. In International Conference on Machine Learning, pages 27521-27541 PMLR, 2023.
[46] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online, 2021. Association for Computational Linguistics.
[47] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11:2111-222, 2015.
[48] Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruning for deep neural networks. In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam, editors, Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pages 31.1-31.12. BMVA Press, 2015.
[49] Xue-Lian Sun, Zhen-Hua Chen, Xize Guo, Jingjing Wang, Mengmeng Ge, Samuel Zheng Hao Wong, Ting Wang, Si Li, Mingze Yao, Laura A Johnston, et al. Stem cell competition driven by the axin2-p53 axis controls brain size during murine development. Developmental Cell, 58(9):744-759, 2023.
[50] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficien transfer learning for lanin-aan tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[51] Johan AK Suykens and Joos Vandewalle. Chaos control using least-squares support vector machines. International journal of circuit theory and applications, 27(6):005-615, 1999.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307 0009888, 2023.
[53] Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7725-7735, 2023.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 December 4-9, 2017, Long Beach, CA, USA, pages 6008, 2017.
14

---

[55] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[56] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable fine- tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 95 14-9528, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
[57] Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6071-6077
IEEE Computer Society, 2017.
[58] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104-12113, 2022.
[59] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study representation learning with the visual task adaptation benchmark. ArXiv preprint, abs/1910.04867, 2019.
[60] Qizhe Zhang, Bocheng Zou, Ruichuan An, Jiaming Liu, and Shanghang Zhang. Mosa: Mixture of sparse adapters for visual efficient tuning. ArXiv preprint, abs/2312.02923, 2023.
[61] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. ArXiv preprint, abs/21 1.09300, 2021.
[62] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. ArXiv preprint, (4672, 2022.
[63] Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang, Gradient-based parameter selection for efficient fine-tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 18566-28577, IEEE, 2024.
[64] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schützze Masking an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2020. Association for Computational Linguistics.
Online,
15

---

Caltech
Sun397
Pets
Camelyon
Resisc Retionpathy
CLEVR
DMLab
dSpr-ori
Natural Dataset Group
Specialized Dataset Group
Structured Dataset Group
Figure A6: Samples of different downstream tasks in VTAB-IK.
A More Details of Experimental Setup
A.1 More Details of Contenders.
Full: fully tunes all the model parameters (including backbone and classification head).
Linear: freezes all the backbone parameters and only tumes the linear classification head.
Bias
S
freezes all model parameters except for the bias and the linear classification head.
Partial-1: freezes the backbone except for the last layer and also tunes the classification head as described [30].
MLP-3: freezes the backbone and tumes the classification head implemented by a trainable 3-layer multi-layer percepton as described in [30].
VPT-Shallow [30]: freezes all the backbone parameters while introducing additional trainable prompts to the imput space of the pretrained ViT.
VPT-Deep 33) freezes the backbone while appending additional trainable prompts to the sequence in the multi-head self-attention layer of each VIT block.
Adapter-r 226 freezes all the backbone parameters while adding down projection, ReLU non-linearity, and an up projection layer sequentially in the ferw-fo network (FFN) of each visual Transformer block. We report the performance implemented by [22] for comparison.
27 freezes all the backbone parameters while adding a concurrent branch including two low-rank matrices to weight matrices the multi-head self-attention layers to approximate efficiently updating them. The low-rank matrices can be merged into the backbone weights after fine-tuning. We report the performance implemented by [2 for comparison.
Lora-r
SPT 22 identifies the tunable parameters for a given task in data-dependent way, and utilizes LoRA (SPT-LoRA) Adapter (SPT-Adapter) for weight matrices with a large number of tunable parameters and sparse tuning for weight matrices with small number of tunable parameters.
VQT 53
introduces handful of learnable query tokens each layer for adaptation.
DoRA 36): decomposes the pre-trained weight into two components, i.e., magnitude and direction, for line-tuning. It specifically employs LoRA for directional updates to efficiently minimize the number of trainable parameters.
GPS (63) identifies task-dependent tunable weights and applies sparse tuning these weights.
A.2 Dataset Samples for the Downstream Tasks
We visualize some sampled images from different downstream tasks of VTAB-IK 1k (59) in Figure A6 The VTAB-IK benchmark encompasses a diverse range of tasks, including natural images, remote sensing, medical images, etc. Notably, our SNELL has achieved state-of-the-ae (SOTA) performance on these datasets, demonstrating its general effectiveness.
A.3 Implementation details of Figure a)
Given a random matrix W(st) E T we fit this matrix by merging two low-rank learnable matrices E R E Raxr with different kernel functions K.
min
w (gt)
<(B. A)
(A7)
16

---

Table A6: Expression of kemne function utilized in the main text.
Kernel Function
Expression
Linear
<(x, x') = x x
P
Piecewise Linear
<(x, x') Σ 0yy133/r////////44/// (IAXa///iocoo///XX///maaaaaalll Trp/P///ee4///llll
p=1
Sigmoid
x(x,x') = a(1 + exp( -5x +7 +7
RBF
K( <(x, x') alexp( - B ( +7
Table A7: Top-1 accuracy (%) on VTAB-1k benchmarks using ViT-B/16 backbone pre-trained on ImageNet-21k supervisedly. The best result is in bold, and the second result is underlined.
Natural
Specialized
Structured
Methods
SNORE Azim
Cifar100
Caltech
Flover102
SVHN
Sun397
Camelyon
EuroAAT
Resise45
Retinopathy
Cleve-Coum Count
Clevr-Dist
DMLab
KITTI-Dist
dSpr- Loc
dSpr-Ori
sNOR8 Ele
Meun Ace.
DTD
Pets
Full
68.99
87.7
64.3
97.2
87.4
38.8
86.9
79..7 95.7 84.2
13.9
58.6
41.7
65.5
57.5
46.7
25.7 29.1 66.6
Additional -ased methods
MLP-3
62.3
77.0
70.2
58.1
53,2
VPT-Shallon Shallow
77.7
86.9
62.6
97.5
74.5
51.2
87.3
78.2
92.0
75.6
72.9
50.5
58.6
40.5
67.1
68.7
36.1
20.2
34.1
64.9
VPT-Deep
78.8
90.8
65.8
98.0
78.1
49.6
88.3
81.8
96.1
68.4 83.4
68.5
60.0
46.5
72.8
73.6
47.9
32.9
37.8
69.4
Adapten-8
69.2
68.0
81.9
80.9
78.3
41.6
Adapter-32 32
68.7
92.2
69.8
98.9
84.2
53.0
90.3
83.2
95,4
83.2
74.3
81.9
63.9
48.7
80.6
76.2
47.6
30.8
36.4
71.5
NOAH
69.6
92.7
70.2
99.1
86.1
53.7
90.4
84.4
95.4
83.9
75.8
82.8
68.9
49.9
81.7
81.8
48.3
32.8
44,2
73.2
SPT-Adapter
72.9
72.5
88.8
55.8
86.2
85.5
83.0
68.0
81.2
82.4
51.9
74.1
Reparameterized based methods
Linear
63.4
85.0
63.2
97.0
36.6
51.0
86.3
78.5
87.5
68.6
74.0
34,3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
52.9
Partial-1
66.8
85.9
62.5
97.3
37.6
50.6
85.5
78.6
89.8
72.5
73.3
41.5
34.3
33.9
61.0
31.3
328
16.3
22.4
56.5
Bias
72.8
59,2
59.9
51.4
85.3
78.7
72.9
61.5
55.6
55.9
15.7
25.1
62.0
LoRA-8
67.1
91.4
69.4
98.8
85.3
54,0
90.4
84,9
95.3
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
31.0
44.0
72.3
LoRA-16
68.1
91.4
69.8
99.0
86.4
53.1
90.5
85.1
95.8
84.7
74.2
83.0
66.9
50.4
81
80.2
46.6
32.2
41.1
72.6
SPT-LoRA
73.5
72.5
85.7
85.9
82.0
51.1
74.1
SNELL-8
73.7
92.7
72.4
89.2
55.4
84,9
96.1
86.4
75.2
84.0
68.5
53.5
81.0
49,9
33.9
39.2
74.2
SNELL- 16
74.2
93.4
72.5
99.3
90.2
55.7
91.4
85.7
95.8
86.5
76.3
84.4
68.2
53.0
82.0
82.2
49.6
33.3
40.6
74.4
SNELL-32 32
74.5
93.4
73.1
99.3
91.1
55.9
91.5
85.5
96.1
86.5
76.2
83.4
68.6
52.2
81.3
83.2
50.7
35.9
39.0
74.6
We use gradient descent for 1e5 optimization steps, employing the Adam optimizer with a learning rate of lE -e-4 We fit 10 randomly generated matrices for each kernel function presented in Table A6 and report the average MSE Loss in Figure a).
B Introduction of Utilized Kernel Functions
Kernel Function Definition (positive semi-definite). Consider vector space R", kernel function K: RR x R - called a positive semi-definite kernel on RR"" if
(C)K(x,x)) 20
(A8)
holds for all XI. X,, RT, C,, ER
Given two vectors x, x' E R", we show the utilized kernel functions in Table A6 We introduce additional learnable parameters (the e.g. a for Sigmoid and RBF kernel, ap for piecewise linear kernel) that enable the merged adaptation matrix A AW to accommodate both positive and negative values. The additional parameters select certain elements in the matrix and assign them negative values, without compromising the high-rank property of the merged adaptation matrix A A. We set P = 2 for the piccewise linear kernel.
17

---

Table A8: (a) Performance comparisons on FGVC between SNELL and GPS. (b) Memory usage comparison between SNELL and GPS (batchsize=8) on different pre-trained models.
(b)
Oxford Flowers
Stanford Dogs
Stanford Cars
Memory Usage (Mh)
Method
CUB
NABirds
Mean
Method
ViT-B
VIT-L
VIT-H
GPS
89.9
86.7
99.7
92.2
90.4
91.8
GPS
2428
7522
16119
SNELL-32
89.9
87.0
99.4
92.0
90.5
91.8
SNELL-32
1673
4519
9692
Table A9:
Performance comparisons on VTAB-Ik 1k between SNELL and VQT with ViT-B/16 pre- trained on ImageNet-21K. The best result is in bold.
Method
Natural
Specialized
Structured
Mean Acc.
VQT
72.7
84.5
49.3
68.8
SNELL-8
82.0
85.7
61.6
76.4
SNELL-16
82.4
86.1
61.7
76.7
SNELL-32
82.7
86.1
61.8
76.9
C Additional Experiments
C.1 More Comparisons with Existing Methods
Given that some methods do not provide performance or implementation details on both FGVC and VTAB benchmarks, we present a comparison between SNELL and these methods the appendix rather than in Table I First, we provide comparisons with GPS [63] on the FGVC benchmark in terms of performance and memory usage in Table A8 With comparable performance, SNELL has significant advantage over GPS in terms of memory usage. Then, we compare SNELL with VQT [53 on VTAB Ik dataset in Table A9 SNELL significantly outperforms VQT (76.9% vs. 68.8%).
C.2 Per-task results the VTAB-1k benchmark
We provide the per-tasks results on the VTAB-1k benchmark using ViT-B/16 supervised pre-trained on ImageNer21 in Table A7 Our SNELL has demonstrated superior performance by achieving SOTA performance on 13 downstream tasks. Additionally, SNELL achieves SOTA performance on the mean accuracy across all tasks (74.6% vs. 74.1%), indicating its effectiveness in various domains.
C.3 More ablation studies.
C.3.1 Comparison between Competition- based Sparsification and Pre-defined Weight Mask
To verify the effectiveness of the proposed competition-basee sparsification mechanism, we compare the performance on FGVC datasets between kernelized LoRA (KLoRA-8-Fixed) with pre-defined fixed masks, and SNELL in Table A10 The weight masks are generated by SPT [22]. For fair comparison, we utilize the same data augmentation as SPT. Compared to our dynamic masking strategy, pre-defined fixed masking can hardly identify and adjust the most task-relevant weights in an end-to-end fashion, which leads performance degradation (89.4 vs. 90.3).
Table A10: Performance comparisons FGVC benchmark between kernelized LoRA with fixed weight masks (KLoRA-S-Fixed) and our dynamical masks (SNELL-8).
Method
CUB-200
NABirds
Oxford Flowers
Stanford Dogs
Stanford Cars
Mean
ALoRRA8-Fivee
88.0
82.1
99.0
89.4
88.4
89.4
SNELL-8
89.0
83.9
99.3
90.6
88.6
90.3
18

---

Table A11: Comparisons between LoRA and kernelized LoRA (KLoRA) on VTAB- using ViT- B/16 pre-trained on ImageNet21k lk supervisedly. Better performance for the same rank is in bold.
Method
Natural
Specialized
Structured
Mean Acc.
LoRA-8
80.8
84.9
59.6
75.1
KLoRA-8
80.8
85.5
60.5
75.6
LoRA-16
80.6
85.6
58.5
74.9
KLoRA-16
80.9
85.7
59.7
75.4
LoRA-32
79.4
85.4
57.8
74.2
KLoRA-32
80.8
85,4
59.4
75.2
Table A12: Memory usage comparison between SNELL and LoRA. A Mem. denotes the incremental memory usage of SNELL in comparison to LoRA.
Pre-trained
LoRA-8
SNELL-8
Mem. SNELL-8 Mem.
Model
Mem. (MB)
Mem. (MB)
ViT-B/16
1546
1673
0.076
ViT-L/16
4325
4519
0.043
ViT-H/16
9325
9692
0.038
C.3.2 Comparison between Kernelized LoRA and LoRA
We compare the performance of LoRA and kernelized LoRA the VTAB-1k benchmark, where all weight matrices of the pre-trained models are fine-tuned to ensure a fair comparison. The experimental results presented in Table L1T Through experiments with different ranks, observed that kernelized consistently outperforms LoRA across various task groups The replacement of the inner product with nonlinear kernel functions leads to stronger expressive ability, which in turn contributes to improved performance on downstream tasks.
C.3.3 Additional Memory Usage from Nonlinear Kernel Functions
In Figure 3 a), we observe that SNELL requires additional memory usage compared LoRA due to the incorporation of nonlinear kernel functions. To explore whether the impact of this additional usage hinders the usability SNELL on large models, we compare the memory usage between SNELL and LoRA models as the model size grows (in Table ATb)e As the model size expands, the incremental memory usage of SNELL becomes negligible.
C.4 Experiments on Large Language Models
We apply SNELL on LLaMA2-7B 52 to adapt to the commonsense reasoning benchmark. As Table A13 shows, SNELL achieves a better performance than LoRA. This shows the applicability of SNELL to NLP tasks. Many other vision PEFT approaches lack this capability, as they necessitate full level of memory usage for fine-tuning as Figure (a) shows.
C,5 Training Time Analysis
Table provides comparison of training time costs between SNELL and other PEFT methods using NVIDIA GeForce RTX 4090 GPU. The training time of SNELL-8 is slightly higher than LoRA-8 (0.557 vs. 0.443). By further comparing the training time of SNELL-8 and SNELL-8 (saving WW, becomes apparent that the increase time cost primarily stems from the recomputation
Table A13: Performance on commonsense reasoning benchmark with LLaMA-2-7B.
Model
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
LoRA-32
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
SNELIL-32
71.4
82.9
80.7
82.1
80.9
82.6
68.0
80.8
78.7
19

---

Table A14: Training time cost on ViT-B/16 of different PEFT methods
KLoRA-8
SNELL-8
Method
LoRA-8
KLoRA-8
SNELL-8
(saveing WW)
(saving A AW)
Training time (s/img)
0.443
0.522
0.446
0.557
0.448
Cifar
did
Sun397
RetinoPathy
16.0
75.3
73.1
Ascuncc
Accumcy
Accunicy
Accunayy
37.0
14.5
70.8
$6.9
0.75
0.75
0.25
0.50
Spunsity Ratio
Sparsity Ratio
Spursity Ratio
Spursity Ratio
Figure A7: Accuracy of different sparsity ratios using SNELL-8. The pre-trained model is the ConvNeXt-B pre-trained on ImageNet-21k.
of the merged adaptation matrix AW. Conversely, the time cost associated with sparsification (KLoRA-8 SNELL-8) and kemelization (LoRA-8 vs. KLoRA-8 (saving A AW) is relatively small. Indeed, despite the slight increase time cost due to the recomputation, one significant advantage is the significant performance improvement and memory efficiency shown in Figure
D Additional Visualization
D.1 Performance of Different Sparsity Ratio
Figure A7 depicts the accuracy of different sparsity ratios datasets in VTAB-Ik. Different
downstream tasks exhibit diverse preferences for the sparsity ratio. For instance, CIFAR-100 tends to favor smaller sparsity ratio (0.2), while DTD prefers a larger sparsity ratio (0.8). Both Sun397 and Retinopathy tasks also lean towards a larger sparsity ratio (0.99). This highlights the need to consider the specific characteristics of each task when determining the optimal sparsity ratio.
D.2 Analysis of Tunable Parameters
We analyze the tunable weights of SNELL for different downstream tasks. In Figure A8 we compute the number of weights in the weight matrix Wq of self-attention (54) selected by multiple tasks. We find that most of the weights Moreover, we find that in blocks of different depths, there will be a small part of weights that are selected by multiple downstream tasks, which indicates that there exists small number of crucial parameters to improve the model's performance on downstream tasks.
only selected by single downstream task (Tuned Times=1),
Black 0
Block
Block
Black 11
Weight Num
Weigon Nium
Weiggtt
Times
Tuned
Times
Times
Figure A8: Number of weights in Wo selected by multiple tasks with sparsity ratio = 0.99.
20

---

E Discussion
E.1 Tunable Parameter Volume Computing
We justify our choice to omit to report the volume of learnable parameters. First, computing the volume of tunable parameters in SNELL is difficult. the case LoRA, the volume corresponds to the shape of the learnable low-rank matrices. Conversely, for sparse tuning, the volume is determined by the number of updated weights. However, SNELL employs low-rank matrices as learnable parameters and achieves additional updated weight reduction by sparsifying the merged matrices. When using the parameter volume computation method LoRA, caleulating the reduction in parameters due to sparsification becomes challenging. Conversely, applying the parameter volume computation method sparse tuning would be inherently unfair, given that SNELL specifically optimized using low-rank matrices. Second, the parameter efficiency is pathway to achieve high performance and low memory usage rather than an objective for model improvement, because performance improvement and memory usage reduction hold practical value. In our experiments, SNELL has demonstrated its advantages terms high performance and low memory usage, which consider more valuable than the pursuit of fewer learnable parameters.
E.2 Limitation Discussion
Despite achieving state-of-the-art performance with low memory usage, SNELL requires more training time than LoRA. The additional training time cost comes from the recomputation of the merged matrix A W in the backpropagation process presented in Appendix C.5) However, it is crucial to note that this limitation can be solved. Firstly, given the unique characteristies of the kernel matrix, more efficient methods [29] can be employed to calculate the merged adaptation matrix A W. Secondly, by designing appropriate GPU operators, it is possible to avoid explicitly calculating AW [45 during the fine-tuning process like LoRA and reparameterize the learnable low-rank matrices into the pre-trained weight matrices after the fine-tuning process.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix E.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4.1 Appendix A and the released codes
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Please refer to the released codes
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htc/://bbb///udddd///eeeeeueisss ssiniiiicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper does not report error bars following the practice of previous studies 30 22 53
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Figure 3 and Table A14
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Section 4.1 and Appendix A
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer the released codes
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "b2Tsptcvj8", "reviewer": "Reviewer_CYCL", "summary": "The paper introduces SNELL (Sparse tuning with kerNELized LoRA), a method aimed at reducing memory usage during PEFT of large pre-trained models. SNELL achieves this by decomposing the tunable weight matrices into low-rank matrices and utilizing a competition-based sparsification mechanism, thereby avoiding the storage of large weight matrices and tunable weight indexes. This approach is demonstrated to achieve SOTA performance on multiple visual recognition tasks while maintaining low memory usage, making it viable for large-scale models.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The introduction of kernelized LoRA for sparse tuning is a novel approach that effectively combines the advantages of LoRA's low memory usage with the performance benefits of sparse tuning.\n2. SNELL significantly reduces memory usage compared to traditional fine-tuning and other PEFT methods, which is critical for scaling to larger models.\n3. Extensive experiments demonstrate that SNELL achieves state-of-the-art performance on various benchmarks, including FGVC and VTAB-1k, while maintaining low memory usage.", "weaknesses": "1. While the focus is on visual recognition tasks, it would be beneficial to explore the applicability of SNELL to other domains such as natural language processing. I will expect the authors can further show some results on the large language models.\n2. While LoRA offers benefits such as reduced memory usage, quick task switching, and the ability to serve multiple adapters, SNELL's use of dynamic masking during fine-tuning negates these advantages. I'm curious if the authors could provide results using pre-defined fixed masks instead, which might help retain some of LoRA's beneficial properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments that help us extend the applicability and strengthen the novelty! We will diligently follow your guidance to further improve our work.\n## W1: Additional Results on Large Language Models.\nFollowing your suggestion, we apply SNELL on LLaMA2-7B to adapt to the commonsense reasoning benchmark. \n\n**Experiments**: We compare SNELL with LoRA and find that SNELL achieves a better performance. This shows the applicability of SNELL to NLP tasks. Many other vision PEFT approaches lack this capability, as they necessitate a full level of memory usage for fine-tuning as Figure 3(a) shows.\n\n| Model    | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average |\n| -------- | ----- | ---- | ---- | --------- | ---------- | ----- | ----- | ---- | ------- |\n| LoRA-32  | 69.8  | 79.9 | 79.5 | **83.6**      | **82.6**       | 79.8  | 64.7  | 81.0 | 77.6    |\n| SNELL-32 | **71.4**  | **82.9** | **80.7** | 82.1      | 80.9       | **82.6**  | **68.0**  | 80.8 | **78.7**    |\n\nIn the revision, we will further explore the applicability of SNELL on different LLMs and benchmarks and update the experiment results.\n\n## W2: SNELL based on Pre-defined Fixed Masks.\nThank you for your suggestion. To address your concern, first, we respectfully clarify that dynamic masking `can preserve the low memory usage and the ability to serve multiple adapters of LoRA`. \n\n- **Memory usage**: SNELL achieves comparable performance with LoRA (Figure 3(b)). \n- **Ability to serve multiple adapters**: Our masking is determined by the learnable low-rank matrices, so it is dynamic during fine-tuning and deterministic after fine-tuning. Therefore, compared with LoRA's reparameterization process, SNELL only adds a fixed mask. Both LoRA and SNELL can facilitate multiple adapters.\n\nSecond, by incorporating pre-defined masks, it is indeed possible to retain LoRA's quick training speed.\n\n1. **Experiments**: We first provide the training time of kernelized LoRA with pre-defined fixed masks (KLoRA-8-Fixed) and SNELL. \n\n    | Method                | K-LoRA-8-Fixed | SNELL-8 |\n    | --------------------- | -------------- | ------- |\n    | Training time (s/img) | 0.629          | 0.657   |\n    \n    Then we compare the performance on FGVC datasets between kernelized LoRA (KLoRA-8-Fixed) with pre-defined fixed masks, and SNELL. The fixed masks are generated by SPT [R1].\n    \n    | Method        | CUB-200 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | Mean |\n    | ------------- | ------- | ------- | -------------- | ------------- | ------------- | ---- |\n    | KLoRA-8-Fixed | 88.0    | 82.1    | 99.0           | 89.4          | 88.4          | 89.4 |\n    | SNELL-8       | **89.0**    | **83.9**    | **99.3**           | **90.6**          | **88.6**          | **90.3** |\n    \n2. **Analysis**: We find that fine-tuning with fixed masks can improve training speed (0.629 vs. 0.657).  \nHowever, compared to our dynamic masking, fixed masking can hardly identify and adjust the most task-relevant weights in an end-to-end fashion, which leads to performance degradation (89.4 vs. 90.3).\n\nNevertheless, we believe that quick task switching is worth studying, and we will keep exploring it in the future. \n\nWe will incorporate the above experiments and analysis in the revision.\n\n[R1] Sensitivity-aware visual parameter-efficient fine-tuning, ICCV'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3sCaV8rOQ", "reviewer": "Reviewer_QyQ5", "summary": "This paper proposes a method called SNELL for achieving sparse tuning of pre-trained models with low memory usage. The authors employ LoRA to reduce the number of learnable parameters in the optimizer and utilize kernel tricks to ensure the merged matrix maintains a high rank. Additionally, during the sparse training phase, the authors introduce a competition-based sparsification mechanism to avoid the storage overhead of weight indices. Experiments on two benchmarks demonstrate that this method achieves leading performance while reducing memory usage.", "questions": "1. During the experiments, was there a performance comparison with sparse tuning applied to the entire pre-trained model?\n2. The paper mentions that the competition-based sparsification mechanism can save memory usage for weight indices. What proportion of the total learnable parameters does this memory usage represent?\n3. In the sparse tuning part, have you tried fixing the sparse matrix during training to evaluate its effect?", "limitations": "The paper mentions that to achieve sparsification, the re-computation of the LoRA merged adaptation matrix ΔW can lead to reduced computational efficiency. Although the authors explain that designing appropriate GPU operators can solve this problem, this may not be feasible in practice or may incur higher costs. If this approach can solve the problem, then the high memory usage issue of sparse tuning itself could also be addressed similarly.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work introduces SNELL, a sparse tuning method based on LoRA, which combines the memory efficiency of LoRA with the performance of sparse tuning.\n2. Experiments conducted on two benchmarks show that SNELL consistently outperforms LoRA and compares memory usage with the baseline. The authors also conduct comprehensive experiments on different pre-training methods, model architectures, and component ablations.\n3. The paper is well-organized and well-written, making it easy to follow.", "weaknesses": "1. The idea of this paper isn't that novel. The overall approach of this work is a combination of existing fine-tuning techniques, LoRA and sparse tuning, with improvements to LoRA relying on existing kernel tricks. The competition-based sparsification mechanism merely sets a soft threshold.\n2. As a work on sparse tuning, it’s really a pity that this paper does not compare with the latest pure sparse tuning work, GPS[1], nor is it mentioned in the related work section. This severely limits the contribution of this work. In fact, comparing with the state-of-the-art sparse tuning work is essential. If the sole aim is to reduce memory usage at the expense of performance, it may not be very meaningful.\n3. As a PEFT method, the paper does not report the number of trainable parameters, citing the difficulty of calculation due to the sparse tuning method. This is unreasonable to me, as other sparse tuning methods (e.g., GPS) have reported this metric. This metric is also an important evaluation standard for PEFT.\n4. Generally, reparameterization-based methods refer to those where the additional modules can be reparameterized into the original model after training, such as LoRA and SSF[2] (which is also not compared with SNELL). In contrast, methods like Bitfit, Partial, and sparse tuning are not in this category and are referred to as specification-based methods in reviews like Delta-Tuning[3].\n\n[1] Zhang Z, Zhang Q, Gao Z, et al. Gradient-based Parameter Selection for Efficient Fine-Tuning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 28566-28577.\n\n[2] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.\n\n[3] Ding N, Qin Y, Yang G, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models[J]. arXiv preprint arXiv:2203.06904, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback, it has been very insightful for our work. Before addressing your questions, we would like to clarify some misunderstandings regarding Weaknesses 1, 3, and Limitation. All tables are presented in the PDF response due to the limitation of character number.\n\n## W1: Novelty of SNELL\nWe want to clarify that our contributions, inspired by the insights on neurodynamic systems and neuroscience, are essentially different from previous methods.\n\n1. **New perspective on LoRA.** Inspired by the neurodynamic system (L75, L173-175), we interpret matrix vectors as coordinates in a dynamical space and endow the space with complex distances to improve expressivity. The complex distance can be measured by kernel functions. Therefore, our contribution is `providing a new perspective for modeling weight matrix during fine-tuning`, far more than just using kernel tricks. In addition, we `introduce nonlinearity to LoRA`, representing a fundamental departure from other linear extensions [R1].\n2. **Novel sparsification** is inspired by the neuron competition phenomenon (L64-66). Instead of a soft threshold function, we propose the ``dynamic threshold``, a method that selects weights based on their ranking and induces weight competition during end-to-end fine-tuning. This distinguishes our approach from others which select weights based on their values and employ a soft threshold as a parameter [R2] or increase using prior rules [R3].\n\nIn the revision, we will detail the theory behind SNELL and clarify our differences from other methods.\n\n## W3: Tunable Parameters Volume\nWe acknowledge the importance of this metric and will try our best to estimate it. Before reporting estimations, we want to stress that the difficulty in computing this metric does not stem from the sparse-tuning but from the ``combination of LoRA and sparse-tuning`` (L644-656). \n\n1. **Difficulty of computing this metric**: In (kernelized) LoRA, this metric counts elements in the two low-rank matrices that are used to recover weight matrices. However, the sparsification in SNELL requires tracing from the recovered matrix to the low-rank matrices and locating the sparsified elements, which is difficult.\n2. **Volume Estimation**: Please see Table R1 in the response pdf for details.\n3. **Memory usage is more demanding** when fine-tuning large models. A PEFT method can require fewer tunable parameters but still more memory usage, such as pure sparse-tuning with larger sparsity compared to SNELL.\n\n## Limitation: Applicability of SNELL and Problems in Pure Sparse-tuning.\n1. **SNELL is applicable in practice**: SNELL only exhibits a very marginal increase compared to LoRA in training time (Table A9), which does not impede its practical application. This is proved by our implementation of SNELL on large models (ViT-L&ViT-H) in Table A8. Appropriate GPU operators can further benefit the SNELL's training speed.\n2. **Pure sparse-tuning CANNOT achieve LoRA-level memory usage by recomputing during back-propagation**:\nThe low-rank matrices in the optimizer allow SNELL to recompute the large merged matrix during the back-propagation without saving them in the optimizer. In contrast, pure sparse-tuning methods without low-rank matrices have to consistently store large weight matrices in the optimizer.\n\n## W2: Comparing with GPS\nThank you for mentioning the important work to us. Regrettably, we did not mention GPS in the submitted draft, partially due to the unavailability of its source code at the submission deadline.\n\nWith the published codes, we now provide fair comparisons with GPS in performance (Table R2 in the response pdf) and memory usage (Table R3 in the response pdf). ``SNELL has performance on par with GPS while requiring lower memory usage``.\n\nIn the revision, we will discuss GPS as the latest SOTA sparse-tuning method and conduct more comparisons with GPS.\n\n## W4: Method Taxonomy\nThis question is interesting and prompts us to discuss the taxonomy of PEFT methods. We acknowledge the lack of a standard taxonomy in the community.\n`We follow the taxonomy of SPT[R4], classifying based on whether using additional modules during inference or not`. This differs from Delta-tuning[3] according to the training mechanism.\nAdditionally, we compare SNELL with SSF[2] (Table R4 in the response pdf). SNELL demonstrates better performance.\n\nIn the revision, we will provide a detailed discussion of this taxonomy and add the comparison results.\n\n## Q1: Sparse-Tuning without KLoRA\nYour question is insightful and has inspired us to explore the relationship between low-rank properties and performance. Please see Table R5 in the response pdf for comparisons between SNELL and pure sparse-tuning. SNELL achieves better performance than pure sparse-tuning.\n\n## Q2: Parameter and Memory Savings of Our Sparsification\nWe want to clarify that our saved memory usage in weight indices does not reduce the tunable parameter volume.\nThese indices are saved as constant float tensors (not parameters) to multiply with gradients during fine-tuning. Compared to the pure sparse-tuning method, which stores an index for each parameter, our sparsification method saves memory proportional to the model's parameter volume. Therefore, the more parameters a model has, the more memory our method saves.\n\nIn the revision, we will discuss the memory-saving effect of our sparsification mechanism.\n\n## Q3: Fine-tuning with fixed sparse matrix\nYour question is very enlightening for us to discover more advantages of our sparsification mechanism. Please see Tables R6 and R7 in the response pdf for details. The performance of our sparsification mechanism surpasses that of fixed sparse matrices.\n\n[R1] Krona, arXiV'22.\n\n[R2] STR, ICML'20.\n\n[R3] ST-3, WACV'23.\n\n[R4] SPT, ICCV'23.\n\n[R5] ETLSRR, AAAI'23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RcnISzXTG2", "reviewer": "Reviewer_hyWE", "summary": "This paper focuses on the sparse tuning task of the PEFT methods, by introducing the kernel function modeling applied into the LoRA two low-rank matrices or adapters, which costs low memory usage and leads to better performances. To achieve this target, a Kernelization for LoRA is proposed to map two low-rank matrices dot-production from a lower rank r to a implicit higher rank d, such that leading to stronger model capacity based on relatively marginal memory cost. Further, a soft mask adaptive strategy is used to implement the sparsification for fine-tuning so as to reduce the memory usage. Extensive experiments are presented to validate both the memory efficacy and performances improvements.", "questions": "1. Among Line47-48, I am confused about fine-tuning implementation that storing only the tunable weight into the optimizer using such as Pytorch is still practical. Hence, we may not need to put all the pretrained and frozen weights into the specific optimizer during the training phase. Can the authors make a clear explanation about this part?\n\nThe other questions please refer to the weaknesses above.", "limitations": "Overall, the presentation of this paper and the extensive experiments of this paper sounds promising. However, I still have the confusions above in terms of the motivation or the actual contribution of the Sparsification part.\n\nI will keep watching the responses from the authors to evaluate if my concerns are well addressed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Mapping the low-rank matrices dot-production into an implicit space using conventional kernel function sounds like a promising direction, since some typical machine learning methods help with the representation ability of the model while reducing marginally small computation cost.\n\n2. The experimental results are extensive and the ablative studies showcase convincing result.\n\n3. The performances look superior to most of previous works, including both the adapter methods and weights decomposition methods.\n\n4. The presentation and writing of this paper are easy to follow.", "weaknesses": "1. The Competition-based Sparsification Mechanism is kind of overclaim by this paper, because the primary idea of this soft-thresholding is widely utilized to re-activate or de-activate some neurons or bits so that leading to sparsification. But competition-based is not really demonstrated on this point, neither showing competition among each bits of the mask nor clear explanation of the ∆W computation. Can you make a better clarity?\n\n2.  For the memory usage comparisons among LoRA, Adapter and SNELL, it seems SNELL displays little advantages over these works. I suppose that the improvements comes from the kernel function modeling as this implementation enable the model with stronger capacity to adapt for downstream task, such as the non-linear modeling ability, etc. Can the author support more comprehensive explanation and experimental comparisons?\n\n3. Some mistaken typos are presented in the paper writing. In Line6, 'accompanied by increases in' should be 'accompanied by increasing in'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments that help us provide better clarification and explanation of our work! We hope the following responses can address your concerns.\n\n## W1: Competition-based Sparsification Mechanism Clarification\n\nThank you for your feedback. We now provide a more detailed explanation of the competition in sparsification regarding intuition and computation.\n\n1. The **Primary idea** of the \"competition\" is the ``dynamic threshold`` rather than a soft-threshold function.\nSpecifically, we utilize the quantile of values in $\\Delta W$ as a dynamic threshold to control sparsity. This ensures that only a fixed proportion of weights remain non-zero. Therefore, the weights have to \"compete\" with each other to be selected instead of just having a larger value than a threshold.\n\n2. **Competition in the computation**. In the forward computation of $\\Delta W$, weights are zeroed out based on their ranks in the competition. However, the competition actually happens in backward computation where weights are updated based on gradient descent. Weights that contribute more to the task will gain more significant value and thus win the competition.\n\nIn the revision, we will incorporate the above clarifications into Section 3.2.\n\n## W2: Comparison between SNELL and LoRA\nThank you for your valuable suggestions that help us better express our novelty. \n\nCompared to LoRA, our improvement is a significant increase in performance.\nWe provide comparisons between SNELL and LoRA in terms of performance and memory usage.\n\n1. **Our Performance Improvement** over LoRA primarily arises from the combination of kernelized LoRA and the sparsification mechanism.    \n    - LoRA with kernel functions achieves performance improvement (KLoRA in Table A8).\n    - LoRA with sparse tuning shows performance degradation (Table 4(a)).\n    - SNELL with both kernel functions and sparse tuning achieves better performance than LoRA (Tables 1, 2, and 3) and KLoRA (Table 4(b)). This is because the higher ranks of the weight matrix in KLoRA better support the sparsification process (L166-174).\n\n2. **Comparable Memory Usage.** For memory reduction, SNELL shares the same mechanism as LoRA, which only stores the small low-rank matrices as learnable parameters (in Figure 3(c) and L275-278). The little memory increase of SNELL compared to LoRA is because of the nonlinear kernel functions. We provide a memory usage comparison of SNELL and LoRA. The impact of the kernel function on memory usage becomes increasingly negligible as the model size grows.\n\n    | Pre-trained Model | Mem. LoRA | Mem. SNELL | Mem. Delta / Mem. LoRA |\n    | ----------------- | --------- | ---------- | ----------------------- |\n    | ViT-B/16          | 1546      | 1673       | 0.082                   |\n    | ViT-L/16          | 4325      | 4519       | 0.045                   |\n    | ViT-H/14          | 9325      | 9692       | 0.039                   |\n\nIn the revision, we will incorporate the above explanation and comparison in the experiment section.\n\n\n## W3: Typos\n\nThank you for your advice! We will change Line 6 from \"by increases\" to \"by increasing\" in the revision.\n\n## Q1: High Memory Usage of Previous Sparse Tuning Methods\n\nThank you for your careful reading of our paper, we will further clarify L47-48.\n\n1. Indeed, Pytorch supports storing only learnable parameters in the optimizer, but these learnable parameters must be stored in the form of structured matrices.\n2. Unfortunately, sparse tuning involves selecting a variable number of parameters at different locations in the matrix, which is unstructured and makes it impractical to store only the tunable parameters in the optimizer. \n3. Thus, current methods have to treat the entire weight matrix as a learnable matrix and incorporate a mask into the gradient during backpropagation. This strategy solely implements sparse updates of the matrix and is not superior to dense updates in terms of memory usage.\n\nIn the revision, we will emphasize the relation between the unstructured nature of sparse tuning and its high memory usage.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSeSwBU5pL", "reviewer": "Reviewer_7e64", "summary": "This paper proposes a method called SNELL for vision model fine-tuning. It extends matrix decomposition from a kernel perspective and designs a novel sparsification mechanism for end-to-end parameter selection. Experimental results show that the proposed SNELL achieves low memory requirements and high performance on downstream tasks simultaneously.", "questions": "Please check detailed comments in the Weaknesses part.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\t The proposed method is novel and can achieve low memory requirements and high performance concurrently, which is valuable for practical applications. This is particularly noteworthy as many existing methods primarily focus on reducing tunable parameter volumes rather than memory usage.\n2.\t The proposed method can perform the data-dependent parameter selection and tuning in low-resource scenarios. I think this approach has the potential to advance the study of other areas, for example, model editing.\n3.\t This paper conducts extensive experiments, especially on large-scale vision models including ViT-L and ViT-H, demonstrating the high performance and memory efficiency of the proposed SNELL.\n4.\t This paper extends LoRA with a kernel perspective for sparse tuning. The kernel perspective contributes to new insights into LoRA and promotes its further improvements.\n5.\t This paper is well-written and easy to follow.", "weaknesses": "1.\t It would be better to introduce the utilized kernel function in the method section for the first time rather than in the experiment section.\n2.\t The authors should mention in the paper that the implementation details of Figure 4(a) are located in Appendix A.3, for readers to easily access the specific details of this figure.\n3.\t For Figure 3(b), authors should explain why the memory savings of SNELL are more significant for larger models.\n4.\t Although the paper includes comprehensive experiments, I still want to know the comparison between the proposed SNELL and other visual PEFT methods, such as MOSA [1] and VQT [2].\n\n\n[1] Zhang et. al., “MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning”, Arxiv:2312.02923.\n\n[2] Tu et. al., “Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning”, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the detailed comments! We will diligently follow your guidance to further improve our work and manuscripts.\n\n## W1: The utilized kernel function.\n\nThank you for your suggestion! We will introduce the utilized kernel function in Section 3.2 in the revised paper. \n\n## W2: Implementation Details of Figure 4(a).\n\nThank you for your advice! We will introduce the location of implementation details in the caption of Figure 4(a) in the revised paper.\n\n## W3: Why memory savings of SNELL are more significant for larger models?\n\n1. **The memory saving mechanism**: As Fig. 3(c) shows, the memory savings of SNELL are attributed to the reduction of learnable parameters stored in the optimizer. For a matrix $\\mathbf{W}\\in \\mathbb{R}^{n\\times n}$, the memory usage of SNELL-$r$ only takes a proportion of $\\frac{2}{n}r$ to the full fine-tuning.\n2. **Memory significance for large models**: As the model size increases, the value of $n$ and the number of weight matrices increase, making the parameter reduction becoming increasingly significant.\n\nIn the revision, We will introduce the memory-saving mechanism of SNELL in the Appendix.\n\n## W4: Comparison with MoSA and VQT.\n\nThank you for mentioning the two methods. We follow your suggestion and further conduct comparisons with MoSA and VQT on the VTAB-1K dataset.\n\n| Method   | Natural | Specialized | Structured | Mean Acc. |\n| -------- | ------- | ----------- | ---------- | --------- |\n| MoSA     | 79.9    | 84.0        | 50.3       | 71.4      |\n| VQT      | 72.7    | 84.5        | 49.3       | 68.8      |\n| SNELL-8  | 82.0    | 85.7        | 61.6       | 76.4      |\n| SNELL-16 | 82.4    | **86.1**        | 61.7       | 76.7      |\n| SNELL-32 | **82.7**    | **86.1**        | **61.8**       | **76.9**      |\n\nThe results demonstrate the superiority of our SNELL over these methods, and we will add these comparisons in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their diligent efforts and valuable suggestions, which have greatly contributed to improving the quality of our manuscript.

**Summary of strengths**:

We sincerely appreciate that you find our method:

- novel and promising (reviewers CYCL, hyWE, and 7e64);

- provides extensive evaluations and convincing results (reviewers QyQ5, hyWE, and 7e64);

- achieves state-of-the-art performance (reviewers CYCL, hyWE, and 7e64);

- exhibits low memory usage (reviewers CYCL, QyQ5, and 7e64);

- scalable to large models (reviewers CYCL and 7e64);

- presents comprehensive ablations (reviewers QyQ5, hyWE);

- well-organized and well-written (reviewers QyQ5, hyWE, and 7e64).

Meta-review: The paper presents SNELL, a novel approach that addresses the memory usage issues associated with sparse tuning in parameter-efficient fine-tuning (PEFT). By introducing kernelized LoRA and a competition-based sparsification mechanism, SNELL effectively combines the advantages of low memory usage and high performance. This innovation is significant for scaling PEFT to larger models. While there are some concerns about the clarity of the competition-based sparsification mechanism and the limited memory advantages over existing methods, the authors have sufficiently addressed the raised concerns, and the overall contributions of SNELL are strong. The approach shows state-of-the-art performance across multiple tasks with reduced memory usage, making it a valuable addition to the field. Therefore, I recommend accepting this paper, with suggestions to improve clarity and provide more detailed comparisons in future work.
Decision: Accept (poster)