Sensitivity-aware Visual Parameter-efficient Tuning
Authors: Haoyu He
Jianfei Cai
Jing Zhang
Dacheng Tao
Bohan Zhuang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Visual Parameter-efficient Tuning
Fine-tuning
Visual Task Adaptation

Abstract: Visual Parameter-efficient Tuning (VPT) has become a powerful alternative for full fine-tuning, which only updates a small number of parameters while freezing the remaining vast majority of parameters to significantly reduce the storage costs for adapting the pre-trained vision models to downstream tasks. Although the storage burden is largely alleviated, VPT approaches still face many challenges, e.g., lower inference speed and lacking effective configurations for trainable parameters tailored for each task. In this paper, we present a simple yet effective approach termed Sensitivity-aware visual Parameter-efficient Tuning (SPT) to tackle these challenges. Given a desired tunable parameter budget, SPT quickly identifies the important parameters to the given task in a data-dependent way before fine-tuning, without the complex selection schedule. To increase the representational capacity at a negligible cost within the same parameter budget, we employ low-rank reparameterization to achieve a better trade-off between parameter efficiency and accuracy. Through extensive experiments on a wide range of downstream recognition tasks, our SPT achieves better overall transfer performance than the full fine-tuning and the other VPT approaches, with no additional computational or memory overhead during inference. For instance, SPT saves 99.35% of the trainable parameters than the full fine-tuning while achieving a 7.3% higher average top-1 accuracy on VTAB-1k benchmark with the supervised pre-trained ViT-B backbone. Notably, SPT is also the first work that bridges the gap between full fine-tuning and VPT approaches for backbones under self-supervised pre-training strategies MAE and MoCo v3 on the challenging VTAB-1k benchmark.

Under review as a conference paper at ICLR 2023
SENSITIVITY-WWARE VISUAL PARAMETER-EFHICIEN TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Visual Parameter-efficient Tuning (VPT) has become a powerful alternative for full fine-tuning, which only updates small number of parameters while freezing the remaining vast majority of parameters to significantly reduce the storage costs for adapting the pre-trained vision models downstream tasks. Although the storage burden is largely alleviated, VPT approaches still face many challenges, e.g., lower inference speed and lacking effective configurations for trainable pa- rameters tailored for each task. In this paper, we present simple yet effective ap- proach termed Sensitivity-aware visual Parameter-efficient Tuning (SPT) to tackle these challenges. Given a desired tunable parameter budget, SPT quickly identi- fies the important parameters to the given task in a data-dependent way before fine-tuning, without the complex selection schedule. Then, SPT adaptively deter- mines the tuning granularity for each weight matrix. Accordingly, for the whole model, we structurally tune the entire sensitive weight matrices that contain large proportion of sensitive parameters (structured tuning), and non-structurally tune the sensitive connections in the insensitive weight matrices (unstructured tun- ing), simultaneously. For structured tuning, SPT approximates the update with the tow-rank reparameterization to preserve the parameter budget. Therefore, our SPT has high flexibility and representational capability while achieving fa- vorable trade-off between parameter-effiiinncy and accuracy. Through extensive experiments on wide range of downstream recognition tasks, our SPT achieves better overall transfer performance than the full fine-tuning and the other VPT approaches, with additional computational or memory overhead during infer- ence. For instance, SPT saves 99.35% of the trainable parameters than the full fine-tuning while achieving 7.3% higher average top-1 accuracy VTAB-1k benchmark with the supervised pre-trained ViT-B backbone. Notably, SPT is also the first work that bridges the gap between full fine-tuning and VPT approaches with backbones under self-supervised pre-training strategies MAE and MoCo v3 on the challenging VTAB-I 1k benchmark.
INTRODUCTION
The pre-training and fine-tuning paradigm has underpinned the most recent breakthroughs in vision, yielding stunning empirical performance on a series of tasks such as segmentation (Chen et al., 2017; Ronneberger et al., 2015) and detection (He al., 2017; Carion et al., 2020). Transformer (Vaswani et al., 2017) has been widely adopted as the standard architecture for pre-trained vision models, with representatives including CLIP (Radford et al., 2021), MAE (He al., 2022b), BETT (Bao al., 2022), etc. To effectively adapt the pre-trained representations to the downstream tasks, the de-facto choice is fine-tuning, which initializes the model with the pre-trained weights and tunes all the pa- rameters. However, vanilla fine-tuning needs to store separate instance of parameters for each task and each deployment scenario. can be extremely storage-intensirg as the storage cost grows linearly with the number of possible cases, considering there are vast varieties of downstream tasks and dynamic deployment environments, especially when deploying the large vision models (Doso- vitskiy et al., 2021; Liu et al., 2021; Xu et al., 202 to mobile systems. For example, even storing a single large pre-trained ViT-H (He et al., 2022b) model on a local disk requires at least 2.3GB, while the top-10 U.S. apps required only collectively 2.2GB in May 2021.!

---

Under review as a conference paper at ICLR 2023
Ci Camelyon KITTI-Dist
Pretroined Racklorne
Top-1 Acc.
76.0
*
*
1.1%
2.3%
74.0
Task-specilie Sensifivity
2
72.0
SPT
70.0
Foll Finetaming
Armpter
Unstractonal
Tuning
Sinactored
Tuning
00.0
LORA
Prompt
66.0
BIFR
0
2
6 Black
10
06
0.8
1.0
Task
Tuned/ / Total porams (c)
Figure 1: (a) The block-wise parameter sensitivity with supervised pre-trained ViT-B back- bone (Dosovitskiy et al., 2021) for three sampled tasks from VTAB-Ik (Zhai et al., 2019a). "TPS" denotes our task-specific parameter sensitivity (importance). We show averaged scores over all 800 training samples. The sensitivity of each block varies markedly across different tasks. (b) Our pro- posed Sensitivity-aware visual Parameter-efffcient Tuning (SPT) identifies the task-specific impor- tant positions and adaptively combines unstructured and structured tuning to enjoy both flexibility and high capacity. The blue and red lines represent the frozen and trainable parameters, respectively. (c) Accuracy vs. parameter efficiency with the supervised pre-trained ViT-B backbone. Our SPT has no extra computational overhead during inference, surpasses full fine-tuning by large margins, and performs favorably against other VPT approaches.
Notably, an emerging trend is to replace the full fine-tuning with Visual Parameter-efffiient Tuning (VPT) (Jia et al., 2022; Chen et al., 2022; Zhang et al., 2022), which only tunes a small number of trainable parameters (newly introduced or inherently in the model) to cooperate with a frozen backbone that is shared by multiple tasks. As VPT approaches exhibit less than 1% of the trainable parameters, the storage burden is largely alleviated. Another attractive property of VPT is that tuning fewer parameters cases the optimization difficulty and mitigates the overfitting issue for the large models, thereby achieving comparable or even better performance than fine-tuning (Jia et al., 2022) (see Figure (c)). Although promising, the existing VPT approaches suffer from two major issues. First, they specify the positions to add the trainable parameters with different heuristics, and the importance of these positions has not been well studied. For instance, Prompt tuning (Jia et al., 2022) and Adapter (Houlsby et al., 2019) add trainable parameters to the input space and each Transformer (Vaswani al., 2017) block, respectively. Moreover, these approaches keep the same configuration for the trainable parameters across different downstream tasks, neglecting their domain gaps and characteristics. Second, the additional parameters lead to a negligigee sacrifice on the inference efficiency in terms of speed and memory consumption. Taking Prompt tuning (Jia et al., 2022) as an example, with the enlarged input space (200 prompts), it exhibits 2x slower inference speed and consumes 2 of the GPU memory than the full fine-tuning counterpart.
To this end, in this work, we present a novel Sensitivity aware visual Parameter-effiient Tuning (SPT) that identifies and tunes the parameters at task-specific important positions while being inference-efficent. Based on the assumption that not all pre-trained parameters contribute equally to the performance across different tasks, first propose new criterion to efficiently measure the sensitivity (importance) of the pre-trained backbone parameters to a specific task for our SPT. Inspired by model pruning methods (Srivastava et al., 2015; Molchanov et al., 2019), propose to use loss reduction for the sensitivity measurement, which can be efficiently approximated with a first-order Taylor expansion. The resulting parameter sensitivity is solely computed from the gradi- ents, and therefore it can be quickly derived ahead of fine-tuning, We show an example of parameter sensitivities using a pre-trained ViT-B (Dosovitskiy et al., 2021) backbone in Figure (a), where the sensitivities vary across different tasks.
Next, an intuitive solution to only tune the parameters with the highest sensitivity, which we name as unstructured tuning following (Han et al., 2015; 2016). Despite its simplicity and flexibility, unstructured tuning still lacks representational capability as only a few parameters are tuned to cap- ture the domain gap. To this end, our SPT further incorporates unstructured tuning with structured tuning (Figure (b)). Specifically, after identifying the sensitive parameters of the pre-traines back- bone, SPT adaptively determines the tuning granularity for each weight matrix. Accordingly, for the whole model, we structurally tune the entire sensitive weight matrices that contain a large propor- tion of sensitive parameters (structured tuming), and non-structurally tune the sensitive connections in the insensitive weight matrices (unstructured tuning), simultancously. To preserve the parameter budget, for structured tuning, SPT follows the efficient reparameterization strategy of LoRA (Hu
2

---

Under review as a conference paper at ICLR 2023
et al., 2022) to optimize the low-rank decomposition. Therefore, our SPT merits high flexibility and representational capability from both tuning granularities. After fine-tuming fine-tuning, our SPT only needs to store subset of task-specific weights and indexes, which can be merged into the backbone, thereby being parameter-efficient while having no extra cost during inference.
This paper has the following key contributions. 1) We introduce sensitivity criterion measure the importance of the pre-trained backbone parameters, which is fast, effective, and can be applied to backbones with various pre-training strategies. 2) Based on the sensitivity criterion, we propose a parameter-efficient tuning approach to tune parameters at task-specific important positions, which includes not only unstructured tuning but also structured tuning to achieve high flexibility, large ca- pacity, and favorable tradeoff between parameter- parameter-efficieney and accuracy. 3) Extensive experiments on a total of 24 downstream recognition tasks with vision Transformer backbones under supervised, MAE (He et al., 2022b), and MoCo v3 (Chen et al., 2021) pre-trainings show that our SPT achieves the overall best performance, outperforming vanilla fine-tuning and the other SOTA VPT methods by clear margins (Figure I (c)). Moreover, to the best of our knowledge, SPT for the first time bridges the gap between full fine-tuning and VPT approaches on VTAB-IK under MAE and MoCo v3 self-supervised pre- -re-trainings
2 RELATED WORK
The full fine-tuning is the most predominant approach when adapting a large-scale pre-trained model to downstream tasks, where the model is initialized from the pre-trained weights with all parameters trainable. Yet, when a model becomes larger, parameter-efficient tuning (Lester et al., 2021; Li & Liang, 2021) is highly desirable, which transfers pre-trained model to the downstream tasks by tuning only a tiny portion of parameters to alleviate the storage burden. The general parameter- efficient tuning approaches can be categorized into addition based methods (Jia et al., 2022; Houlsby et al., 2019; He et al., 2022a; Chen et al., 2022) and reparameterization based methods (Zaken et al., 2022; Hu al., 2022; Xu et al., 2021a; Guo et al., 2021; Xu et al., 2021a).
Addition-based methods attach additional trainable parameters to the backbone and only tune these parameters. Apart from Prompt tuning (Jia et al., 2022) and Adapter (Houlsby et al., 2019), recent addition-based methods study connecting or combining existing VPT methods. For instance, He al. (2022a) connect Prompt tuning and Adapter and provide a unified view that all VPT approaches share the same design to modify the backbone outputs. Zhang et al. (2022) search for the optimal configurations to combine multiple VPT approaches following once-for-all approaches (Cai al., 2020; Wu et al., 2021). However, the additional parameters require extra computations compared to the full fine-tuning, thereby introducing additional lags during inference.
Reparaanterizaation-be based methods tune the parameters that are inherently in the backbone or new parameters that can be merged into the backbone, thereby yielding no extra computational costs during inference. For epparameterization-aase methods, one line of work tunes the same set of trainable parameters for different tasks, e.g., tuning the bias terms (Zaken et al., 2022) or the last several layers (Yosinski et al., 2014; Caelles et al., 2017). The representative work LoRA (Hu et al., 2022) optimizes two low-rank matrices to approximate the update of each weight matrix in the self-attention modules, which can be merged into the pre-trained weights during inference. Another line of work explores task-specific trainable parameters by jointly optimizing the model parameters and tuning configurations (Guo et al., 2021: Zhao et al., 2020). For example, Guo et al. (2021) seek task-specific trainable parameters by optimizing sparse mask with Lo norm and Zhao et al. (2020) optimize binary masks identify trainable parameters. However, learning the extra masks to identify the tunable parameters requires a formidable amount of training time and GPU memory, which makes these methods hard to be applied to large vision models (Liu et al., 2021; Xu et al., 20. Our method also belongs to the reparameterivaaion-baan existing ones in two aspects. On one hand, we follow importance estimation approaches (Molchanov et al., 2019; Lee et al., 2019) and adopt an effective sensitivity criterion to quickly identify the task- specific trainable parameters before fine-tuning. On the other hand, we incorporate both structured and unstructured tuning granularities to enable higher flexibility and representational power than the sole unstructured counterpart, offering consistent accuracy gains for a fixed parameter budget.
methods but differs from the
Apart from the fine-tuning strategies, the choice of the pre- pre-trained models affects the adaptation per- formance greatly. Recently, models pre- trained with self-supervised strategies (Caron et al., 2021; He et al., 2020; 2022b; Xiao al., 2021) have gained much popularity they better tackle the over-
3

---

Under review as a conference paper at ICLR 2023
fitting issue than supervised pre- pre-trained models (He et al., 2020). Although several works (He al., 2022b; Ericsson et al., 2021) argue better transferability for the self-supervised pre-trained models, their effectiveness is barely studied under the VPT settings. Recently. Jia et al. (2022) show that unlike the supervised pre-trained models, applying the SOTA VPT approaches with representative self-supervised pre-trained strategies, i.e., MAE (He et al., 2022b) and MoCo v3 (Chen et al., 2021), achieves inferior results than the full fine-tuning, AdaptFormer (Chen et al., 2022) achieves higher performance than the vanilla fine-tuning on the video domain with MAE pre-training, while it still falls behind the full fine-tuning in the image domain. In contrast to the previous work, to the best of our knowledge, our SPT for the first time bridges the gaps on the popular VTAB- benchmark under the backbones pre-trained with both MAE and MoCo v3.
METHOD
To introduce our Sensitivity-wwar visual Parameter-efficient Tuning (SPT), we first describe our simple yet effective criterion measure the task-specific sensitivity for pre-trained backbone pa- rameters in Section 3.1. We then introduce how SPT employs the task-specific sensitive parameters to adaptively incorporate both unstructured and structured tuning granularities, while having no ad- ditional computational overhead during inference in Section 3.2.
3.1 TASK-SPECIFIC PARAMETER SENSITIVITY
Based on our assumption that not all parameters contribute equally to the performance across dif- ferent tasks, we first propose new criterion to measure the sensitivity for the parameters in the pre-trained backbone, that is specifically tailored for given task.
Specifically, given the training dataset Dr for the t-th task and the pre-trained model weights w [W1, 22...... [ E R where N is the total number of parameters, the objective for the task is to minimize the empirical risk: minl E(Dr, w). We denote the sensitivity criterion as S = (S,,...,SNy The sensitivity S, for parameter Wn, is measured by the empirical risk difference
when tuning it
Sn E(Dr, w) E(D,, w wn = w;), where wn arrgmin(E(D, w)). We can reparameterize the fine-tuned parameters w W,, +
(1)
Aw. where Au., denotes the change for wn after tuning. However, it is computationally intensive to compute Eq. (1) for two reasons. Firstly, getting the empirical risk for N parameters requires forwarding the entire network N times, which is too time-consuning. Secondly, it is challenging to
derive A A.. as we have to tune each individual wn until convergence.
To overcome the first barrier, we simplify the empirical loss by approximating S, in the vicinity of w by first-order Taylor expansion:
S(1)) = -n.
where the gradient g 0E/Ow and 9n. is the gradient for the n-th element.
To address the second barrier, following (Liu et al., 2019; Cai et al., 2019), we take the one-step unrolled weight as the surrogate for w,, and approximate Aw, in Eq. (2) with single step of gradient descent. We can accordingly get s((") s găe. where is the learning rate. Since e is the same for all parameters, we can eliminate it and finally get s(1) R [gx]. This essentially states that the sensitivity for a parameter is solely measured based on the magnitude of its gradient. This is intuitive since the pre-trained weights with higher sensitivity more likely to affect downstream tasks fine-tuned with SGD. Getting S is also very simple, which can be derived once prior to fine- tuning.
It's noteworthy that it would make only slight differences if we use a higher order Taylor expansion in Eq. (2). Without loss of generality,
example and get si(2) Rs ge - _ g,, Hng, where H, is the n-th row of the Hessian matrix H 8 E E/ōw? e N Since e is small, the first term is much larger than the second one. Thus, we directly use the first-order Taylor expansion in this paper.
take second-order Taylor expansion
In practice, we accumulate S from a total number of C training samples to generate accurate sensi- tivity, where C is a pre-defined hyper-parameter. We derive S ahead of fine-tuning simplify the process as shown in Algorithm 1.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Computing task-specific parameter sensitivities.
Pre-trained model with network parameters w. training set D, for the t-th task, and number of training samples C' used to calculate the parameter sensitivities. Output: Sensitivity set S = .S...... S.).
Input:
Initialize S = {0}%
for i E .I,...,,) C) do
Get the i-th training sample of Dr. Compute loss E.
Compute gradients g. for n E ,I....,NY NN do Update the n-th sensitivity: Sn Sn + 1gul- end for end for
3.2 SENSITIVITY-WWARE VISUAL PARAMETER- TUNING
After obtaining the sensitivity set S and a desired parameter budget т, a straightforward solution is unstructured tuning, which directly tunes the top-7 most sensitive unstructured connections (param- eters). Specifically, we get the top-r sensitive connections to form T from S. For any weight matrix Rd Xdous computed by
W Rd Xdout
we aim to get binary mask M
M 1
W ET W T
0
where Wi and M are the j-th element in W weight matrix can be formulated as W <- W <gw @ M, where Jw is the gradient for W. In this way, only the sensitive parameters are updated while the other parameters are frozen.
and M, respectively. Accordingly, the updated
However, considering VPT approaches generally limit the proportion of train- able parameters to less than 1%, tuning only small number of unstruc- tured connections might not have enough represen- tational capacity handle the datasets with large do- main gaps from the pre- training data. Therefore, we propose to improve the representational capability under the same parameter budget by incorpo- rating unstructured tuning with structured tuning and adaptively determine the tuning granularity for each weight matrix. In structured tuning, we update the entire sensitive weight matrices instead of only some unstructured connections as depicted in Figure 2. Formally, we employ pre-defined threshold hyper--arannn parametr a and the updated weight matrix W' with our SPT can be formulated as:
Task Sensitivity
Task-specific Sensitivity
Weight
Top-r Q0O O0O
I Top-r O 0000
Low-rank Matrices
Matrix
Ooo
o00 ooo 1000
OOO
Structured Tuning
Unstructured Tuning
Unstructured Tuning
Comparison between unstructured tuning and adaptively combining unstructured and structured tuning granularities. The latter structurally tunes the sensitive weight matrices that a large proportion of their parameters are sensitive. The blue and red lines represent the frozen and trainable parameters.
Figure 2:
Xinnn Xdant M 2i=0
w'
W W
if
Xdout
(4)
egw & M
otherwise
where W! denotes the structurally updated W. Eq. (4) essentially states that we structurally tune W when its proportion of the sensitive parameters is higher than a. To implement W one can directly tune all the parameters in W and get W t- -W-- W-rgw- However, directly tuning the entire e.g., weight matrix in the feed-forward layer takes up 2.7%
weight matrix is parameter-consumine of the total parameters for ViT-B (Dosovitskiy et al., 2021). To preserve the parameter budget when tuning the entire sensitive weight matrices, we freeze the pre-trained weights and reparameterize the update with two trainable low-rank weight matrices et al., 2022). Therefore, W.p doot are randomly initialized Iearnable low-rank matrices to approximate the update of W and the rank r is hyper-parameter that & min(din, dout). By letting _ du.ddant don xdont ensure that the total number of trainable parameters does not exceed preserve the parameter
parameter-effiiien way as in LoRA (Hu implement W f- W + Waown Wup- where Wdown E dh and
5

---

Under review as a conference paper at ICLR 2023
budget. In this way, SPT incorporates both structured and unstructured tuning granularities adaptively with our sensitivity criterion to enable higher flexibility and stronger representational power simultancously. leading noticeable gains in parameter-cfficieneyy accuracy tradeoff. We investigate the effectiveness of both tuning methods in Section 4.3.
Discussions. Although our SPT employs LoRA (Hu et al., 2022) when conducting structured tuning. it is fundamentally different from LoRA. First, similar to the other VPT approaches (Jia et al., 2022; Houlsby al., 2019). LoRA neglects the domain gaps across different downstream tasks and adds the low-rank reparameterizations to positions with heuristics. To tackle this fundamental challenge, our SPT estimates the parameter sensitivity and accordingly identifies the task-specific sensitive positions. We show that applying the VPT modules (i.e., visual prompts (Jia et al., 2022), LoRA (Hu et al., 2022), and adapter modules (Houlsby et al., 2019)) to the task-specific instead of heuristic positions achieves solid performance gains (Section 4.3). Second, LoRA employs only structured tuning that updates entire weight matrices. In contrast, under arbitrary parameter constraints, SPT integrates both structured and unstructured tuning granularities, and adaptively determines the tuning granularity for each weight matrix according to their proportion of sensitive parameters. In this way, our SPT can achieve higher flexibility and stronger representational power simultaneously, leading to noticeable gains in parameter-effiienny vs. accuracy trade-off (Section 4.2) and higher scalability than LoRA (Section D).
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets and metries. We evaluate our SPT on total 24 downstream tasks in two groups follow- ing (Jia et al., 2022). 1) FGVC is benchmark for fine-grained visual classification, which includes CUB-200-2011 (Wah et al., 2011), NABirds (Van Horn et al., 2015). Oxford Flowers (Nilsback & Zisserman, 2008), Stanford Cars (Gebru et al., 2017), and Stanford Dogs (Khosla et al., 2011) datasets. Each of the FGVC datasets contains between 55 to 200 classes and a few thousands of images in total for train, validation, and test. We follow (Jia et al., 2022) for their validation splits if the validation set unavailable. 2) VTAB-IK (Zhai et al., 2019b) is a large-scale transfer learning benchmark consisting of a collection of 19 visual classification tasks. VTAB-IK can further be di- vided into three groups, including Natural tasks with natural images, Specialized tasks with images captured by specialized equipments, e.g., medical images, and Structured tasks with images mostly generated from synthetic environments. Each of the VTAB- Ik dataset has only 1,000 training sam- ples, while the test set sizes vary. We use top-1 accuracy (%) averaged within each group our main metric following (Jia et al., 2022).
Pre-trained backbones. We conduct experiments the representative plain vision Transformer backbone ViT-B/16 (Dosovitskiy et al., 2021) that is pre-trained on ImageNet (Krizhevsky et al., 2012) with different pre-training strategies following (Jia et al., 2022), including supervised pre- training, self-supervised pre-training with MAE (He et al., 2022b) and MoCo v3 (Chen et al., 2021).
Contenders. We categorize the baseline methods into two groups which are addition-based and reparameterization-boo based methods, respectively. Unless specified, all baseline methods keep the backbone frozen. Addition-based methods require extra computations during inference, which in- cludes ADAPTER-k (Houlsby et al., 2019), PROMPT-SHALLON (Jia et al., 2022) that adds trainable prompts only to the input space, PROMPT-DEEP (Jia et al., 2022) that adds the prompts to each layer, NOAH (Zhang et al., 2022), and MLP-/ that replaces the classification head with trainable k-layer multi-layer perceptrons as described in (Jia et al., 2022). Reparamelerizzaion--ann no additional computational overhead during inference, which includes LINEAR that only tunes the classification head (details in (Jia et al., 2022)). PARTIAL-K that fine-tunes the last k layers (details in (Jia et al., 2022)). BITFIT (Zaken et al., 2022), and LORA-K (Hu et al., 2022). We refer the readers to Section 2 for details of the baseline methods. We also define FULL as the full fine-tuning.
methods have
4.2 MAIN RESULTS
We first evaluate the effectiveness of our method by comparing SPT with baseline methods. The results with ViT-B under supervised pre- vee-training are presented in Table 1, and those under self- supervised pre-training are presented in Table 2.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison with the other visual parameter-efficien tuning methods FGVC datasets and VTAB-IK datasets using supervised ViT-B/16 pre-trained on ImageNet-21lk (Dosovitskiy et al., 2021). "Tuned/Total" denotes the fraction of the trainable parameters. Top-1 accuracy (%) and inference speed (ms/img) are reported.
ViT-B/16
Total
FGVC
VTAB-Ik Sprcialized
Inference speed
(85.8M)
params
Total
Mon
TToal
Structured
Ace.
FULL
24.02
100%
88.5
100%
75.9
83.4
47.6
69.0
2.8
Addition Aaaed methods
1.35
1.42%
72.8
57.1
2.9
PROMPT-SHALLOW
1.04x
0.31%
84.6
0.13%
76.8
79.7
47.0
67.8
3.7
PROMPT-DEEP
1.18x
0.98%
89.1
1.14%
78.5
82.4
55.0
72.0
3.8
ADAPTER-8
1.06x
0.39%
85.5
0.23%
79.0
84.1
58.5
73.9
2.9
ADAPTER-32
1.19x
0.95%
0.71%
79.6
84.0
2.9
NOAH
0.50%
80.2
84.9
61.3
75.5
3.3
Reparameterization methods
LINEAR
1.02x
0.12%
79.3
0.04%
77.2
26.8
57.6
PARTIAL-1
3.00x
8.38%
82.6
8.30%
69.4
78.5
34.2
60.7
BrTFIT
1.05x
0.13%
88.4
0.13%
73.3
78.3
44.1
65.2
LORA-8
1.07x
0,55%
86.0
0.23%
79.5
84.6
60.5
74,9
2.8
LORA-16
1.18x
0.69%
79.8
84.9
SPT (Ours)
1.13x
0.66%
88.7
0.53%
81.8
85.8
60.8
76.1
SPT (Ours)
1.17x
0.91%
89.2
0.65%
81.8
85.9
61.1
76.3
Table Comparison with the other visual parameter-efficient tuning methods on VTAB-1k datasets using self-supervised ViT-B/16 pre-trained by MAE (He al., 2022b) and MoCo v3 (Chen al., 2021). "Tuned / Total" denotes the fraction of the trainable parameters. Top-1 accuracy (%) is reported.
VIT-R/16
Tital
VTAB-1k MAF Specialadd
VTAB-Ik MoCo v3
(85.8M)
Spectaloud
FULL
38.02x
100%
59.3
79.7
64.3
100%
72.0
84.7
42.0
69.6
Addition- methods
ADAPTHR-8
x
0.23%
57.2
78.4
54:7
63.4
0.23%
27.6
70.9
48.4
49.0
ADAPTER-32
1.28x
0.95%
55.3
78.8
53.3
62.5
0.95%
29.2
73.4
49.8
50.8
PROMPT- SHALLOWW
1.02x
0.12%
40.0
69.7
67.5
59.1
0.12%
67.3
82.3
37.6
62.4
PROMPT-DEEP
1.05x
0.48%
36.0
60.6
26.6
41.1
0.48%
70.3
83,0
42.4
65.2
Reparameterization brase methods
LINEAR
1.02x
0.12%
18.9
52.7
23.7
32.1
0.12%
67.5
81.1
30.3
59.6
PARTIAL-1
4.16x
8.30%
58.4
78.3
47.6
61.5
8.30%
72.3
84.6
47.9
68.3
BITFTT
1.06x
0.13%
54,6
75.7
47.7
59.3
0.13%
72.9
81.1
53.4
69.2
LORA-16
1288x
0.69%
57.3
77.1
59.9
64.8
0.69%
16.0
64.0
48.7
42.9
SPT (Ours)
0.40%
61.6
78.7
65.7
0.39%
83.0
70.4
SPT (Ours)
1.20x
0.56%
61.7
78.7
$7.8
66.1
0.51%
71.7
84.1
58.6
71.5
First, SPT outperforms full fine-tuning by remarkable margins with supervised pre-trained ViT-B. As shown in Table 1, SPT has 7.3% higher top-1 accuracy VTAB-Ik than the full fine- tuning with only 0.65% of the trainable parameters. With abundant data, SPT also outperforms the full fine-tuning by 0.7% mean top-1 accuracy in FGVC datasets while saving 99.09% trainable pa- rameters. Our conjecture is that the full fine-tuning is likely to overfit in our studied scenarios of adapting large pre-trained model overall performance than the SOTA VPT methods with supervised pre-trained ViT-B. For instance, Table 1, SPT with 0.53% proportion of the trainable parameters outperforms the perfforernn addition- based method NOAH with similar amount of tuned parameters by 0.6% mean top-1 1 accuracy on VTAB-IK, without affecting the inference efficiency. SPT also achieves a 1.3% top- accuracy gain with 0.04% less trainable parameters than LORA-16 that has the highest perfor- mance within the reparameterization-boe
small/medium target dataset. Second, SPT achieves better
baselines. We speculate that the superiority of our SPT comes from: !) allocating the trainable parameters to more important task-specific positions with our sensitivity criterion; 2) incorporating both unstructured and structured tuning, which achieves higher flexibility and representational capability. Third, SPT bridges the gap for self-supervised pre- trained backbones between the existing VPT approaches and the full fine- tuning as shown in Table 2. VPT approaches generally exhibit inferior results than full fine-tuning with the self-supervised pre-

---

Under review as a conference paper at ICLR 2023
Maperr
LoRA-16 TPS 75.4
25.0
we strudtured we unstrctured
75.0
Lo8A-16 75.0
LoRA
74.0
Adspice-32 74.0
(MM
20.0
(400
73.0
Adapper 73.9
Promptsderp + TPS 75.1
15.0
330
72.0
71.0
Prongt deep 72.0
2.0
70.0
5.-
68.0
-1.0
66.0
Promp--llyllow
Specisized
Structundd
Tured /Toual Parms
(b)
Figure 3: (a) The domain gap vs. performance for the three dataset groups in VTAB-IK. We measure the performance by the mean top-1 accuracy gain of our SPT against PROMPT-DEEP (Jia et al., 2022). The domain gap is measured by Maximum Mean Discrepancy (MMD) distance (Tzeng al., 2014) between the averaged last-layer features of the pre-training images in ImageNet (Krizhevsky al., 2012) and the fine-tuning images. Employing the structured tuning improves the averaged performance on the Structured datasets significantly.
(b) Results for applying other popular VPT modules to the positions identified by our sensitivity criterion on VTAB-Ik. "TPS" refers to our task-specific parameter sensitivity. Our criterion brings consistent performance gain.
trained backbones MAE and MoCo v3. We observe that our SPT consistently outperforms the full fine-tuning with both pre-training strategies. Especially, SPT outperforms full fine-tuning for non- neglectable margins of 1.8% and 1 1.9% top-1 accuracy on VTAB-1k dataset using only 0.56% and 0.51% trainable parameters for MAE and MoCo pre-trained backbones, respectively. Together with our observation in Sections A and B in appendix that: 1) parameters of pre-trained backbone are with diverse sensitivities to different tasks; 2) self-supervised pre-trained backbones have higher sensitivity variance among the datasets, we conjecture that tailoring the task-specific trainable pa- rameters with SPT can better handle the large variance of the important parameters across tasks and enable accurate adaptation for the self self-supervised pre-trained backbones. We report more multi-seed results of our SPT in Table C.
4.3 ABLATION STUDY
Effect of structured and unstructured tuning. We investigate the effectiveness of combining unstructured tuning with structured tuning described in Section 3.2. The results are presented in Table 3. We observe that under similar fractions of trainable parameters, SPT without unstructured tuning has 0.9% top-1 mean accuracy drop and SPT without structured tuning has a significant 2.4% top-1 mean accuracy drop, suggesting both factors essential for SPT's superior performance while structured tuning contributes more. Further investigating this phenomenon, we find that the main difference lies in the performance for the Structured datasets in VTAB-Ik k shown in Figure (a). Figure 3 (a) also shows that Structured datasets have a larger domain gap to the pre-training dataset than the other datasets (see examples in Section C in the appendix). Our structured tuning updates all channels of weight matrix efficiently with the low-rank reparameterization. which mitigates the large domain gap for the Structured datasets due its enhanced representational power.
Effect of the sensitivity criterion. We investigate the effectiveness of our sensitivity criterion de- scribed in Section 3.1 by employing visual prompts in (Jia et al., 2022), adapter modules (Houlsby et al., 2019), and LoRA modules in (Hu et al., 2022) to the positions identified by our criterion. We present the results Figure 3 (b). It can be seen that our criterion brings 1.1%, 1.1%, and 0.4% performance gains for PROMPT-DEEP, ADAPTER-32, and LORA-16, respectively. The consistent performance gains demonstrate the superiority of our sensitivity to select task task-specific important po- sitions. It is also indicated that criterion has wider applications and can be seamlessly combined with the other VPT approaches.
Effect of the number of training samples C for sensitivity scores. We investigate the effect of the number of training images C (Algorithm 1). We randomly sample the training samples when C/N 1.0 and report the median over three runs. The results shown in Table 4. When varying
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablate the structured tuning and unstructured tuning for SPT VTAB-1k (Zhai al., 2019b). Top-1 accuracy (%) is reported. We set different parameter constraints to align the fractions of the trainable parameters for these cases.
Specialized
SPT
0.65%
81.8
85.9
61.1
76.3
SPT w/o unstructured
0.66%
81.1
85.1
60.1
75.4
SPT w/o structured 0.74%
80.9
84.9
55.8
73.9
Table 5: Cost comparison with full fine-tur tueittnnng PROMPT-DEEP (Jia et al., 2022), and LoRA (Hu et al., 2022). PROMPT-DEEP and SPT are evaluated with around 0.70% fractions of the trainable parameters. We report the latency (ms/img) and the peak memory usage (GB).
Method
Inference I Latency (m</igg)
Inference Memory (GB)
Fine-tuning Memory (GB)
FULL
2.8
1.3
11.9
PROMPT-DEEP
3.8
1.9
13.2
LORA
2.8
1.3
8.2
SPT w/o unstructured
2.8
8.3
SPT
2.8
1.3
12,5
the ratio from 0.3 to 1.0, get similar proportions of trainable parameters and find that half of the training samples suffice getting the highest performance.
Computational cost analysis. We investigate the computa- tional cost of SPT by comparing with the full fine-tuning, the inddition-baeed method PROMPT-DEEP (Jia al., 2022), and LoRA (Hu et al., 2022), which is evaluated on a single GeForce 3090 GPU. The results are presented in Table 5. We observe that PROMPT-DEEP has inevitably higher inference latency and infer- ence GPU memory due to the additional prompts. In contrast, since the updated parameters after fine-tuning can be reparame-
Table 4: Ablate the num- ber of training samples used to get the sensitivity VTAB-IK. Top-1 accuracy (%) is reported.
C/N
0.3
0.5
0.7
1.0
Mran Ave.
76.1
76.3
76.3
76.3
terized and merged into the pre-trained model, our SPT and LoRAare more efficient than PROMPT- DEEP during inference. We observe that our SPT has higher fine-tuning memory than the full fine- tuning and LoRA which is taken up by masking the gradients in Eq. (4). Nevertheless, compared to the pre-training, which takes thousands of TPUv3-core-days (Dosovitskiy et al., 2021), the fine- tuning process for the total 19 datasets the VTAB-IK benchmark only takes several GPU hours and is computationally friendly. Therefore. we argue that slightly more fine-tuning memory (0.6 G higher than the full fine-tuning) is affordable and saving fine-tuming memory is not the main pur- pose of parameter-efficient tuning. Moreover, one employ LoRA the task-specific positions found by our SPT (SPT w/o unstructured tuning) to achieve similar fine-tuning memory as LoRA but higher performance (Figure (b)).
5 CONCLUSION AND FUTURE WORK
In this paper, we have proposed novel visual parameter-efficient tuning approach for the adap- tation of large pre-trained vision models to downstream tasks. Specifically, we have proposed criterion quickly measure the importance of the pre-trained parameters for specific tasks during fine-tuning. Based on the criterion, we have also proposed to tune the task-specific important weight connections for accurate adaptation. To remedy the lack of representational power while achiev- ing favorable tradeoff between parameter efficiency and accuracy, we have proposed to integrate both structured and unstructured tuning, enabling the model to have high flexibility and transfer- ability while being inferencceeffficient. Experimental results have demonstrated the effectiveness of our proposed SPT on a total of 24 downstream tasks. Notably, we have shown that our approach consistently bridges the gap between vanilla fine-tuning and VPT approaches for the backbones pre- trained using MAE and MoCo v3 on VTAB-1k. In the future, we will explore employing SPT to adapt large vision models to more downstream tasks, e.g., segmentation, detection, and pose esti- mation. Another promising direction to further enhance the parameter and inference efficiency by simultancously considering compressing these models. Finally, will explore masking the gra- dients the unstructured tuning our SPT in a hardware- hardware-frienll way to save fine-tuning GPU memory consumption.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICIR, 2022.
Sergi Caelles, Kevis-Koking Maninis, Jordi Pont-Tuset, Laura Leal- Leal-Taixé, Daniel Cremers, and Luc 2017.
Van Gool. One-shot video object segmentation. In CVPR, pp-
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search target task and hardware. In ICLR, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213-229.
Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Liang-Chies Chich Chen, George Papandreou, lasonas Kokkinos. Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, 40(4):834-848, 2017.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurlPS, 2022.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pp. 9640-9649, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. Why do self-supervised models transfer? investigating the impact of invariance downstream tasks. In CVPR, 2021.
Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine-grained car detection for visual census estimation. In AAAI, 2017.
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL-IUCNPP, 2021.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NeurlPS, 28, 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. Eie: Efficient inference engine on compressed deep neural network. Comput. Archit. News, 44(3):243-244, 2016.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick,, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In ICLR, 2022a.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961- 2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, pp. 16000-16009, 2022b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficicet transfer learning for nlp. In ICMI, pp. 2790-2799, 2019.
10

---

Under review as a conference paper at ICLR 2023
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In CVPRW, 2011.
Alex Krizhevsky, Ilya Sutskever. and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. NeurIPS, 25, 2012.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In IJCNLP, pp.
Association for Computational Linguistics, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam, 2018. URL https:
/ /ppeneeeeeww
Pavlo Molchanov, Arun Mallya, Stephen Tyree. Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, pp. 1264-11272, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 722-729. IEFE, 2008
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual 2021.
models from natural language supervision. In ICML, pp.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, pp. 241-241, 2015.
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. NeurlPS, 28, 2015.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Grant Van Hor, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pp. 595-604, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 3998-6008 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uce birds-200-2011 dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology, 2011.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. NeurIPS, 14-22419-224300 2021.
Tete Xiao, Xiaolong Wang. Alexei Efros, and Trevor Darrell. What should not contrastive in contrastive learning. In ICLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Rumxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. In EMNLP, 2021a.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. In NeurlPS, 2021b.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NeurlPS, 27, 2014.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. In ACL, pp. 1-9, 2022.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1eXiv:1910004800 2019a.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04861 2019b.
Yuanhan Zhang. Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206. 2444:22200.46673,, 2022.
arXiv preprint
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking as an efficient alterna- tive to finetuning for pretrained language models. In EMNLP. 2020.
12

---

Under review as a conference paper at ICLR 2023
Appendix
A MORE TASK-SPECIFIC SENSITIVITY PATTERNS
We show more patterns of block-wise task-specific sensitivity for tasks sampled from VTAB- Ik (Zhai al., 2019b) in Figures B, and We observe the similar trend that given pre-trained backbone, the important parameters for different tasks diverse greatly. This aligns with our motiva- tion tune task-ipecific parameters.
B COMPARISON ON SENSITIVITY VARIANCES ACROSS THE PRE-TRAINING STRATEGIES
We further compare the variance of the task-specific sensitivities on VTAB-IK for backbones with different pre- tre-training strategies. We present the results in Figure D. We observe that the backbones with self-supervised pre-training have higher variance sensitivities than the supervised pre- trained ones across the 19 downstream tasks. Especially, the variance for MAE (He et al., 2022b) twice as large as the supervised pre-training strategy. We conjecture that our SPT yields higher performance than the other methods since it can better handle the large variance and tailor suitable trainable parameters for specific tasks.
Mirk2
TEN
MACES
12%
Kodo
Bacce
11%
Moda
1PL
Lhckk
to
Back
Leck
Caltech101
Canselyon
Mock
blckk
23%
2VB
Monk
bock
12%
XX
2N
1/5
Eacco
beck
Figure A: The block-wise parameter sensitivity with VIT-B (Dosovitskiy et al., 2021) backbone under supervised pre-training for six sampled tasks from VTAB-Ik.
C VISUALIZATION FOR DOMAIN GAPS
We visualize some sampled data for ImageNet, Natural tasks, and Structured tasks in Figure show the domain gaps. The domain gap between the pre-training data and the samples from Structured datasets is large as they mostly contain synthetic images. The visualization aligns with finding that structured tasks have large domain gaps in the pre-training data which is handled by our SPT with structured tuning.
13

---

Under review as a conference paper at ICLR 2023
Xed
LNCC4
36dk
Mark
Mock
134
Mank
Ned
PS
2N6
beok
Modet
teck
Cuandyyu
Mahh
APR
Bick
375
Mick
Meck
Eaceo
Bace7
14
EuroAT
Clevs
Figure B: The block-wise parameter sensitivity with VIT-B backbone under MAE pre-training for six sampled tasks from VTAB-IK.
Modk
JUGEY
Modk3
blcck
FIRE
Midi2
Mock
Mick
Mork
14G
125
Medcil
Citur 1o
Cahoch101
Camelyon
Verk
16-44
Makk
ERCE
17
In
1%%
EuroAAT
Figure C: The block-wise parameter sensitivity with VIT-B backbone under MoCo v3 (Chen et al., 2021) pre-training for six sampled tasks from VTAB-1k.
D SCALABILITY COMPARISON WITH LORA
We evaluate the scalability of SPT and compare with LoRA. The results shown in Figure D (b). We observe that LoRA saturates quickly while our SPT has consistent performance gain when the percentage of the trainable parameter increases. We conjecture that SPT finds task-specific
14

---

Under review as a conference paper at ICLR 2023
Variance
Top-1 Acc. (86
77.0
76.0
0.30
75.0
0.000
0.001
SPT
LoRA
0.000
72.0
0.3
0.4
0.5
0.8
0.7
Pre-training strategy
Tuned / Total params (%)
(a)
(b)
Figure D: (a) Comparison the sensitivity variances across backbones with different pre-training strategies on VTAB-1k. (b) Comparison on the scalability between our SPT and LoRA (Hu et al., 2022) VTAB-IK. 1k. LoRA saturates quickly while our SPT has consistent performance gain when the percentage of the trainable parameter increases.
Table A: Per-task results on the VTAB-1k benchmark from Table "Tuned Total" denotes the fraction of the trainable parameters. Top-1 accuracy (%) is reported.
Meani
Spdiiked
Smorind
8.3
10.3
74.
612
110
35.5
13.5
123
212
313
56.1
JAB
813
083
627
113
61.3
12.3
340
134
45%
910
16.
18.5
LANGAR
013
K1.3
KTE
96h
V13
94.5
113
12.6
363
9%
910
750
TARILA
0.899
11.
094
094
655
513
103
540
64.0
151
110
638
0476
6.99
0,0%
0.75
6315
8320
6400
0476
3.76
099
0335
3390
079
6205
0306
3396
3270
6780
0265
0305
sensitive positions and adaptively combines unstructured and structured tuning granularities, thereby better allocating the trainable parameters than LoRA.
E HYPER-PARAMETER AND AUGMENTATIONS.
We follow (Jia et al., 2022) for the hyper-parameter settings on MLP-K, PROMPT-SHALLOO. PROMPT-DEEP, PARTIAL-K, and BITFIT. For our SPT and the other methods, we use the AdamW optimizer (Loshchilov & Hutter, 2018) with cosine learning rate decay, batch size 64, learning rate x 10-3, and weight decay 10-4 following (Zhang et al., 2022). We also follow (Zhang al., 2022) to set the rank r for structured tuning in SPT as 8. For the prpee-ppraaeters specific to our SPT, we set the threshold a = 0.2 in Eq. (4) by grid search on the validation set. The effect of varying a is analyzed in Section 4.3. Given the budget constraint 7 the number of trainable pa- rameters (by default, less than 1%), we search for the optimal number since fewer parameters may have better performance. Therefore, validation set. We search over (0.2M, 0.4M, 0.6M, 0.8M} and (0.2M, 0.4M, 0.6M. 0.8M, 1.0M} for VTAB-1k datasets and FGVC datasets, respectively. The full results and details for the searched number trainable parameters for each task are presented in Table A and the appendix. We set the number of training samples M used calculate our parameter sensitivities to be 800 and 1,600 for VTAB-Ik and FGVC datasets, respectively.
follow (Jia et al., 2022) and conduct grid search the
F EFFECT OF THE THRESHOLD
O.
We investigate the effect the threshold appam-err o (in Eq. (4)). The results are shown in Table D under around 0.70% fractions of the trainable parameters. We see that when varying
15

---

Under review as a conference paper at ICLR 2023
Table B: Per-task results the FGVC benchmark from Table 1. "Tuned Total" denotes the fraction of the trainable parameters. Top-1 accuracy (%) is reported.
CUB-200-2011
Stanford Dogs
Stanford
Addition-ben
MLP-3
1.50%
85,1
77.3
97.9
53.8
79.8
PROMPT- HROMPT-SBLLLOW SHALLOW
0.31%
78.8
98.4
90.7
68.7
84.5
PROMPT-DEEP
0.98%
88.5
84.2
99.0
90.2
83.6
89.1
ADAPTER-8
98.4
85.5
ADAPTER-12
0.95%
87.2
84.3
98.5
89.6
68.4
85.6
Reparaantterization-n methods
LINEAR
0.12%
85.3
75.9
97.9
86.2
51.3
79.3
PARTIAL-1
8.38%
85.6
77.8
98.2
85.5
66.2
82.6
BITHT
0.13%
98.8
91.2
79.4
88
LORA-8
0.55%
84.9
79 (1
98.1
88.
79.8
86.0
72.0
SPT
0.06%
88.5
82.4
98.7
89.8
83.9
88.7
Tumed/ Total (%)
0.71%
0.70%
0.71%
0.47%
1.07%
0.6646
SPT
0.91%
88.5
82.8
99.0
89.8
85.7
89.2
Tuned/ Total (%)
0.71%
1.14%
1.15%
0.47%
1.07%
0.91%
InngeNet
Plowers 102
Pvis (h)
60
Clev-Coout
sNORB-Arth
Figure E: Dataset samples from ImageNet, Natural tasks, and Structured tasks. We can observe that the samples from Natural tasks in VTAB-Ik ((a), (b), and (c)) have clearly smaller domain gaps to the pre-training ImageNet (Krizhevsky et al., 2012) samples than the ones from Structured tasks in VTAB-Ik ((d), (e), and (f)).
Table C: Multi-seed results for SPT with VTT-B/16 under supervised, MAE (He al., 2022b), and MoCo v3 (Chen et al., 2021) pre-training strategies. "Tuned / Total" denotes the fraction of the trainable parameters. The mean and standard deviations of the Top-1 accuracy (%) with three random seeds are reported.
Method
Tuned / Total
Natural
Specialized
Structured
Mean Acc.
SPT, supervised
0.62.10.04
81.77.0..17
85.67.10.06
61,01
76.20
SPT, supervised
0.54.10.04
81.65.1.111
85.8510.007
60.75110.09
76.02-10.07
SPT, MAE
0.54.10.4
62.10.10.29
79.02.10.57
57.87
66.32.10.41
SPT, MAE
0.42-10.02
61.95110.45
78.72-10.50
57.23-10.53
65.97-10.28
SPT, MoCo v3
0.56-10.06
72.26-10.45
84.38-10.25
58.58110.20
71.74-10.22
SPT. MoCo v3
0.48+10.07
71.0110.45
83.51+0.45
56.89-10.4
70.47+0.10
from 0.05 to 0.40, the performance only differs for 0.1-0.2% mean top-1 accuracy. While setting 0.80 has significant performance drop. As higher indicates employing more unstructured tuning, we conjecture that allocating too many trainable parameters to unstructured tuning limits the model's representational capability. As 0.2 achieves the best performance, we consistently use 0.2 for all experiments.
16

---

Under review as a conference paper at ICLR 2023
Table D: Ablate on the threshold a for SPT on VTAB-Ik. Top-1 accuracy (%) is reported.
0.05
0.10
0.20
0.40
0.80
Mean Acc.
76.1
76.2
76.3
76.2
73.9
17


Reviews:
[{"review_id": "pfZMKG2Pph", "reviewer": "Reviewer_hvdx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to perform parameter-efficient adaption for vision tasks, where only a small number of parameters are trained to adapt to new tasks (image classification tasks in this paper). To address this problem, the authors propose to exploit the estimated sensitivity to decide which parameters to be tuned, and the sensitivity is measured according to the gradients. Based on the ratio of sensitive parameters in the weight matrix, the authors use structured or unstructured tuning. In the experiment, the authors show that their proposed SPT can perform better using fewer parameters than the previous parameter-efficient methods in VTAB-1k and FGVC experiments. Additionally, they show some analyses, including using self-supervised pretrained backbones, using or not using structured tuning, and other hyper-parameter tunings.\n\n", "summary_of_the_review": "This paper provides a new parameter-efficient method for visual tasks, and they provide extensive results on different datasets and show the improvements against prior works. Given the current status of the paper, I would recommend the score \"marginally above the acceptance threshold\".", "strength_and_weaknesses": "**Strength**\n- The paper is well written, and most of the proposed components are easy to understand. The figures with a clear explanation help the reader follow the idea. \n\n- The proposed idea is novel and can potentially adapt to different datasets with task-wise numbers of trainable parameters. Applying structured and unstructured tuning in parameter-efficient learning is new to the community to the best of my knowledge. \n\n- The authors show extensive results on different datasets and pretrained models, and the following analyses confirm the effectiveness of the proposed method.\n\n**Weakness**\n- If I understand correctly, one potential weakness is the computation time of two-stage training. The proposed SPT requires an additional stage to estimate the sensitivity of each parameter based on the gradient and decide which parameters are decomposed into low-rank matrices. Then the proposed method tunes these chosen parameters to adapt to new tasks. This might make the computation time longer than simple one-stage training methods. \n\n- For $\\Delta_w$ in equation 6, it is confusing when the authors explain  $\\Delta_w$ means in the following paragraphs. First $(i,j) \\notin P$ is confused (it might mean that these parameters are not in the selected parameters set, but this notation is quite confusing). My guess for what the authors try to express in the second line of Eq. 6 is that the unstructured tuning is applied on these weight matrices where only a few parameters need to be tuned. I would suggest the authors clarify this part.\n\n- I would suggest the authors report the variance of reported results since the parameter-efficient method might be sensitive to the randomness in the training. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jwEM0QwJPap", "reviewer": "Reviewer_Z771", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to first recognize influential visual parameters by its averaged gradients and then tune those parameters. In order to improve adaptation capability, the paper also propose to switch to a low-rank reparameterization of the weight matrix that contains enough sensitive parameters. The paper evaluated on both FGVC and VTAB dataset and achieve even better results compared to fully fine-tuned model due to the small size of the fine-tuned dataset. ", "summary_of_the_review": "The paper propose a pipeline for recognizing and tuning visual parameters. The idea is interesting but my concern is on the clarity part", "strength_and_weaknesses": "Strengths: (1) a simple way to identify sensitive visual parameters.\n(2) a low-rank reparameterization approach to update the visual parameters\n(3) good results on FGVC and VTAB datasets\n\n\nWeakness: (1) The introduction on sensitive visual parameter identification is kind of verbal. The criteria is simply the first order gradient and  it would not be efficient to compute second-order hessian matrix. Introducing eq 2 and 3 would be helpful because you can not find optimal in anyways.\n(2) Missing introduction of how to compute W_up and W_down\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_VAmW6U_SC", "reviewer": "Reviewer_MqQs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper assumes that not all parameters contribute equally to the performance across different tasks. This paper proposes a criterion to measure the sensitivity of parameters, which is the magnitude of the gradient. Reasonable results are reported.", "summary_of_the_review": "I rate it a rejection because the novelty is not satisfying. To me, the proposed method seems yet another LoRA with trivial modifications (i.e., in some cases we should select some parameters and finetune them in an unstructured way).", "strength_and_weaknesses": "Strengths\n\nS1. The idea is simple and easy to follow.\n\nS2. The experimental results are acceptable.\n\nWeaknesses\n\nW1. The writing can be significantly improved. It is difficult to get what is new and what is significant in this paper. For example, after mentioning the criterion (Page 2), the authors commented finetuning only the important parameters (referred to as unstructured tuning) is not enough. The authors proposed to use LoRA (referred to as structured tuning). In other words, the proposed method only works in some cases and should be combined with LoRA. This significantly undermines the proposed method.\n\nW2. The proposed method actually increases memory usage (Table 6), so why is that called efficient finetuning?\n\nW3. As seen in Table 2, the proposed method is not impressively effective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JN-0W8AEzv3", "reviewer": "Reviewer_Sx5t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the efficiency issue with visual parameter-efficient tuning (VPT), such as prompt learning and adapters. In particular, the idea is to tune only those important parameters of a pre-trained model per task, under an assumption that the importance of parameters is task specific. The proposed method is based on identifying the sensitivity of each parameters w.r.t a specific task based on previous model pruning methods. For performance, the whole parameters are still updated subject to a low-rank constraint for parameter efficiency, and further merge them into the backbone after fine-tuning. \n", "summary_of_the_review": "This paper aims to adapt a pertained model to downstream tasks at no additional inference cost. The key idea is to find out what are top sensitive parameters for a given task, on which the update should be focused on. Previous pruning methods can be used to measure such sensitivity, and the final model needs to combine with LoRA according to some threshold. The overall method is some incremental over LoRA. ", "strength_and_weaknesses": "**Strength**\n\n- The problem of VPT efficiency is valid and valuable to be solved\n\n- The idea of measuring the parameter sensitivity using network pruning methods is sensible and also novel in the VPT context. Technically, using the first derivative of Taylor expansion is a good choice with efficiency\n\n- The results are strong in comparison.\n\n**Weakness**\n\n- It is some strange that the proposed method still update the whole parameters when a high proportion of parameters are sensitive to a specific task. \n\n- Method: It is largely similar to LoRA except an introduction of a threshold based choice /branch. SPT w/o unstructured is equivalent to LoRA. So it is very incremental over LoRA. \n\n- Experiment: Table 6 lacks comparison to LoRA.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-Ed5mSs58_U", "reviewer": "Reviewer_y14U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work mainly focuses on the lower inference speed and lacking effective configurations for trainable parameters tailored for each task. It proposed a simple but effective approach named Sensitivity-aware visual Parameter-efficient Tuning (SPT) for these challenges. It quickly identifies the important parameters of the given task in a data-dependent way before fine-tuning without the complex selection schedule. Also, low-rank reparameterization is employed to achieve a better trade-off between efficiency and accuracy. The experiments are conducted on 24 downstream tasks.", "summary_of_the_review": "This paper proposed a novel criterion to efficiently measure the sensitivity (importance) of the pretrained backbone parameters, and proposed a parameter-efficient tuning approach SPT to tune parameters at task-specific important positions.\nHowever, there are a few concerns about if the experiments validate the claimed contributions.", "strength_and_weaknesses": "*[Strength]*\n1. The motivation that focuses on task-driven parameter tuning is contributed.\n2. This work proposed a novel criterion to efficiently measure the sensitivity (importance) of the pretrained backbone parameters to a specific task.\n\n*[Weakness]*\n1. The paper claimed there are challenges about lower inference speed, but in the experiments, there are no related experiments with other SOA methods in Table 1 (only comparisons with the PROMPT-DEEP in Table 6).\n2. The experiments are about the ratio of the tuned parameter to the total parameter, instead of the comparison between the total tuned parameter number. There is a concern about the smaller ratio of tuned to total coming from more total parameters.\n3. Does such a strategy lead to higher training time (fine-tuning time), compared with other SOA in table 1?\n4. In section 3.2, how to get $P$ from $S$?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject