Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
Authors: Dongze Lian
Zhou Daquan
Jiashi Feng
Xinchao Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformer
Efficient
Fine-tuning

Abstract: Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF. 

Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
Dongze Lian Daquan Zhou Jiashi Feng²
Xinchao Wang'
National University Singapore "ByteDance
{dongze, xinchao) Cnus. edu. @choudaquan21jjshfffe (amggg)@mmml..mmmmmm com
Abstract
Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, propose new parameter-eeficient fine-tuning method termed SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine- tuning. In this way, SSF also surprisingly outperforms other parameter-efficien fine-tuning approaches even with smaller number of tunable parameters. Fur- thermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 1 1.48% (73. (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-IK 1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parame- ters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MIPs) and datasets. Results on image classification datasets in total and 3 robustness & -uitt---ist datasets show the effectiveness of SSF. Code is available at /thhtts://// /ttttp://githbb.... com/ waamm/dongeeiian//SS
Introduction
With the popularity of the data-driven methods in the deep learning community, the dataset scale and the model size have both got huge explosions. There is a tendency to explore large models and then adopt these rre-trained models in downstream tasks to achieve better performance and faster convergence, which gradually becomes common way.
However, the current procedure depends on full fine-tuning heavily, where all the parameters of the model are updated. It inevitably causes the model be over-fitted the small target dataset and thus cannot be used for other tasks after the fine-tuning. As result, the device will need to save a dedicated set of model parameters for each task, which causes a huge amount of storage space, especially for today's 's large models (e.g., ViT-G/14 II 1.8G, CoAtNet 5 2.4G).
simple solution for the above problem is linear probing [16], where only the last head layer is fine-tuned. However, this practice usually yields inferior performance compared to the full fine-tuning proxy. Motivated by the success of the parameter-efficie
fine-tuning strategy with prompt in the the recent work implements similar proxy on vision tasks (2)], termed Visual Prompt Tuning (VPT). Specifically, VPT [9] proposes to insert learnable prompts as inputs and append them to the original image tokens. These prompts
field of natural language processing (NIP) 21
32
23
19
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unitol
No
Fire-Graired Vieual Canseficaion Calosets
Visual Tark Aciaptation Denchmork
Acc.
(M)
jenneter qpace
promm.
Full
93.83
55.88
Linear peoblog
88,70
0.08
Adapier 11
93.34
0.31
VPT [30
93
0.54
SSF (ours)
13,99
0.28
Table 1: Characteristics of different fine- tuning methods. Acc. means the Top-1 ac- curacy (%) on CIFAR-100 with a pre-trained VTT-B/16 for tuning. Params. means the learn- able parameters at fine-tuning. Our SSF has unified learnable parameter space and does not require extra inference parameters while obtaining superior performance.
Eno
0.25
0.50
2355 120 (M)
0.5
2.55
05.75
0000
Figure I: Performance comparisons of seven fine- tuning methods with pre-trained ViT-B/16 model on the FGVC dataset and VTAB-Ik benchmark. Our SSF (red dots) achieves state-of-the- per- formance only with about 0.3M average learnable parameters.
will interact with the image tokens by performing self-attention and are updated during the fine-tuning process. In this manner, a significant performance improvement can be achieved in downstream tasks compared to a linear probing proxy. Nevertheless, compared to the full fine-tuning and linear probing, it additionally raises two issues: i) VPT tunes the number of prompts for different tasks, which introduces task-dependent learnable parameter space. The fine-tuning performance is sensitive to the number of prompts for each task and needs to be carefully designed. Too few or too many prompts might either degrade the accuracy of fine-tuning or increase the redundancy of the computation (e.g., 200 prompts Clevr/count Adapter-base based methods [2] 42], introduces additional parameters and computational cost in the inference phase compared to the original pre-trainee model. For instance, VPT introduces additional inputs for self-attention with image tokens. Adapter- methods insert additional modules into the pre-trained model. These methods change the specific backbone architecture the input of the network, which might result in frequent structure modifications and heavy workload, especially for those models that are already deployed in edge devices (e.g., mobile phones).
1 prompt on Flowers102). ii) VPT (29], well as other
To cope with the above issues, we attempt to find a general proxy for parameter-efficient fine- tuning, where the learnable parameter space is unified (fask-independent) and no additional inference parameters are introduced. Inspired by some feature modulation methods [58, 22 45), we propose new parameter-efficient fine-tuning method named SSF, where you only need to Scale and Shift your deep Features extracted by a pre-trained model for fine-tuning. The intuition behind our approach come from the fact that the upstream datasets and downstream datasets have different data distributions [50]. Therefore, is difficult to apply the model weights trained in the upstream dataset to the downstream dataset. For instance, naive linear probing strategy with keeping the weights of backbone frozen will cause performance degradation. To alleviate the above problem, SSF introduces scale parameters and shift parameters, which could be considered variance and mean to modulate the features of the downstream dataset extracted with the pre-trained model on the upstream dataset, such that the modulated feature falls in discriminative space. These scale parameters and shift parameters do not depend on any input and have unified learnable parameter space for different tasks. Another advantage of SSF is that only introduces linear transformations because we scale and shift the extracted features. These linear transformations could be further merged into the original pre-trained weight via model re- pe-arammtteriztiionn [10] in the inference phase, thus avoiding the extra parameters and FLOPs for downstream tasks. For a deployed model in edge devices, only the updated weights after fine-tuning need to be uploaded instead of changing the backbone architecture. [ shows the specific characteristics comparisons between SSF and other fine-tuning methods. SSF simple, effective, and efficient, which also conforms to Occam's Razor principle. Therefore, we explore this new baseline and find that it surprisingly outperforms all other
paramcter-efficicnt tinettunigg methods
We evaluate our method on 26 classification datasets in total and 3 robustness & out-of-distribution datasets. SSF obtains state-of-the-arr performance compared to other parameter-efficient fine-tuning methods with the trainable parameters and accuracy trade-off (Table and Figure TT. Compared to the full fine-tuning our method obtains 2.46% (90.72% vs. 88 (8.5%%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB 1k in terms of Top- accuracy but only with about 0.3M trainable parameters. Furthermore, our SSF does not require additional parameters during the inference phase. It is plug-and-play and is very easy to extend to various model families (CNNs, Transformers, and MIPs). Our SSF establishes a new baseline and we hope that brings more insight into the field of the efficient model tuning.
2

---

2
Related Work
2.1 Model Families
Convolution has been used for a long time as the main module to extract the image features in
computer vision tasks, and CNN-based architectures have been studied 49 18
59
39 60 37 63
with extension on graph-based data [62] 61] 36). Recently, another architecture family, Transformer,
has gained widespread attention owing to its great success in NLP [56]
g 23 Following this
direction, Dosovitskiy et al. [11] first employ a transformer in the domain of computer vision and introduce new architecture paradigm, ViT, which achieves promising results 664
48 Subsequently, models, such as DeiT [53] and Swin Transformer [88 are introduced and
various transformer-base
shown to be effective on a variety of tasks such as object detection, semantic segmentation, action recognition [40], etc. In another line, Tolstikhin et al. (52) propose a pure MLP-based architecture, and subsequent papers 20 33 have interestingly demonstrated that the MLP-based architectures can catch up to transformers. However, in addition to the well-designed modules, their excellent performance is also attributed the deployment of large-scale models. Given a large-scale model pre-trained on a large dataset, how to perform parameter-efficieet fine-tuning in downstream tasks is essential but is currently less explored. In this paper, we propose SSF as a new baseline and show its promising performance with comprehensive validation in a wide variety of tasks.
2.2 Pre-training and Fine-tuning
Early models [8 24 22 59, [T] are usually pre-trained on the ImageNet- IK dataset, and then fine-tuned downstream tasks to achieve faster convergence [17] or better performance. Such procedure is called pre-training and fine-tuning, or transfer learning. Recent works tend to employ larger models (e.g., ViT [ and Swin Transformer V2 (38) and train them larger datasets (e.g., ImageNet-21K and JFT-300M) in pursuit of better performance. Both in the domains of NLP and computer vision, these large models 8 38,
70 achieve enormous performance improvements compared to the small-scale models and provide pre-trained weights for downstream tasks. Some other works attempt explore how to efficiently fine-tune the pre-trained models 13 [1 on the target tasks. For instance, given a target task, SpotTune [I3] investigates which layers need be fine-tuned. Touvron al. [54] find that fine-tuning the weights of the attention layers and freezing weights of the other parts is sufficient to adapt the vision transformers to other downstream tasks. Some works also propose to insert adapters into the network fine-tune in a parameter-efffcient way. These adapters can be a small non-linear network (21), hyper-network that generates model weights 43], or compactor 42) which performs a low-rank decomposition to reduce the parameters. Some works have also tried to only update the bias term 12 66]. More recently, VPT (29 proposes to insert a small number of learnable parameters (prompts) and optimize them while freezing the backbone, which achieves significant performance improvement compared to the full fine-tuning. During the submission of this work, some methods k
47
15
69
68 are also proposed for parameter-efffcient fine-tuning, e.g., inserting a adapter module or neural prompt search. Different from all the above works, we propose to scale and shift deep features extracted by a pre-trained model, which is simple but effective and outperforms other parameter-effiient fine-tuning methods.
2.3 Feature Modulation
Many works have attempted to modulate features to obtain better performance. The most relevant ones to our work are various normalization methods (26 [ (58. BN, LN, and GN usually normalize the features and then transform them linearly with scale and shift factors to modulate feature distribution, which has been verified to be effective in amounts of tasks. STN to spatially transform feature maps. In the field of image generation, AdaIN (55 generates scale and shift factors to characterize specific image styles. Self-modulation [4] shows GANs benefit from solf-modulation layers in the generator. In vision-language tasks, Conditional BN [6) and FiLM (45) are often utilized to modulate the features of two modalities. Unlike some algorithms such as BN, our SSF is not limited to the modulation of normalization layer, and it has different motivation that is to alleviate the distribution mismatch between upstream tasks and downstream tasks for ceaameter--fffcieen fine-tuning. As a comparison, we also conduct experiments in Sec. 4.3 and show that our SSF is more effective compared to only tuning the normalization layer. Compared to STN, AdaIN, FiLM and so on, our method is input-independent and these scale and shift parameters model the distribution of the whole dataset so that they can be absorbed into the original pre-trained model weights in the inference phase.
2X introduces learnable module
3

---

2.4 Model Re-parammeeerization
Model re- e-parameterization has been a common practice to improve inference efficiency. One of the representative techniques is batch normalization folding used in the model compression algorithms [27]. The parameters introduced by the batch normalization layers [26] are merged into the convolutional layers usually stacked before them. This technique is further utilized to merge different branches of networks into new branch [65] TO, 9 Similarly, our SSF fully adopts linear transformations, which allows the scale and shift parameters in the training phase to be merged into the original pre-trained model weights, thus avoiding the introduction of the extra parameters and computational cost during the inference phase.
3 Approach
3.1 Preliminaries
Transformers. In a vision transformer (ViT) [T, an RGB image I E R3xHXX is divided into N N non-overlapping patches, and then these image patches appended a class token fed into an embedding layer followed by the L-layer vision transformer blocks with self-attention as the core operation. The input a E IR(N+ +1xxd where d is the embedding dimension, is first transformed to keys K E [ values V <g(xx+))xd and queries Q E +1)xd After that, we can calculate global self-attention by
QKT Vd
Attention(Q, (Q, K,K) V) = Softmax VV.
The output of the attention layer will be fed to two-layer MLP extract information in the channel dimension.
Adapter. Adapter [2] is inserted into the transformer layer for efficient fine-tuning, It is a bottleneck module with a few trainable parameters, which contains a down-projection to reduce the feature dimension, non-linear activation function, and an up-projection to project back to the original dimension. Therefore, given the input R R[N2+1)xd the output is calculated by
out W of Wyownn >")]"
(2)
where Wroown gd''dd (where d' < d), o. and Wup E Rdxdd represent the down-projection matrix, non-linear function, and up-projection matrix, respectively.
VPT. VPT [29] inserts some learnable parameters (i.e., prompts) into the input space after the embedding layer. These prompts interact with the original image tokens by performing self-attention. During the fine-tuning, the weights of the backhone network are kept frozen and only the parameters of the prompts are updated. VPTTShallow inserts prompts in the first layer while VPT-Deep inserts prompts in all the layers of the transformer. Assuming that the input is E 1))++1xxd denote the inserted prompts as p E Rndd where is the number of prompts, the combined tokens x' is
= [z;p].
where x' E C(N***++ +n+1)xd will be fed into the transformer block for self-attention (Eq.
1
3.2 Scaling and Shifting Your Features for Fine-tuning
Different from the above methods, we introduce both the scale and shift factors to modulate deep features extracted by pre-trained model with linear transformation to match the distribution of a target dataset, as mentioned in Sec. T Five main properties are covered our method: i) SSF achieves on-par performance with the full fine-tuning strategy; ii) all downstream tasks can be inputted to the model independently without relying on any other task; iii) the model only needs to fine-tune very few parameters; iv) unlike VPT 22), which adjusts the number of prompts for each task, the set of parameters for fine-tuning in SSF does not change as the task changes, making it feasible further fine-tune the parameters later by adding more tasks for multi-task learning or continuous learning?) v) thanks to the linear transformation, SSF avoids the introduction of the extra parameters and computational cost during the inference phase, making our method zero overhead.
I provides more flexibility, which is not contradiction to ii).

---

The design of SSE. SSF performs the linear transformation to modulate the features for parameter-effiient fine-tuning as shown in Fig- 2, In Figure 2 (a), given a model pre-trained in the upstream task, we insert SSF-ADA after each operation (OP) of the network to modulate features. There are K OPs in total and these op- erations might contain multi-head self-attention (MSA), MLP and layer normalization (LN), etc. During the fine-tur ain-ttnnng, the pre-trained weights in these operations are kept frozen and the SSF- ADA parameters are kept updated. The spe- cific SSF-ADA structure is shown in (e), where the features output from the previous oper- ation performed dot product with scale fac- tor and then summed with a shift factor, which input-independent. Formally, given the input E (V++1))xd the output (R(N+11)xd (is also the input of the next operation) is calculated by
ADA
DP
OP,
Layer
[e] Training
scale shift
Head
GP
OF,
Layer
Input
(b)
Pre-trained model & Inference
(c) SSF-ADA
Frozon
Trainable
The overall pipeline of SSF (a) Training pipeline via SSF, where an OP means an operation. egg., MSA, MLP or LN. (b) A pre-trained model or inference pipeline. (c) Our SSF-ADA.
Figure 2:
y 7 x 3
where Y E Rd and B Id are the scale and shift factors, respectively. is the dot product.
Re-parameeterzattoo. Since SSF-ADA a completely linear transformation, we can re-parameterize it by absorbing the scale and shift terms into the previous linear layer as follows
= O x
3 (w + b) + 3 = (7 @ w)) 7 11+770+++
(5)
where and are the weight and bias terms, respectively. represents the 'convolution' operation in the convolutional layer or the 'multiplication' operation in the MLP layer. is the input of the previous linear layer. Since and b are frozen and y and | are updated in the fine-tuning, y and can be merged into the original parameter space (w and b) in the inference stage through the above formulation. From this perspective, SSF-ADA makes it possible perform downstream tasks without adding any extra parameters and computational costs, as shown in Fig
Discussion. The first question is why want the input 7 and / to be input-independent. As FiLM 45 and AdaIN (25) show, we could obtain y and 3 by conditioning an image sample, however, this might cause two shortcomings. One is that we want 7 and / to be input-independent to represent the distribution of the whole downstream dataset so that we can modify the previous weight distribution to fit the downstream dataset by modulating the feature. Secondly, the conditional input requires the introduction some additional networks (e.g., MLPs) to generate y and B, which introduces more trainable parameters. More importantly, to better generate 7 and B3 non-linear activation function might be required, which will lead to the intractability of the re- re-parameterization. Therefore, we directly perform fully linear transformation to merge the y and / factors into the original pre-trained weights, so that weights can be easily uploaded to the edge devices without any modification of the backbone architecture.
The second question is which operations should followed by SSF-ADA. Our experience that search
you insert SSF-ADA after each operation with linear coefficient in VIT. Although for some optimal layers or operations with Neural Architecture Search (NAS) [4 35 [4 34], to reduce the number of the trainable parameters, we believe that method will produce better results (or not worse than NAS) without introducing too many trainable parameters that can be merged for inference, will be shown in Sec. 4.3
3.3 Complexity Analysis
We also compare the complexity of Adapter, VPT and our SSF Take VIT example, the dimension and number of the tokens are d and N Assuming that Adapter projects features from d-dim to d' dim (where d' d) so that the extra trainable parameters 2dd in each layer,
Here, refer to our proposed method SSF and the specific module SSF-ADA.
5

---

VPT inserts n prompts to obtain nd ex- tra parameters in each layer, and SSF
Method
Adugeer
VPT-Stallow
VPT-Dopp
SSF (ours)
Exra. Purass.
mid
mdd (0)
inserts SSF-ADA after each operation with a linear coefficient to obtain md extra parameters in each layer, when the total number of layers is L. the complexity of Adapter, VPT and SSF shown Table 2 The specific num- ber of additional parameters used by Adapter, VPT and SSF depends on the values of ,' n and m. However, in practice, SSF outperforms Adapter and VPT-Deep even with slightly fewer parameters in the training stage as we will see in Sec. 4 Further, in the inference stage, borrowing the model re-parameterization strategy, the extra parameters and FLOPs of SSF are zero. However, the complexity of Adapter and VPT remain the same compared to the training, which establishes the strengths of our approach.
Extra FLOPs
2N (b)
2at2N? mid
2ut2N nlLd()
mdd
Table 2: The complexity comparisons of Adapter 211. VPT 299 and our SSF. '(I):: the same parameters and FLOPs for
training and inference; '(0):: : no additional parameters and FLOPs are required for inference.
Experiments
4.1 Experimental Settings
Datasets. We mainly conduct our experiments on series of datasets that can be categorized into three types as detailed below:
FGVC. Following VPT [29]. we employ five Fine-Graine GrainGr Visual Classification (FGVC) datasets to
evaluate the effectiveness of our proposed SSF, which consists of CUB-200-2011 (57). NABirds (55).
Oxford Flowers 44. Stanford Dogs (30] and Stanford Cars [12]
VIAB-Ik. VTAB-1k benchmark is introduced in (67. which contains 19 tasks from diverse domains: i) Natural images that are captured by standard cameras; ii) Specialized images that are captured by non-standard cameras, e.g., remote sensing and medical cameras; iii) Structured images that synthesized from simulated environments. This benchmark contains a variety of tasks (e.g., object counting, depth estimation) from different image domains and each task only contains 1,000 training samples, thus is extremely challenging.
General Image Classification Datasets. We also validate the effectiivnness of SSF on general image classification tasks. We choose the CIFAR- 100 [3I] and ImageNet-IK 7 datasets as evaluation datasets, where CIFAR-100 contains 60,000 images with 100 categories. ImageNet- K contains 1.28M training images and 50K validation images with 1,000 categories, which are very large datasets for object recognition.
Models. For a fair comparison, we follow VPT 22 and mainly select ViT-B/16 [1] model pre- trained on ImageNet-2IK IK as the initialization for fine-tuning. In addition, we also generalize our method to backbones of different model families, including the recent Swin Transformer (88 (Swin- B). ConvNeXt-B [39] and AS-MLP-B [33. The former builds hierarchical transformerbbsee architecture, and the latter two belong to CNN-based architecture and MLP-based architecture respectively.
Baselines. We first compare our method with the two basic fine-tune tne-tuning methods: i) full fine-tuning. where all parameters of the models are updated at fine-tume tne-tuning; ii) linear probing, where only the parameters of the classification head (an MLP layer) are updated. We also compare our method with recent parameter-efffcient fine-tuning methods: iii) Adapter (21), where a new adapter structure with up-projection, non-linear function, and down-projection inserted into the transformer and only the parameters of this new module are updated; iv) Bias (66). where all the bias terms of parameters are updated: v) VPT (29), where the prompts are inserted into transformers as the input tokens and they updated at fine-tuning.
Implementation Details. For the FGVC datasets, we process the image with a randomly resize crop to 224 x 224 and random horizontal flip for data augmentation. For VTAB- we directly resize the image to 224 x 224, following the default settings in VTAB [67] For CIFAR-100 and ImageNet- IK, we follow the fine-tuning setting of ViT-B/16 in [II, where the stronger data augmentation strategies are adopted. We employ the AdamW 41 optimizer to fine-tune models for 100 epochs for CIFAR- 100, and 30 epochs for ImageNet- K. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up used in the first 10 epochs for CIFAR- and epochs for ImageNet-IK.

---

Dataset
CUB-200
Oxford Flowers
Stanford Dogs
Stanford Cars
Params,
NABirds
Mean
Method
-2011
(M)
Full fine-tuning
87.3
82.7
98.8
89.4
84.5
88,54
85.98
Linear probing
85.3
75.9
97.9
86.2
51.3
79.32
0.18
Adapter [1
87.1
84.3
98.5
89.8
68.6
85.67
0.41
Bias [66
88.4
84.2
98.8
91.2
79.4
88,41
0.28
VPT-Shallow 29
86.7
78.8
98.4
90.7
68.7
84.62
0.25
VPT-Deep 29
88.5
84.2
99.0
90.2
83.6
89.11
0.85
SSF (ours)
89.5
85.7
99.6
89.6
89.2
90.72
0.39
Table 3: Performance comparisons on five FGVC datasets with ViT-B/16 models pre-trained on ImageNet-21K.
Natural
Specialized
Structured
CIFAR- 100
Flowers 102
Paich Camelyoo
Retioopatty
Clevrlouun
Clevrtiiuunce
KiTTiidistnce
dSpritesooc
dSpriteskoii
SmallNOR
SmalINOR Biele
Params, (M)
Dataset
Callech
DTD
Pot
Sun397
EueSAT
RessedS
DMLaab
Mcon
Method
Full line tonin- 29
66.99
95.7
84.2
73.9
56.3
58.6
41.7
65.5
57.5
65.57
85.84
Lincar probing
29
63.4
85.0
63.2
97.0
86.3
36.6
$1.0
78.5
87.5
68.6
74.0
34.3
30.6
33.2
53.4
12.5
20.0
9.6
19.2
52.94
0.04
Adaptor
21
74.1
86.1
63.2
88.0
73.1
70.5
45.7
37.4
31.2
53.2
30.3
25
55.82
0.27
Bias (60
72.8
87.0
59.2
97.5
85.3
59.9
51
78.7
91.6
72,9
69.8
61.5
55.6
32.4
55.9
66.6
40.0
15.7
25.1
62.05
0.14
VPT-Shallow
29
37.7
86.9
62.6
97.5
87.3
74.5
51.2
78.2
92.0
75.6
72.9
50.5
58.6
40.5
67.1
68.7
36
20.2
34.1
64.85
0.11
VPT-Deep
X0
78.8
90.8
65.8
98.0
88.3
78.1
49.6
81.8
96.1
83.4
68.4
68.5
60.0
46.5
728
73.6
47.9
32
37.8
69:43
0.60
SSF (ours)
69.0
92.6 75.1 99.4
91.8
90.2
52.9
187.4
95.9
87.4
75.5
75.9
62.3
53.3
80.6
77.3
54.9
29.5
37.9
73.10
0,24
Table 4: Performance comparisons on the VTAB 1k benchmark with ViT-B/16 models pre-trained on ImageNet-2IK.
4.2 Performance Comparisons on Image Classification
We compare the performance of our SSF and other baseline methods in 26 image classification tasks and the results on FGVC and VTAB-Ik are shown in Table 3 and Table 4 (also see Figure D. respectively, and the results on CIFAR-IO0 100 and ImageNet- 1K are shown in Table which are evaluated in Top-1 accuracy (%). In these three tables, the bold font shows the best accuracy of all methods and the underline font shows the second best accuracy.
We have the following findings by observing them: i) In Jad Table where the last column is the average of the fine-tuned parameters for each method on the corresponding datasets, our SSF outperforms VPT [29] and other parameter-efficient fine-tuning methods, and even achieves better performance than full fine-tuning, which is mainly owing the linear transformation applied on the features. Specifically, SSF obtains 1.81 % (90.72% vs. 89. 1..1 11%) and 2.46% (90.72% vs. 88.54%) accuracy improvement on five FGVC datasets, and 5.29% 173.10% vs. 69.43%) and 1 11.48% (73.10% vs. 65.57%) improvement on the VTAB-Ik benchmark compared to VPT and full fine-tuning. Meanwhile, SSF also uses fewer trainable parameters compared VPT-Deep in both datasets (0.39M vs. 0,85M, 0.24M vs. 0.60M). SSF maintains a unified learnable parameter space for different tasks with a few parameters while VPT 29 needs to design the different number of prompts for each task, which also shows the conciseness of our approach; ii) In Table 5 i.e., in CIFAR-100 and ImageNet- IK, SSF and other garameter-efficieee fine-tuning methods have difficulty in achieving the similar performance to the full fine-tuning, probably because these datasets have sufficient data to prevent over-fitting of the model, especially in ImageNet- K. contrast, in the VTAB-1k benchmark, the amount of data is not very large (e.g., only 1,000 training images), which might cause over-fitting of the model for the full fine-tuning. Nevertheless, in CIFAR- 100 and ImageNet-IK, our SSF still outperforms previous parrameter-efficientt fine-tuning methods (Adapter, Bias, and VPT), which shows the effectiveness of our method; iii) In Table the results of our SSF with Swin Transformer, ConvNeXt, and AS-MLP models consistently outperform those of other parameter-efficient fine-tuning methods, which also verifies the effectiveness of SSF on a wide variety of models.
Computational cost. To validate the efficiency of our method, we show the computational cost of SSF in Figure 3 We employ a batch size of 16 for the training stage and inference stage, and use mixed precision training. All running results in Figure R are measured single GeForce RTX 2080Ti GPU, We can see that SSF has similar training time and training memory with VPT but with
7

---

Model
VIT-B/16 1T
Swin-B 33
ConvNeXt-B [39
AS-MLP-B 33
ImageNet- IK
ImageNer- IK
ImageNet- IK
CIFAR- 100
Params. (M)
Params. (M)
CIFAR- 100
Parums. (M)
Params. (M)
CIFAR-10 100
Params. (M)
Params. (M)
CIFAR-10X 100
Parums. (M)
Dataset
Method
Full fine-tuning
93.82
85.88
83.58
86.57
93.85
86.85
85,20
88.03
94.14
87.67
85.80
88.85
89.96
86.83
Linear prohing
88.70
0.08
82.04
0.77
89.27
0.10
83.25
1.03
89.20
0.10
84.05
1.03
79.04
0.10
Adapter 21
93.34
0.31
82.72
1.00
92.49
0.33
83.82
1.26
92.86
0.45
84.49
1.37
88.01
0.33
Bias [66
93.39
0.18
82.74
0.87
92.19
0.24
83,92
1.16
92.80
0.23
84.63
1.16
87.46
0.26
VPT-Shallow
20
90.38
0.23
82.08
0.92
90.02
0.13
83.29
VPT-Deep 29
93.17
0.54
82.45
1.23
92.62
0.70
83.44
1.63
SSF (ours)
93.99
0.28
83.10
0.97
93.06
0.37
84.40
1.29
193.45
0.36
184.85
1.28
88.28
0.37
Table 5: Performance comparisons on CIFAR-I00 and ImageNet-IK with various model families, where ViT-B/16, Swin-B, and ConvNeXI-B are pre-trained on ImageNet-2IK, and AS-MLP-B is pre-trained on ImageNet-IK.
140
60
120
1.76
Toaiiing Iim (nee)
100
40
Takiing mansoy 2
Sinw
1.00
8
20
8
0.50
10
1
0.25
Useer pottig
Adapor
MP dieter Ver-Dup
Abss
(ous)
For dinr posing
Asaye
peraaaade sercohop aur (nor)
Fir treekoong
Uner pustig
Austo
yor Shadee Dee
tiee
usr (our)
Forl tnokkring Uswe pooig
0.00
ser Chatore
(our)
Figure 3: Computational cost of different tuning methods. From left to right: training time, training memory, test time, and test memory.
less inference time and inference memory. Here, we show the computational cost of VPT with 200/50 prompts (the same number of prompts to obtain the performance in 5) for VPT-Shallow and VPT-Deep, respectively. When adding the number of prompts, the time cost and memory will be larger but our SSF achieves zero-overhead inference, which is more advantageous.
4.3
The Impacts of Different Designs
As the core operation SSF, we thoroughly evaluate how SSF-ADA affects results, e.g., the insertion locations, the initialization of SSF-ADA and its components. We conduct experiments to analyze the impacts of different designs for fine-tuning. All experiments are implemented with pre-trained ViT-B/16 models on CIFAR-100 and the results are shown in Table
The impact of the number of layers. We directly insert SSF-ADA into different layers to evaluate the effect of inserting layers, and the results are shown in table ba The values in the #layers column indicate the number of layers with SSF-ADA, where #layers- represents linear probing. From the first and second rows, we find that the results will improve from 88. 70% to 92.69% and grow with small number of trainable parameters (0.08M vs. 0.1 when only inserting SSF-ADA into the first two layers. Keep adding SSF-ADA in the subsequent layers will make the results better. The growth of the results is almost linear with the number of layers of inserted SSF-ADA. Therefore, we directly choose to insert SSF-ADA into all (12) layers of vision transformer to bring the best results (93.99%) with 0.28M trainable parameters.
The impact of the different insertion locations. Based on the different operations of VIT, we evaluate the impact of the insertion locations of SSF-ADA. We separately remove SSF-ADA after these operations and the results are shown in Table 6b We find that removing the SSF-ADA in the MLP operation achieves inferior results than removing those in the Attention operation (93. (93.46% vs. 93.69%) with comparable trainable parameters (0. vs. 0.21M), which suggests that performing feature modulation for the MLP operation might be more important. Although one can use NAS to search for the importance of different operations and thereby insert SSF-ADA in specific locations, the results might not better than inserting SSF-ADA in all operations. Therefore, in order obtain excellent performance, we do not perform NAS but directly insert SSF-ADA into all operations.
8

---

#layers
Acc.
Params.
location
Acc.
Params.
initialization
Acc.
case
Acc.
Params.
88.70
0.08
w/o, mip
93.46
0.19
random
90.11
w/, scale
93.49
0.18
2
92.69
0.11
w/o. attn
93.69
0.21
constant
93.91
w/o, shift
93.74
0.18
4
93.30
0.15
w/o. embed
93,91
0.28
uniform
93.87
only norm
93.26
0.11
8 93.60
0.22
w/o, norm
93.80
0.25
sone__ormal
93.93
scalar scale
93.59
0.18
12 (ours)
93.99
0.28
ours
93.99
0.28
normal (ours)
93.99
ours
93.99
0.28
(c) Table 6: The impacts of different designs. (a) The impact of the number of layers with SSF-ADA. (b)
(a)
(b)
(d)
The impacts of the different insertion locations of SSF-ADA. (c) The impacts of initialization. (d)
The impacts of different components. Acc.: Top-1 accuracy (%);; Params.: parameters (M).
The impact of initialization. We also investigate how different ways of initializing the scale and shift factors affect performance in Table 6c| In our experiments, we first randomly initialize both scale and shift parameters with mean value of zero, but find that the performance is inferior (90. 11 1%) and cannot converge in some experiments. After that, we randomly initialize the scale factor with a mean value of one and find better performance, which implies that the weights of pre- pre-trained model should not be completely disrupted in the fine-tuning, instead, we should start from this pre-traine pre-trained model optimize our model. Experiments show that using the normal initialization achieves the best performance, where the mean values of the scale factor and shift factor are one and zero, respectively.
The impact of different components. We also evaluate the impacts of different components in SSF-ADA and the results are shown in Table 6d We find that removing the scale term yields worse performance than removing the shift term with the same trainable parameters, which shows that the scale term might be more important than the shift term. Also, note that the difference between 'w/o. scale' and the 'Bias' method in Table that fine-tune the model with an additional shift term in 'w/o. scale', while 'Bias' fine-tunes the model based on the original biases, suggesting that fine-tuning the model in a res-like manner can obtain slightly better performance (93, (93,49% vs. 93.39%) We also try to only fine-tune all scale and shift factors in the normalization layer (LN), or fine-tune the model with SSF but set the scale term scalar. These experiments yield inferior performance than SSF (93.26% vs. 93.99% 93.59% vs. 93.99%), but could probably be considered as an alternative due to the fact that they only use about half of the trainable parameters of SSF
4.4 Performance Comparisons on Robustness and OOD Datasets
We also conduct experiments to analyze the robustness and Out-Of-Distribution (OOD) ability of our SSF method with
Dataset
IN-1K (t)
IN-A
IN-R
IN-C
Method
fine-tuning Linear probing
83,58
34.49
$1.29
46.47
the following datasets: ImageNet-A. ImageNet-R and ImageNet-C. Please fer to Appendix for their details. We per- form the robustness and OOD evaluation on these three datasets with the fine-tuned models ImageNet- 1. All experimen- tal results are listed in [
82.04
33.91
52.87
46.91
Adupter 20
82.72
42.21
54.13
42.65
Blas (6
82.74
42.12
55,94
41,90
VPT-Shallow
h0
82.08
30.93
53.72
46.88
VPT-Decp 20
82.45
39.10
53.54
40.10
SSF (ours)
83.10
45.88
56.77
41.47
Table 7: Performance comparisons on robustness and out-of-distribution datasets. 'IN' means ImageNet. The performance on IN-IK, IN-A and IN-R is evaluated in Top-1 accuracy (%). The performance on IN-C is evalu- ated in mCE (mean corruption error). The lower (L), the better.
From this table, we can see that our SSF obtains better performance than VPT and other parameter-efficient fine-tuning meth- ods on three datasets, which shows our fine-tuning method has stronger robust- ness and out-of didto---dittrbutiinn generalization. Furthermore, although SSF has lower accuracy than full I IK, the performance ImageNet-A, ImageNet-R and ImageNet-CI is better, which also shows the performance between ImageNet- KK and ImageNet-A/R/CC is not absolutely positive relevant. Such improvements in robustness and OOD datasets might come from the fact that SSF freezes most of the pre-trained parameters, which maximally preserves the knowledge learned from the large-scale dataset and thus maintains better generalization ability.
fine-tuning ImageNet-
4.5
Visualization and Analysis
Although our goal is to modulate the features extracted by pre-trained model, the scale and shift parameters are input-independent indeed. Therefore, these parameters can also be regarded as encoding information of the whole downstream dataset. After re re-parameterization. these scale and

---

Seiglt DEG
Weigh patocton
WIgT CHEDANY
RIigE EAI BARY
Wiight ESSTMARY
Welkh derteeke
Lyvil
Bix 2430
Blu CO putinn
3/ Sorttutte
(a) Pre-trainee model Fine-tuned model via SSF
(b) Pre-trained model vs. Full fine-tunck model.
Figure 4: Comparisons of parameter distribution hetween the original pre-trained model and different fine-tuning methods. The first row shows weight distribution and the second row is bias distribution. The blue histograms show the original pre-trained model, and the orange ones show the fine-tuned model via SSF in (a) and full fine-tuned model in (b).
shift parameters are absorbed into the original model weights. To better understand information learned by the SSF, we visualize the distributions of weights and biases before and after fine- tuning via SSF in Figure 44 We can see that the scale and shift parameters adjust the original weights and biases, and change the distribution of weights and biases to fit the downstream task. As comparison, we also visualize the original weight distribu- tion and the weight distribution after full fine-tuning in Figure 4b. from which we can find an interesting phenomenon that full fine-tuning does not change the distribution of weights and biases much, but probably only a small portion of the values is changed. It is worth noting that although SSF does not match the weight distribution of full fine-tuning, it achieves better performance (93.99% vs. 93.82% in T on CIFAR-100.
0.9
proting
Festure similantty
0.8
VPT Deep
SSF (ours)
0.7
0.5
0.3
To further investigate why SSF can achieve superior perfor- mance, beyond weight distribution, we also visualize the fea- ture similarities between full fine-tuning and linear probing. full fine-tuning and VPT-Deep. full fine-tuning and SSF, as shown in Figure 5) In the last layer, SSF has the most similar feature to full fine-tuning and the accuracy is also the closest. This shows that even if the weight distribution learned by SSF is different from full fine-tuning, SSF is also able to extract the features of the images in the downstream task very well, which validates the effectiveness of our method.
02
10
Layers
Figure 5: The visualization of the feature similarities between full fine-tuning and linear probing, full fine-tuning and VPT-Deep. full fine- tuning and SSF, in different layers of ViT-B/16.
Conclusion
In this paper, we focus on parameter-efffiient fine-tuning and propose an SSF method scale and shift the features extracted by pre-trained model. The intuition behind our method comes from alleviating the distribution mismatch between upstream tasks and downstream tasks by modulating deep features. SSF surprisingly outperforms other parameter-efficient fine-tuning approaches with a small number of learnable parameters. Besides, the introduced scale and shift parameters during the fine-tuning can be merged into the original pre-trained model weights via re-parameterization in the inference phase, thereby avoiding extra parameters and FLOPs. With the proposed SSF method, our model obtains 2.46% (90.72% vs. 88. 88.54%) and 11.48% (73. 73.10% vs. 65.57%) performance improvement on FGVC and VTAB-Ik in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. Experiments on 26 image classification datasets in total and 3 robustness & distr--distintution datasets with various model families (CNNs, Transformers, and MLPs) show the effectiveness of SSF, which establishes new baseline.
A Acknowledgement
The authors acknowledge the support from the Singapore National Research Foundation ("Cog- ViVision - seergy-autonomouu always-on cognitive and attentive cameras for distributed real-time vision with milliwatt power consumption" grant NRF-CRP20-2017-000 www. www.green-ic. orgg CogniVision] Xinchao Wang is the corresponding author.
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.0644 0.65000 2016.
[2] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11285-11297.
Curran Associates, Inc.. 2020.
[3] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping
Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint
arXiv:2205. 13052 2022.
[4] Ting Chen, Mario I Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019. [5] Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977,
2021.
[6] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. Advances in Neural Information
Processing Systems, 30, 2017.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[8] Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480 1810.04805, 2018.
[9] Xiaohan Ding, Chunlong Xia, Xiangyu Zhang, Xiaojie Chu, Jungong Han, and Guiguang Ding. Repmlp: Re-parameterizing convolutions into fully-connected layers for image recognition. arXiv preprini arXiv:2 0183, 2021.
[10] Xiaohan Ding. Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733-13742 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[12] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[13] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 4805-4814 14, 2019.
[14] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pages 544-560. Springer, 2020.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738. 2020.
Se. [17] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the TEEF/CVF International Conference on Computer Vision, pages 9918-4927, 2019. [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 770-778, 2016.
[19] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter- efficient fine-tuning for vision transformers. arXiv preprint arXiv:2203. 16329, 2022.
[20] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mip-like architecture for visual recognition, arXiv preprint arXiv:2106.11 12368, 2021.
11

---

[21] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for nip. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019. [22] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. To Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 704.0486,, 2017.
[23] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1ttt:2/06.09960 2021.
[24] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[25] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages
1501-1510, 2017.
[26] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages
148-456 2015.
[27] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient aiteger-arithmetii-nnly inference. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2704-2713, 2018.
[28] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks.
Advances in neural information processing systems, 28, 2015.
[29] Menglin Jia, Luming Tang, Bor- Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.
[30] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Graine Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
arXiv preprint w:Xiv:2/01.00///
2021.
[33] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architec- ture for vision. In International Conference on Learning Representations (ICLR), 2022. [34] Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations (ICLR), 2020.
[35] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR). 2019.
[36] Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI Conference on Artificial Intelligence, 2021.
[37] Songhua Liu, Jingwen Ye, Sucheng Ren, and Xinchao Wang. Dynast: Dynamic sparse trans- image generation. In Proceedings of the European Conference on
former for exemplar-guiee Computer Vision. 2022.
[38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[39] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[40] Ze Liu, Jia Ning, Yue Cao. Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106.11 13230, 2021.
[41] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[42] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint wwXivv:20660444 04647, 2021.
[43] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv 04489, 2021.
preprint arXiv:2 106.04489
12

---

[44]
Maria- Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEER, 2008
[45] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[46] Hicu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095-
4104. PMLR, 2018.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning,
pages 7748-8763. PMLR, 2021.
[48] Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self- attention via multi-scale token aggregation. In IEEE Conference on Computer Vision and Puttern Recognition, 2022.
[49]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
[50] Baochen Sun, Jiashi Feng. and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[51] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pages 6105 61 14, 2019. [52] Ilya Tolstikhin, Neil Houlsby. Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. MLP-Mixer: An all-mip architecture for vision. arXiv preprint 105iv:2205,016600 2021.
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa. Alexandre Sablayrolles, and Hervé Jégou. Training Mata-efficien image transformers & distillation through attention. In International Conference Machine Learning (ICML), 2021.
[54] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. arXiv preprint urXhv:2203.09795 2022.
[55] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessic Barry, Panos Ipeirotis, Pictro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-604, 2015.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2017.
[57] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2000 201-2011 dataset. 2011.
[58] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-, 2018
[59] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[60] Xingyi Yang. Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In
Proceedings of the European Conference on Computer Vision, 2022.
[61] Yiding Yang. Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. In Conference on Neural Information Processing Systems, 2020.
[62] Yiding Yang. Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[63] Jingwen Ye. Yifang Fu, Jie Song, Xingyi Yang. Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. In Proceedings of the European Conference on Computer Vision. 2022.
[64] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[65] Sergey Zagoruyko and Nikos Komodakis. Diracnets: Training very deep neural networks without skip-connections. arXiv preprint arXiv:1706.00338 2017.
13

---

[66]
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efftcient fine-tuning for transformer-basee masked tngguage-models arXiv preprint arXiv:2106.10199
2021.
[67] Xiaohua Zhai, Joan Puigeerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910048667 2019.
[68] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint
arXiv:2206. 04673,, 2022.
[69] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. DeepViT: Towards deeper vision transformer. arXiv preprint ar arXiv:2 1886,
2021.
[70] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Anima Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness vision transformers. arXiv preprint arXiv:2204. 12451, 2022.
[71] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint 1rX1v:2109.011 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See abstract, introduction and experiments.
(b) Did you describe the limitations of your work? [Yes] See appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See abstract and experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See experiments.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] A part of the experiments is tested several times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See experiments.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See appendix.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "vIKRmxes-WV", "reviewer": "Reviewer_SF4A", "summary": "The authors propose a simple linear adapter for parameter-efficient fine-tuning. Unlike previous methods that involve complex non-linear operations or prompting, they explore linear feature scalability of vision transformers and propose LIFTs-ADA. Furthermore, their linear adapters can be merged or subsumed into the original modules of the network during inference stage making them not requiring any additional parameters and FLOPs. Their experiments show that their proposed method performs at par or better than previous competing methods such as VPT with substantially less number of parameters during fine-tuning for downstream tasks.\n\n\n", "questions": "1. Please review and explain this section better (lines 175-177)\n\nHere, we consider that g is a linear function for its simplicity of the linear function but the lack of comprehensive investigation and the properties of the linear function make it possible to be merged with other operations of the network in the inference phase.\n\nI do not understand this line. Looks like 2 unrelated sentences have been merged.\n\n2. The claim (lines 27-27) that the you do not have to store the pretrained seperately for each fine-tuned model seems flawed. If the Linear adapters are subsumed once into the pre-trained network during inference stage, they you would have to do this for each downstream task and store a different model for each downstream task. ", "limitations": "1. Lack of results on robustness or OOD performance.\n2. Writing quality at some places is not up-to the standards.\n3. Lack of theoretical grounding (but this can be excused).\n4. For each downstream task, you still have to store a separate model since the parameters of the linear adapters are subsumed into original parameters of the pre-trained network thus updating the parameters effectively.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Method is simple, effective and easy to use.\n2. Inference stage does not require any additional parameters and FLOPs, thereby reducing computational costs. They also provide numbers (#params and #FLOPs) for comparison with another parameter-efficient fine-tuning method such as VPT.\n3. Authors have performed a comprehensive set of experiments with various datasets and downstream tasks to support their method and claims.\n4. Authors also perform a series of ablations to understand the effectiveness of locations for placing their linear adapters.\n\nWeaknesses\n1. Paper is poorly written in some places. Some specific lines 103, 175, 176, 177. In general the paper lacks the written maturity of a Neurips paper.\n2. Lack of theoretical grounding. But their simplicity of idea and empirical effectiveness seem to be sufficient according to me.\n3. Lack of study on robustness and OOD performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oPGM2NUJ9hf", "reviewer": "Reviewer_eK8a", "summary": "The paper introduces a parameter-efficient transfer learning method for vision transformer, named LInear Feature Scalability (LIFTs). It inserts a linear-structured adapter layer to the transformer block, while keeping other model parameters unaltered. The micro and macro insertion positions are carefully searched through experiments. The trained linear adapter layer can be fused into the original network structure by applying the reparameterization tricks, without additional computational overhead at inference time. LIFT shows promising results on a variety of evaluations.", "questions": "1.\tAs the paper put a lot of effort on the where to insert the LIFT block, the current version seems to a simplified Neural Architecture Search with grid search strategy. For example, 3 macro designs, called LIFTs-Shallow, LIFTs-Shared and LIFTs-Deep are evaluated. But there are still other hybrid micro and macro designs that are untested. I am wondering whether the author tried some NAS techniques for such combinational optimization problem.\n\n2.\tIn table 3, why not include the ViT-B structure for CIFAR-100 and ImageNet1k experiments? Since the Visual prompt tuning was originally designed for ViT-structured network, I believe it is more appropriate and important to report the results for CIFAR and ImageNet with ViT, since it provides a fair test bed for two methods.", "limitations": "Please provide some discussion on the limitations and societal impact in the paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tSimple but efficient design. The linear-structured adapter can be easily implemented and plugged into any existing vision transformer structure without hustle. \n2.\tThe proposed LIFT adapter takes the idea of reparameterization method, thus avoid the additional inference cost. \n3.\tThe performance improvement is encouraging.\n\nWeakness:\n1.\tSince the paper propose a new adapter-based architecture for transfer learning, please compare the computational complexity (Table 1) and model performance (Table 3 and Table 4) with original adapter-tuning and recent vision adapter structures. I do not see the results in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "USpmRis22fK", "reviewer": "Reviewer_RByJ", "summary": "In this submission, instead of optimizing all the parameters to adapt to a new task, the authors propose a LInear Feature Scalability (LIFTs) method that keeps all the model parameters from the pre-trained network but only learns the task-specific linear feature scalability adaption layer for each block. In this way, the authors alleviate the dilemma of useful information elimination and expensive computation burden as in previous all-parameter-optimized fine-tuning approaches, making the model readily and effectively reusable across various tasks. Experiments on both the fine-grained visual classification and general classification datasets convincingly demonstrate the superiority of the proposed LIFTs over prior works. Further results on segmentation and detection are also provided in the supplement.", "questions": "- Could you analyze the limitations of the proposed method? It will be helpful for follow-up research on this line of fine-tuning schemes with the feature scalability.", "limitations": "Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The proposed module is plug-and-play, which is easy to be incorporated in various architectures. The motivation is clear. The authors analyze the flaws of the existing full-parameter fine-tuning methods in details, and then demonstrate their solution of LIFTs accordingly.\n\n- The detailed algorithm workflow with the key code is provided, which makes it easier to understand the method details. Detailed discussions on the complexity are given in the method section, which is critical to understanding the contributions of the proposed LIFTs. The paper is easy to follow. The illustrations are quite clear, by firstly giving a definition of the key term of the feature scalability in this paper.\n\n- Extensive ablation studies and visualization results are provided, by exploring the impacts of various designs. \n\n[Weaknesses]\n- Please explicitly denote in Tab. 4 on what device the running time is obtained.\n\n- The authors are suggested to use the same font across all the figures. \n\n- It will be great if the authors could discuss the limitations of the proposed method, which may showcase the future directions on this line of research to some extent.\n\n- Grammars: Ln 329 “ … which makes the network applied to downstream tasks not require any additional parameters and FLOPs …”\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Km9fnfbVGUl", "reviewer": "Reviewer_r389", "summary": "A new fine-tuning method for vision transformer is proposed in this manuscript. The authors have explored the Linear Feature Scalability of vision transformers, which defined as the fine-tuning performance with only linear transformation of the feature map. A simple module named LIFTs-ADA is designed for a fine-tuning, where some additional linear transformation operations are injected after the MLP and MSA. Re-parameterization is designed to make the fine-tuning parameter-free during inference. It has shown its efficiency on different tasks and models.\n", "questions": "Please refer to the weaknesses part", "limitations": "The limitations are mainly about the unclear statement of some parts and the insufficient analysis about the learning behaviors (the different visualizations of the linear transformation).\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1, The definition of feature scalability and the observation that applying only a linear transformation of the feature map can lead to great performances on different downstream tasks are interesting. \n2, The proposed method is quite simple yet efficient for vision transformers with respect to the naive fine-tuning and vision prompt fine-tuning. The additional FLOPS of the proposed method is indeed lower than that of the prompt-based method.\n3, The Re-parameterization trick can neurally integrate the linear transformation learned for different tasks into the original linear layer of transformer, leading to a zero-overhead inference.\n4, The ablation studies about different configuration of ADA is good.\n\nWeaknesses:\n1, Some parts of the manuscript is unclear. For example, line 195, A_2^2 should be A_i^2. It is unclear whether the transformation A is different among different attention heads within the same layer.\n2, For the Fig. 3 (b), it is recommended to provide details explanation about how the similarity matrix is computed. Also, it would be interesting to show the dissimilarity matrix of the linear transformation before and after fine-tuning, since they are comparable after re-parameterization.\n3, For Tab. 2, why the Tuned/Total rates are so different between NABirds dataset and others?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept