Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing
Authors: Wei Dong
Dawei Yan
Zhijun Lin
Peng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: computer vision; vision transformer; visual adapter; transfer learning

Abstract: The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter Re-Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to further reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at https://github.com/DavidYanAnDe/ARC.

Efficient A Adaptation of Large Vision Transformer via Adapter Re-Composing
Wei Dong
Dawei Yan'
Zhijun Lin?
Peng Wang"
College of Information and Control Enginvering,
Xi' an University of Architecture and Technology.
Schooo of Computer Science, Northwestern Polytechnical University. "School of Computer Science and Engineering.
University of Electronic Science and Technology of China.
'Xi' an Hypersonic Measurement Technology Co., Ltd.
Abstract
The advent of high-capacity pre-trained models has revolutionized problem paallem-solviing solving in computer vision, shifting the focus from training task-specifi models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose novel Adapter Re- Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers reusability of adaptation parameters and introduces parameter-sharing scheme. Specifically, we leverage symmetric
to construct bottleneck operations, which are shared across layers. By learning now-dimensional re-scaling coefficients, effectively re-compose layer-adaptive adapters. This prrameter-sharing strategy in adapter design allows to further reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at https//gghhhb..omm//avvvaa/aa//
1 Introduction
The utilization of large-scale pre-trained models for various downstream tasks has garnered significant interest in the computer vision community 1 4 These models continually push the boundaries of downstream task performance while eliminating the need for task-specific model design and training. In early attempts, a commonly adopted transfer learning strategy involved directly fine- tuning a pre-trained model on downstream tasks. However, the full fine-tuning strategy suffers from two major drawbacks: (1) Updating large-scale parameters is prohibitively expensive and typically requires a substantial amount of training data to prevent overfitting. (2) As the sizes of otate-of-the-art pre- pre-trained models continue to increase, it becomes impractical and unsustainable to store distinct set of model weights for each downstream task.
Corresponding author. Email address: p. wange@hotmaai..... com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In contrast to fine-tuning the entire pre-trained network, recent research has focused on parameter- efficient model adaptation. The core idea behind this line of work is to keep the majority of pre-trained parameters frozen and only update or introduce a small fraction of task-specific parameters. Several methods fall under this umbrella, including prompt tuning [ (), visual adapter 7 8, and linear feature modulation (9. These methods have demonstrated competitive or even superior performance compared to full fine-tuning while significantly reducing the adaptation cost. They differ in the design of lightweight adapters and how these adapters interact with the pre-trained parameters. Prompt tuning methods [5) 6) adapt the features of pre-trained Vision Transformers by introducing trainable task-specific tokens into or multiple attention layers. Visual adapter M 7 injects non-linear lightweight adapter with bottleneck architecture between layers of the pre-trained model to adjust the feature distribution. Such non-linear adapters are simplified using linear transformations such as shifting and scaling 9 to directly modulate the pre-trained features.
In this work, we not only focus designing lightweight adapters but also emphasize the importance of adaptation parameter reusability in further compressing the adaptation cost. We adopt low-rank design for the adapter using bottleneck operation but propose a novel approach. Unlike other methods that place the adapter in different layers and directly learn different parameters for each adapter to cater to layer-wise variation, we propose sharing the down/up projections in the low-rank adapter across different layers and simply learning low-dimensional re-scaling coefficients to re- compose the linear projections into layer-adaptive adapters. The idea of Adapter Re-Core Re-Composing (ARC) is motivated by the observation that naturally derived adaptation matrices can exhibit extremely low-rank characteristics, even when not explicitly designed such. This implies the possibility of using a shared "basis" to re-compose the adapters. Furthermore, we design the low-rank adapters using symmetric down-projection and up-projection matrices, which further reduces the parameter size. Due to their linear nature and careful positioning design, our adapters can be seamlessly integrated into the pre-trained network, in inference phase.
9 [0] without adding extra computation during the
We evaluate our method on various large Vision Transformer models, including ViT-B 1I and its variants such as ViT-L (T) ViT-H [Л, and Swin-B [T]], using 24 downstream image classification benchmark datasets. The experimental results demonstrate that method achieves compelling transfer learning performance while maintaining a smaller parameter size.
The key contributions of this paper are summarized as follows:
We approach efficient pre-trained model adaptation from a novel perspective by exploring the reusability of adaptation parameters, which goes beyond existing works that primarily focus on the lightweight design of adapter structures.
We introduce the Adapter Re-Composing (ARC) strategy, which shares bottleneck oper- ation's 's down--uup-rojeections across layers and utilizes lower-dimensional re-composing coefficients to create prior works.
adapters. This approach enables fewer parameters than
Our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.
Through extensive experiments on various Vision Transformer variations and numerous downstream tasks, we show that our method achieves highly competitive transfer learning performance while maintaining a relatively low level of additional parameter.
2
Related work
In this section, we present a concise review of the existing literature, focusing on two key areas: (1) Pre-training and fine-tuning, and (2) Parameter-efficient transfer learning.
Pre-training and fine-tuning. Pre-training and fine-tuning also known as transfer learning [2
13 [4] is popular approach that utilizes large-scale datasets [5 16
17
18 19 to train models for
adaptation to different downstream tasks. By extracting knowledge from these datasets and encoding into parameters, models can fine-tuned for specific tasks, resulting in improved performance compared to models without pre-training The availability of large-scale datasets [17] has led to
2

---

Weighte
Ontpat
Outpet
Orstgut
Trainable Weight:
Up Projeciua
Transformer Black
Projection
ReLL
Weight
Matris
Down Projection
Dawwn
Scale
Projection
Troinable Trainsble
Iegui
Inpat
Inpst
Token
Tokess
Different
Ii Weight Matrico
Tolarss
Outpet
Oltput
Outgot
Transfermer Block
Projection
Projection
Tokes
Token
Down Projection
Down Projection
Trainable Trainable
Tokem
Token
Input
Inpus
(a) Prompt Learning 6
(b) Visual Adapter
i
(c) LoRA 24
(d) SSF g
Figure 1: Visual summary of typical parameter-efficien pre-trained model adaptation methods.
significant advancements in performance and convergence speed for downstream tasks. Scaling up models [20] 21 to handle the growing data volume enhances data utilization and improves the efficiency and robustness of pre-trained models across various data distributions and noise levels. Large models [ [2] also benefit from self self-super pre- pre-training (2 23 using unlabeled data, reducing the cost, duration, and quality issues associated with human annotation. By leveraging this approach, models effectively extract knowledge from readily available unlabeled data, further enhancing their generalization performance downstream tasks.
Parameter-efficient transfer learning. The exponential increase model parameters presents computational challenge when fine-tuning the entire network on downstream tasks. In the field of NLP,
researchers have explored parameter-efficient transfer learning approaches 24 26 27 28
that
train subset of the model or add new modules with fewer parameters while achieving comparable or even superior performance. Inspired by the success of NLP, several notable works 8 29 30 have emerged in the computer vision domain. One approach, known Prompt Tuning 15 6. addresses the distribution mismatch between pre- pre-training and downstream tasks by learning task-specific tokens. Adapter-lin like methods [8] 7 [9] insert trainable modules, such as MLPs with activation functions and residual structures, into the network to facilitate transfer learning. LoRA [24] exploits the low-rank update large-scale frozen model and introduces bypass the original parameter matrix to mimic the fine-tuning of the entire model parameters. SSF [9] introduces lightweight scaling and shift operations to modulate the pre-trained representations. The core concepts of these aforementioned
works are visually summarized in Fig. [
As advancement in visual adaptation, ARC addresses the limitations of Prompt Tuning, which requires different prompt designs for different downstream tasks. Moreover, ARC introduces the innovative concept of bottleneck matrix reuse, achieving state-of-the-au performance with minimal adaptation cost comparing to other rank-dccomposition strategies. Additionally, ARC employs a
linear design for the adapters and inherits the benefits of re-parameterization T0 24 that inference does not introduce any additional computational complexity.
ensuring
3 Approach
In this section, we start by providing an introduction to the notations, symbols, and background related to Vision Transformers, which is followed by the presentation of our proposed Adapter Re-Composing (ARC) method. ARC focuses on efficient transfer learning for Vision Transformers by reusing rank-decompositiee projections to adaptively compose layer-wise adapters, thereby reducing the size of learnable parameters. Additionally, we discuss the insights gained from our architecture design.
3

---

3.1 Preliminary
plain Vision Transformer model contains a patch embedding layer and multiple encoder layers. Given an imput image X E R "xx xC the patch embedding layer first splits the image into sequence of flattened patches Xpatches R INx((P=.C) where (H, W) is the resolution of the input image, (P, P) is the resolution of each patch, C' denotes the number of input channels, and N = H W/P2 the number of tokens. Subsequently, the image patches are mapped to a D-dimensional embedding space through a linear projection W E (E(P*C))x A learnable [class] token vector E.As E R is then prepended to the sequence of X patches and the position embeddings Xpos E [R(++)x are added to the sequence. The output of the patch embedding layer can be expressed as follows:
Xemb
[Felsi X patches W]
X Xposs
where [ denotes concatenation operation. This output is then fed into several consecutive encoder layers, each consisting of Multi- Malti-Head Attention (MHA) block and
Network (FFN)
block. LayerNorm (LN) is upplied before each block, and residual connections are applied thereafter. The process of 1-th encoder layer is defined as:
X MHA(LN(X
+ X(-I),
x (1) = FFN(LN(XX + +X,,,
where ((-1)) denotes input tokens in I-th layer, X indicates intermediate representations produced by MHA, and the output of [-th layer is X
In MHA block, each Attention Head (AH) module utilizes the weight matrices w E w E DD "' " and w RD ") for the query, key, and value operations, spectively. These operations enable an exclusive attention mechanism on the normalized feature representations x(l-I) ()(x(
Xnorm w!! X Form w())
T X
X AH (x(()) = softmax
(1-1) norm
(3)
where 10 D MF
is the feature dimensionality of the output representations ( for each attention
head and M represents the number of attention heads. The MHA block concatenates multiple (x) (1))
in sequence and generates the outputs through a linear projection w E (R(M-D())xD
x (T) (HHA((((-)) [AHl, (x ((-l))
AHM (x((-I) w!
(4)
The FFN block consists of two linear projections with the GELU activation function in between:
FF (X)) = GELU(X (1)
Form w(9) )
(5)
where w IR 12D0x4 and ( E g4.D0xxxx D denote two linear projection matrices, and (1y
X Aorm L.(
3.2
Adapter Re-Composing method
We observe that the majority of existing adaptation methods introduce adapters into various layers and learn separate parameters for adapting the pre-trained model to specific downstream tasks. Previous studies 88. 24] have shown the effectiveness leveraging the low-rank properties of adapters to fine-tune frozen pre-trained models. Inspired by these findings, we propose a novel approach to create unified linear space across different adapters enhance parameter efficiency and adaptation performance.
Architecture. The ARC architecture incorporates bottleneck operation for adapters, which consists of three key components: a linear down-projection, layer-specific re-scaling coefficients, and a linear up-projection. This architecture illustrated in Fig. 2 To facilitate the reusability of adaptation matrices, we have developed sharing and re-composing scheme for the ARC method.

---

Fruzen Weight eights
Trainable Weighls:
Bottleneck Operation:
Layer
Different Weight Diagonal Matrices
Down
Up Projection
Up
Projection
Projection
Projection
Ingoe
Mulli-Head Attetioon Block (MHA)
Feed-Forward (FFN)
Rapa
Re- BeaScaling
Re-Scaling
Coefticients
Identical Symmetric Wiight Matrioes for MHA
Identical Symmetric Weightt
W Weight Sharing
Layer
Layer 1+1
Bottleneck
MHA
Bottleneck for FEN
FFN
Bottleneck for MHA
Bottleneck
FFN
for MHA
for FEN
Weight Sharing
Figure Illustration of the proposed Adapter Re-Composing Method.
This scheme involves sharing linear projections across layers and learning low-dimensional re-scaling coefficients re-compose the layer-adaptive adapters. In addition, enhance parameter compression, we leverage symmetric down-projection and up-projection single bottleneck operation:
W
(Wdwwn)"
(6)
where W Woown& E RRDx and W up E DDD with D' < D denote shared down (own-projection and up- projection matrices across different layers; D' represents the hidden dimensionality of the projections. To accommodate the variations across different layers, we learn re-scaling coefficients re-compose the layer-adaptive adaptation matrices. These coefficients are then diagonalized into a diagonal matrix C(") E D D specific to each layer This diagonal matrix allows for efficient and effective adjustment of the adaptation parameters at each layer. Formally, given an input Xin E (NV+))xD the output of our ARC module is:
Xout A X,n down CO + Xin
Unless otherwise specified, we adhere to the default configuration of inserting our ARC modules sequentially before both the MHA and FFN blocks in the Vision Transformer. The influence of adapter positions will be discussed in Section 4.3 Therefore, the Vision Transformer incorporating
ARC modules can be formulated as follows:
MHA(ARCMMA (UN(X((-11)) + (X-11 (1) (1)
X FFN (LN(( (LN(X (I)) +X
Note that ARCMHA and ARCFEN are two independent ARC modules, meaning that the projection matrices of the two modules are not shared between them. During the fine-tuning phase, we exclusively update the learnable parameters of our newly added ARC modules. This involves freezing all original parameters of the pre-trained model and solely focusing on updating the parameters of our ARC.
Inference. Our ARC employs a completely linear transformation so that we can re re-parameterize it by fusing the module to the original framework of the pre-trained model. Take the ARC module of FNN as an example, the process can be defined by:
( GELU CUBLU(ARCER) (ARCFFN XX (1) W/ !!
(9)
where ARCFFN (X can be represented by (!)) (W down (O)) W up (!y
according to Eq. 7 with
I E IR DxD being an identity matrix. Furthermore, we can construct W (1y by:
w( (
(W down C(' + !"
(10)
5

---

Layer 1
Layer 5
Layer 9
600
400
400
200
200
200
0
0.2
0.4
0.2
0.3 0.4
(a) The singular value distribution of MHA adapter.
Luyer
Layer
Layer
600
400
400
400
200
200
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.0
0.2 0.4
0.6
(b) The singular value distribution of FFN adapter.
Figure 3: Singular value distribution adaptation matrices without the bottleneck structure. Two adaptation matrices of both MHA and FFN blocks are fine-tuned on the DTD downstream task. The X-axis represents the singular values, while the Y-axis represents the count of singular values within specific ranges. Complete visualization is available in the appendix.
Therefore, replace the matrix w(O by w/ in the inference stage, thereby avoiding extra computational overheads.
3.3 Insights of architecture design
In this work, we propose an approach that involves adopting low-rank design for the adapters and sharing the bottleneck projections across layers. Our motivation for adopting this approach stems from the assumption that the layer-wise adapters can be effectively re-composed by re-scaling small number of base linear projections. To validate this assumption, we conduct an analysis of the singular value distribution of adaptation matrices W full R learned without the bottleneck operation, which are anticipated to be full-rank. In Fig. 3 we observe that the singular values exhibit extreme sparsity and follow a power-law distribution, with the majority of singular values being close to zero. This indicates that the learned adaptation matrices naturally exhibit low-rank characteristics. Furthermore, the pronounced sparsity of the singular values suggests that the adaptation matrices can be effectively reconstructed using limited number of basis vectors. These findings provide strong evidence supporting the rationale behind our adaptation parameter-sharing strategy.
Experiments
In this section, we present the experimental settings, comparison to existing solutions, and ablation studies to unveil the key properties of the proposed method.
4.1 Experimental settings
Datasets. We evaluate the effectiveness of our ARC approach on two sets of visual task adaptation benchmarks, comprising total of 24 datasets. The list of datasets used for evaluation is provided below:
FGVC. We conduct experiments with the default settings in the VPT 6
a collection of five
Fine-Grained Visual Classification (FGVC) datasets, known as FGVC. The FGVC dataset collection includes
(31), NABirds 32 Oxford Flowers 33 Stanford Dogs [34), and Stanford
Cars 355
VTAB-Ik lk. We also evaluate our ARC method on the VTAB- 1k benchmark 36 which consists of 19 diverse visual classification tasks. These tasks are divided into three groups: the Natural group, which contains images captured through standard cameras; the Specialized group, which includes images captured by specialist equipment such as remote sensing and medical imaging; and the Structured group, which comprises synthesized images from simulated environments, such as object counting and 3D depth prediction. Each downstream task in the VTABB- benchmark consists of 1000 training examples. Following VPT (6), we set aside 200 samples from the training set as the validation set to

---

Table Comparison ARC with baselines and state-of-the-ans art efficient adaptation methods on five FGVC datasets. All methods utilize ViT-B/16 pre-trained on ImageNet-21k as the backbone. "SSF#" denotes the performance reported in the original SSF paper [9] which incorporates advanced data augmentations like cutmix [39 mixup [4), and regularization techniques such as label smooth- ing [41] To ensure fair comparison, we reproduced the SSF method using the code provided by 191, while employing the same basic data augmentations as approach, and we denote SSF's reported performance as "SSF"" and ARC's performance augmented with SSF's data augmentation as "ARC*" The bold font shows the best accuracy of all methods and the underline font shows the second best accuracy.
Method
CUB-200-2011
NABirds
Oxford Flowers
Stunford Dugs
Stanford Curs
Mean
Params
K2.7
150.5
H8.5
KS
Lincur poobing
75.9
97.9
86.2
79,3
Adapper
57.1
$4.3
985.5
89.8
05.0
85.7
0.41
Bins
998.8
19.4
0.28
78.88
98.4
94.6
VPT-Deeo 0
$9.5
54.2
99.0
90.2
83.6
89.1
0185
LoRA 2
19.3
85.0
99.2
91.0
83.2
89.5
0144
85.9
08.5
SSP
55.7
99.6
89.6
90.7
0.33
ARC..
58.4
55.0
99.4
90.1
R2.7
89.1
0.22
ARC
99.3
85.7
90.7
0125
select hyperparameters. Subsequently, we train the model on the full training data using the selected hyperparameters.
Pre-trained backbone. To evaluate the adaptation capacity of the proposed ARC method, we apply the ARC strategy to two typical types of Vision Transformers: VIT [[] and Swin Transformers ([T]. For ViT, we conduct experiments using three different backbone variants with varying model sizes: ViT- WTT-Base/Tarre/Fugge All the backbones are pre-trained on the ImageNet-21K [1] dataset.
Baselines and existing methods. In our comparative analysis, we evaluate the performance of the ARC method against two baselines and several state-of-the-ar efficient pre-trained model adaptation methods. The two baselines we consider are: (1) Full Fine-tuning: This baseline involves updating all the parameters of the pre-trained model using the training data of the downstream task. (2) Linear Probing: This baseline focuses on learning linear classification head on the downstream task while keeping the remaining parameters of the pre-trained model frozen. In addition to the baselines, we compare our ARC method with the following state-of-the-art solutions: (1) Adapter 7 This method inserts lightweight adaptation operations, consisting of a down-projection non-linear activation, and an up-projection, into the pre-trained model. (2) Bias 377: The Bias method fine-tunes only the bias terms of the pre- pre-trained models while keeping the remaining parameters frozen. (3) LoRA (24): This approach introduces trainable low-rank adaptation matrices into each layer of the Transformer architecture. (4) VPT [6]: The VPT method incorporates extra Icarnable tokens into the input or all attention layers of the frozen Transformer. (5) SSF 9: This method adds linear transformation parameters, including scaling and shifting, to modulate the pre-trained features. By comparing the performance of our ARC method with these baselines and state-of sate-of-the-art solutions, aim to demonstrate its superiority in terms of efficiency and effectiveness in pre-trained model adaptation.
Implementation details. To ensure fair evaluation of the effectiveness of our proposed ARC method, we have opted for a simple training setup without too many additional bells and whistles. Similar to VPT (6) we have used standard data augmentations during the training phase, which include image normalization using ImageNet means and standard deviation, random resize crop to 224 x 224 with random horizontal flip for FGVC datasets, and resize to 224 x 224 for VTAB-Ik lk. We have used grid search to select hyper-parameterd size, using the validation set of each task, as in VPT 6.. All experiments were conducted using the Py PyTorch (38) framework on an NVIDIA A40 GPU with 48GB of GPU memory. Further details can be found in the appendix.
such as the learning rate, weight decay, and batch
7

---

Table 2: Comparison of ARC with baselines and state-of-the-arr efficient adaptation methods on VTAB-IK 1k benchmark. All methods utilize ViT-B/16 pre-trained on ImageNet-2 2 as the backbone. To ensure fair comparison, we reproduced the SSF method using the code provided by (9, while employing the same basic data augmentations as our approach.
Natural
Specialad
Struckaeed
Calechar
DFD
SYNH
SU
Menn
Canobra
Meann
Oler-Coune
Oler--Di
DMLL
NITH DiG
SNORD Adin
ANOR8-20 33
Moun
Paruno M)
Dataset
Metiel
Thl Bie-ttxing
65.9
87.7
66.3
V1.3
87.4
18.6
75.9
VSL1
54.3
71.4
$1.4
56.1
556
41.7
65.5
51.9
46-7
25.7
29-1
42.6
65.6
85.8
Liner pobling
63.4
85.0
03.2
91.0
86.3
30,6
51.0
68.9
78.5
57.5
68.6
74.0
77.2
34.3
33.6
33.2
55.4
12.5
20,0
9/6
19.3
36.9
52.9
0.04
Adqurr
72.8
87.0
59.2
V1.3
$5.3
99.9
51.4
13.3
01.5
72.9
78.3
61.5
35.6
55.9
56.6
15.2
25.1
62.1
0.54
8
77.7
06.9
02.6
07.5
87.3
74.5
51.3
76.8
78.2
90.0
75.6
77.4
75.7
50.5
58.6
08.5
67.1
66.7
36.1
703
54.1
47.0
61.9
0.11
788
05.8
68.0
75.1
09.6
81.8
96.1
$3.4
81.4
08.5
66.5
72.8
32.9
12.
55.0
09.4
0.60
00.4
12.5
7766
79.0
263
58.2
0:29
SSP
59m
89.8
7155
$6.9
90.7
95.5
52.9
76.7
M6.7
95.2
86.4
71.4
85.9
68.7
610
82.8
10.7
31.3
48.5
77.6
51.1
56.9
70.6
0.34
535
69:1
97.
75:1
99.4
01.3
14:2
52.9
81
67.
95.9
$7.4
75.5
96.6
75,8
67.5
53.
10.5
71.3
$1.9
70.5
52.9
99.0
75.1
0.31
ABC
71.7
90.1
12.7
49.0
91.9
54.4
81.9
95.1
16.7
15.8
88.7
18.7
81.6
18.2
31.8
99,9
00.
0.11
ABC
71.2
90.9
75.9
09.3
01.1
9038
52.0
11.8
87.4
06.5
57.
26.4
$3.0
$1.3
61.1
54.
11.1
61.
5339
319
41.5
61.4
74.5
0.33
4.2 Experimental comparisons
In this section, we provide a comprchensive comparison of our ARC method with baseline methods and other state-of-the-ant solutions. We evaluate the performance in terms of classification accuracy on downstream tasks as well as the parameter size. The results are summarized in Table and Table 2 with all results obtained using the ViT-B/16 backbone. Based on these comparative results, we make the following observations:
(1) The ARC method demonstrates highly competitive classification accuracy on both sets of visual adaptation datasets, while maintaining a low parameter size. As shown in Table [ and Table 2 under fair comparison, ARC achieves the best mean accuracy and outperforms the other methods on the majority of the 24 datasets. This confirms the effectiveness of our proposed pre-trained model adaptation strategy. Furthermore, thanks to the parameter-sharing strategy in ARC, we are able to noticeably reduce the parameter size compared to other rank deccomoosoitionnn based adapters such as Adapter (7) and LoRA (24) VPT-Shallow (6) also exhibits parameter efficiency as it only introduces learnable tokens in the input layer. However, this is achieved at the cost of a significant performance sacrifice, resulting in considerably inferior performance compared VPT-deep (6) and ARC. Another parameter-efffcient method, Bias [42] focuses updating only the bias terms in the pre-trained network, but also leads to a significant compromise in classification performance on downstream tasks. To further decrease the parameter size, we evaluate ARCaus which omits the adapters applied to Feed-Forward Network (FFN) blocks and focuses solely on adapting the Multi-Head Attention (MHA) blocks. This approach achieves nearly a halving of the parameter size while experiencing only a minimal 1% drop in performance.
(2) In comparison to the efficient adaptation solutions presented in the tables, full fine-tuning yields comparable or even worse classification accuracy across various downstream tasks, despite updating significantly larger number of parameters. This observation further emphasizes the importance and potential of lightweight adaptation designs. On the other end of the spectrum, linear probing requires minimal parameters but exhibits noticeable performance degradation.
Experiments on larger-scale VIT backbones. In addition to evaluating the ARC method on ViT-B/16 backbone, we also conducted experiments on larger-scale ViT backbones to assess its performance on more computationally demanding models. Specifically, we tested the ARC method on ViT-Large and ViT-Huge backbones. The results, presented in Table Baland and Table 3b] demonstrate that the ARC method maintains its competitive classification accuracy even with larger-scale backbones. It consistently outperforms the baseline methods and achieves comparable or superior performance compared to other state-of-the- thet-rr adaptation methods. Furthermore, the parameter size of the ARC method remains noticeably smaller than rank-decomposition based adapters like Adapter 7 and LoRA 24,, , as well as VPT-deep (6) showcasing its efficiency in terms of parameter utilization. These findings suggest that the ARC method is not only effective on smaller-scale VIT backbones
8

---

Table 3: Performance comparison on VTAB-IK using Vir-Large and ViT-Huge pre-trained on ImageNet-21k as backbone. "(-)" denotes the number of tasks in the subgroup. Expanded results are presented in the appendix.
(a) ViT-Large
(b) ViT-
Natural (?)
Specialized (4)
Structed (8)
Meen Total
Parans
Notural (7)
Specialieed (4)
Structed
Moan Total
Paroms
74.7
$3.5
Fall
Linar grottng
70.9
69.1
2533
51.5
0.88
Linzar prodeng
61.4
79.0
26.1
0066
Adaii
66.6
73.5
290
52.9
2.58
Adatte
65.1
76.4
34.5
$1.5
5:56
VPT-Shallow X
78.7
79.9
436
02.9
0.15
StalSoolow
74.
81.2
43.0
02.5
0.38
K25
83.9
54.1
00.8
0.49
Dee
73.0
83.3
522
68.2
0.6
ARC
$23
$5.6
573
725
0.38
ABC
79.1
848
53.7
69.6
0.2
Table 4: Performance comparison VTAB- Ik using Swin-Base pre-trained ImageNet-2ll as backbone. "(.)" denotes the number of tasks the subgroup. Expanded results are presented in the appendix.
Natural (7)
Specialized (4)
Strueted (8)
Mean Total
Params.
Full fine fine-tuning
79.1
86.2
59.7
72.4
86.8
probing
MLP-4
6
70.6
80.7
31.2
37.7
4.04
Partial
73.1
81.7
35.0
58.9
12.65
Bias
+
74.2
80.1
42.4
62.1
0.25
VPr-Shallow
VeTp-Deep 6
76.8
84.5
53.4
67.7
0,22
ARC
79.0
86.6
59.9
72.6
0.27
but also scalable larger models, making a promising solution for efficient pre-trained model adaptation across wide range of backbone sizes.
Experiments on hierarchical Vision Transformers. We extended the ARC method to Swin Transformer [11] hierarchical Transformer architecture. To accommodate the varying feature dimensionalities in Swin Transformer, we introduced stage-sharing strategy, enabling parameter sharing within each stage. The results the VTAB-Ik benchmark (Table 4 demonstrate the generalizability of ARC. It achieves competitive transfer learning accuracy and maintains favorable parameter scale, except for the Natural group where ARC performs relatively weaker. These findings highlight ARC's versatility and effectiveness in adapting different transformer architectures, showcasing its potential for practical applications in visual adaptation.
4.3
Ablation studies
In order to gain deeper insights into the ARC method, we performed ablation studies to explore its additional properties. The experiments conducted the VTAB-IK benchmark using the pre-trained ViT-B/16 model. The results these ablation studies are presented in Table
Bottleneck dimensionality. In ARC method, adopt the low-rank design for the adapters but with the added feature of sharing the down-/up-projection matrices across layers and learning low- dimensional re-scaling coefficients re-compose adaptation matrices. In this section, investigate the impact of the bottleneck dimensionality on adaptation performance. The results are presented in Table 5a] We find that dimensionality of 50 achieves the best balance between transfer learning performance and parameter efficiency. Further reduction to 10-dimensional space leads to fewer parameters but at the cost of performance degradation. Conversely, higher-dimensional hidden spaces result in inferior performance. These findings validate the effectiveness of our low-rank design, with 50 linear projections providing sufficient flexibility for composing layer-adaptive adapters.
Adapter positioning. By default, our adapters are positioned before the MHA and FFN modules, allowing the adaptation operations be seamlessly integrated into the pre-trained network during inference without additional inference cost. In this section, we investigate the impact of adapter

---

Table Ablation experiments on VTAB-1k benchmark using ViT-B/16 backbone. The table shows average accuracy ("Ace.") and parameter count ("Params.") ") for all downtream datasets.
(a) Bottleneck dimension.
(b) ARC location.
(c) Para. sharing strategy.
(d) Insert selection.
Location
Acc.
Purams
10
12.4
0.07
Refor: MIlA
Anne-MiAA
96.1
72.3
0188
1.005
nenvinita + aum-tnir
Stralogy
Acc.
Params
1..5%
Laver Ind.
Sequenis
Form
Aec.
713
Parems
00:20
73.4
0.13
Before FRN
71.11
0.68
1933 Inter"
22.9
0.10
soquamitd
200
100
73.1
71.4
0.21
0.36
Belcen PFN
Afler MILA
Afle: FFN
89.0
0.1%
0.13
intru inter
0.13
0.11
sequemaa
0.133
position on pre-trained model adaptation performance using different positioning strategies. The results are presented in Table 5b Interestingly. placing the adapters after the MHA and/or FFN modules leads to performance degradation, despite this strategy being commonly adopted in previous works such 7 29)] Moreover, using only one type of adapter for either MHA or FFN results in inferior performance compared to using both types of adapters. This suggests that employing both types of adapters allows for more comprehensive adaptation of the pre-trained model to the target task without significantly increasing the parameter count.
Sharing v.s non-sharing adaptation parameters. In ARC, we adopt a parameter-sharing strategy to effectively reduce the number of adaptation parameters. This strategy encompasses two aspects: intra-layer sharing and inter-layer sharing. Through symmetric down-projection and up-projection matrices, we achieve intra-layer sharing, while inter-layer sharing involves sharing projection matrices across different layers. In this section, we investigate the impact of adaptation parameter sharing by conducting experiments with non- non-sharing or partial sharing designs. "intra + inter*" denotes sharing the bottleneck structure between MHA and FFN. The results presented in Table 5c demonstrate that using momsyymettric projection matrices or layer-independent adaptation parameters does not result in performance gains but leads to noticeable increase in parameters. This validates the effectiveness of our parameter-sharing design.
Adapter insertion. We examine the performance of inserting the proposed adapters in a subset of layers using sequential parallel insertion strategies. The results in Table [d show that the performance of ARC improves as more layers are inserted. Furthermore, we observe that inserting adapters in the first six layers yields better results compared inserting them in the last six layers. Additionally, we explore a parallel insertion setting inspired by [24). but the impact is not significantly pronounced. Another notable aspect is that our parameter sharing scheme in the ARC method prevents a linear increase in parameter size with the number of layers, ensuring better scalability, particularly for larger-scale models.
5 Limitation
The adaptation parameter sharing scheme in the ARC method is built on the assumption that layers have the same dimensionality. This assumption is crucial it enables the sharing of down-/up- projection matrices involved in the bottleneck operation across layers, leading to parameter efficiency. However, it worth exploring strategies to extend this scheme and accommodate dimensionality variation. This research direction holds promise for addressing scenarios where dimensionality varies and further enhancing the flexibility and applicability of the ARC method.
6 Conclusions
Our paper introduced the Adapter Re- Re-Composing (ARC) method, which leverages the reusability of adaptation parameters to efficiently adapt pre-trained models. By sharing rank adapters across layers and learning layer-specific re-scaling coefficients to re-composing layer- adaptive adapters, ARC balances transfer learning performance and adaptation overheads. Extensive experiments demonstrate the compelling performance of our approach with reduced parameter size. ARC offers promising solution for efficient pre-trained model adaptation, showcasing the potential of reusing adaptation parameters for competitive results.
in low-
10

---

References
[1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2020.
[2] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 000000000000 009.
[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PMLR, 2021. pp.
[4] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer, "Scaling vision transformers." in Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, 2022, pp. 12 104 12 113.
[5] K. Zhou, Yang, C. C. Loy, and Z. Liu, "Conditional prompt learning for vision- language mod- els." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16 16611-16688 825.
[6] M. Jia, L. Tang. B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, "Visual prompt tuning." in Proceedings of the European conference on computer vision (ECCV). Springer, 2022, pp. 709-727.
[7] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, "Parameter-efficient transfer learning for nlp." in International Conference on Machine Learning. PMLR, 2019, pp. 2790-2799.
[8] S. Chen, C. Ge, Z. Tong. J. Wang, Y. Song, J. Wang, and P. Luo, "Adaptformer: Adapting vision transformers for scalable visual recognition," in Advances Neural Information Processing Systems (NeurIPS), 2022.
[9] D. Lian, D. Zhou, J. Feng. and X. Wang. "Scaling & shifting your features: A new baseline for efficient model tuning," in Advances in Neural Information Processing Systems (NeurIPS), 2022.
[10] G. Luo, M. Huang, Y. Zhou, X. Sun, G. Jiang, Z. Wang, and R. Ji, "Towards efficient visual adaption via structural re-parameterization. arXiv preprint arXiv:2302 200.0000008066 2023.
[11) 7. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, 7. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedings of the IEEF/CVF international conference computer vision, 2021, pp. 10012-10022.
[12] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. "A comprehensive survey on transfer learning," " Proceedings of the IEEE, vol. 109, no. 1, pp. 43-76, 2020.
[13] S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp.
2010.
[14] M. Iman, H. R. Arabnia, and K. Rasheed, "A review of deep transfer learning and recent advancements." Technologies, vol. 11, no. 2, 40, 2023.
[15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. leee, 2009, pp.
[16] T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor "Imagenet-2lk pretraining for the masses," in Advances in Neural Information Processing Systems (NeurIPS), 2021.
[17] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, "Revisiting unreasonable effectiveness of data in deep learning era," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 843-852.
11

---

[18] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten, "Exploring the limits of weakly supervised pretraining." in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 181-196.
[19] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev et al., "The kinetics human action video dataset," arXiv preprint arXiv: 100.7705559... 2017.
[20] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang. "Transformer in transformer," Advances Neural Information Processing Systems (NeurIPS), vol. 34, pp. 9088-1999, 2021.
[21] D. Zhou, B. Kang, X. Jin, L.. Yang, X. Lian, Z. Jiang, Q Hou, and J. Feng, "Deepvit: Towards deeper vision transformer," arXiv preprint arXiv:2103. 2021.
[22] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners." " Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877-1901,
[23] X. Chen, S. Xie, and K. He, "An empirical study of training supe-supe vision transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9640-9649.
[24] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint arXiv:2106.0966 2021.
[25] B. Lester, R. Al-Rfou, and N. Constant, "The power of scale for parameter-efficient prompt tuning," in Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, 2021, pp. 3045-3059.
[26] X. L. Li and P. Liang, "Prefi--tuning: Optimizing continuous prompts for generation," arXiv preprint arXiv:2/01.00190. 2021.
[27] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing," ACM Computing Surveys, vol. 55, no. 9, pp. 1-35, 2023.
[28] T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, "Autoprompt: Eliciting knowledge from language models with automatically generated prompts," arXiv preprint arXiv:2010. 15980, 2020.
[29] Y.-L. Sung, J. Cho, and M. Bansal, "VI-adapter: Parameter-efficient transfer learning for vision- and-language tasks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5227-5237.
[30] Y. Zhang, K. Zhou, and Z. Liu, "Neural prompt search," arXiv preprint rrXiv:2206.0467. 2022.
[31] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, "The caltech-uese hirds-200-2011 dataset," 2011.
[32] G. Van Horn, Branson, R. Farrell, S. Haber, Barry, P. Ipeirotis, P. Perona, and S. Belongie, "Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-graines dataset collection." " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 595-604.
[33] M.-E. Nilsback and A. Zisserman. "Automated flower classification over large number of classes," in 2008 Sixth Indian Conference on Compuler Vision, Graphics & Image Processing. IEEE, 2008, pp. 722-729.
[34] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li, "Novel dataset for fine-grained image categorization: Stanford dogs," in Proc. CVPR workshop on fine-grained visual categorization (FGVC), vol. 2, no. 1. Citeseer, 2011.
12

---

[35] T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei, "Fine- teme-grained car detection for visual census estimation." in Proceedings of the AAAI Conference on Artificial Intelligence. vol. 31, no. 1, 2017.
[36] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy et al., "A large-scale study of representation learning with the visual task adaptation benchmark," arXiv preprint 19Xiv:1110008800 2019.
[37] E. B. Zaken, Y. Goldberg, and S. Ravfogel, "Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models." in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 1-9.
[38] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances Neural Information Processing Systems (NeurIPS), vol. 32, pp. 8026 8037, 2019.
[39] S. Yun, D. Han, J. Oh, S. Chun, J. Choe, and Y. Yoo, "Cutmix: Regularization strategy to train strong classifiers with localizable features," " in Proceedings of the IEEE/CVF international conference computer vision, 2019, pp. 6023-6032.
[40] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," arXiv preprint 17 2017.
[41] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna, "Rethinking the inception architec- ture for computer vision," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 28 2818-2826.
[42] H. Cai, C. Gan, L. Zhu, and S. Han, "Tinytl: Reduce memory, not parameters for efficient on-device learning." " Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 11 285-11 11 297, 2020.
13

---

Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing
Supplementary Materials
In the supplementary materials involving our work, we demonstrate detailed dataset settings, sup- plemental insights and analysis, extra experimental details, supplemental experiments, and broader impacts, including:
Detailed descriptions for datasets and implementation
Insights of architecture design
Parameter size analysis
D
Experimental details on larger-scale and hierarchical VIT backbones
Experimental details on ablation studies
Expanded experiments with self-supervised pre-training
Broader impacts
Due to the limitation that the file "Supplementary Materials.zip" larger than 100MB can- not be uploaded on OpenReview, the supplementary materials only upload the code for the project. Please refer to the anonymous link huttps//drive wworriieeggogge...o com/file/a///////ooocccc TZ0DhhrTLLLo00GTTAAAAATCCOSSSSS to obtain the complete code, datasets, and models.
14

---

A Detailed deseriptions for datasets and implementation
We describe the details of visual adaptation classification tasks in Table 6 (FGVC) and (VTAB-IK), including the class number and the train/val/test sets.
Table 6: Dataset statistics for FGVC. <*"" denotes the train/val split of datasets following the dataset setting of VPT models [6].
Dataset
Description
Classes
Train size
Val size
Test size
[1]
Fine-grained bird species recognition
200
5,394*
600*
5,794
NABirds 32
Fine-grained bird species recognition
555
21,536*
2,393*
24,633
Oxford Flowers 3
Fine-grained flower species recognition
102
1,020
1,020
6,149
Stanford Dogs
Fine-grained dog species recognition
120
10,800
1,200*
8,580
Stanford Cars
35
Fine Fine-grained car classificatio
196
7,329
815*
8,041
Table Dataset statistics for VTAB- 1k [36].
Dataset
Description
Classes
Train size
Val size
Test size
CIFAR-100
100
10,000
Caltech101
102
6,084
DTD
47
1,880
Flowers
Natural
102
800/1,000
200
6,149
Pets
37
3,669
SVHN
10
26,032
Sun397
397
21,750
Patch Camelyon
32,768
EuroSAT
10
5,400
Specialized
800/1,000
200
Resise45
45
6,300
Retinopathy
5
42,670
Clevr/count
8
15,000
Clevr/distance
15,000
DMLab
22,735
KITTI/distance
711
Structured
800/1,000
200
dSprites/location
16
73,728
dSprites/orienntation
16
73,728
SmallNORB/zzimuth
18
12,150
SmallNORRBeelevtion
9
12,150
Table summarizes the detailed configurations we used for experiments. As mentioned in Section 4.1 we utilize grid search to select syper-parameter such as learning rate, weight decay, batch size, and adapter dropout, using the validation set of each task. Note that we also apply dropout to the middle features produced by our ARC method, which we term as "adapter dropout". Specifically, during the ARC process, we randomly drop partial features before up-projection.
B Insights of architecture design
Similar to Fig. 3 we present more visualization results of singular value distribution of adaptation matrices Whall E DxD learned without the bottleneck operation. As shown in Fig 4 the singular value distribution of adaptation matrices leamed DTD downstream task exhibits power-law
15

---

Table The implementation details of configurations such as optimizer and syper-parameters. We select the best hyper- parameters for each download task via using grid search.
Optimizer
Adam W (0.2, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0001] [0.05, 0.01, 0.005, 0.001, 0] (256, 128, 32) {0.8, 0.5, 0.1, 0) Cosine Decay 100 10
Learning Rate
Weight Decay
Batch Size
Adapter Dropout
Learning Rate Schedule
Training Epochs
Warmup Epochs
distribution across various layers in the downstream tasks. This finding provides further support for
our research motivation.
19N Lagerr
ITN Lapur2
Itay 3
300
400
400
330
300
403
339
200
200
201
203
290
100
133
000
04
of
NIHA of Layer5
NIIA Loyer
IF%
400
FTN alTapur
493
ITN Lfaae
300
303
300
300
200
200
190
133
0.5
8.2
1.4
60
9.2
9.4
0.2
04
94
80
0.4
0.00
0.4
04
19% Layerr?
Lapork
IEN Laayer
400
300
330
330
200
201
230
100
100
193
100
193
MBA ol Loque
MHHA Layeril
FTS Laper
ETN Layer
Efger 12
500
300
300
400
200
233
230
100
100
130
100
900
3.50
0:00
2:78
120
3.
425
0.00
0.75
122
0.00
8m
0.0
0.5
10
1.5
(a) Singular value distribution of MHA adapter.
(b) Singular value distribution of FFN adapter.
Figure 4: Singular value distribution of adaptation matrices without the bottleneck structure. Two adaptation matrices of both MHA and FFN blocks are fine-tuned on the DTD downstream task. The X-axis represents the singular values, while the Y-axis represents the count of singular values within specific ranges.
C Parameter size analysis
To showcase the parameter-efffciencc of our ARC method, we compare its parameter size with other popular lightweight adaptation methods (Table including Adapter 77. VPT (). LoRA (4), and SSF (9. Adapter [7] adds two linear projections to each encoder layer during fine-tuning, resulting in the introduction of 2 D D' L learnable parameters, where D' denotes the hidden dimensionality of the linear projections. Furthermore, due to the presence of non-lincar activations in Adapter, the additional parameters contribute to supernumerary overhead during the inference phase. VPT 6 incorporates m prompts into input space, leading to an increase of m D parameters for VPT-Shallow and m D- L parameters for VPT-Deep. In contrast to Adapter, both LoRA (24) and SSF [9] employ linear adaptation methods without incorporating non-linear functions. This design choice allows them to leverage re-parameterization benefits, thereby mitigating additional computations during inference. Specifically, the adaptation matrix of LoRA, which consists of down- pron-projection and an up-projection, introduces w D D' learnable parameters, where w denotes the number of attention matrices undergoing adaptation. SSF inserts lincar scaling and shifting coefficients after
16

---

operations, resulting in an addition of o D L extra parameters. The proposed ARC method offers additional parameter compression by sharing symmetric projection matrices across different layers. This approach introduces only D D' parameters. Additionally, we learn low-dimensional re-scaling coefficients and bias terms for each layer, resulting total of (D' + D) additional parameters. Overall, the number of parameters in our default ARC is 2 ((D D') (D' + D) L.)
Table Comparison of the additional parameter size both fine-tuning and inference stages with other lightweight adaptation methods.
Method
Adapter
VPT-Shallow
VPT-Deep
6
LoRA
SSF 9
ARC
Stage
Fine Tuning
D'D-
D
2 D
D'
2 D
2 2. D' (07 D)
Inference
D.
D-
We also compare the parameter size with lightweight adaptation methods on backbones of different scales, as shown Fig. 5 Our ARCs demonstrate parameter efficiency across various model sizes, comparable to VPT-Shallow [6] However, the unique advantage of our approach lies in its ability to effectively balance lower overheads and maintain competitive performance. Furthermore, the parameter count of our ARC remains stable scalability of our method with minimal additional resource consumption.
the model scale increases, showcasing the
6
5
4
ViT-Base
3
VIT-Large
Vit-Huge
0
Adapter
VPT-Shallow
VPT-Deep
LoRA
SSF
ARCatt
ARC
Figure 5: The parameter size comparison lightweight adaptation methods on VIT Backbones of Different Scales. The X-axis represents different adaptation methods, while the Y-axis represents the parameter size in Million (M).
0.7
0.6
0.5
4
0.4
0.3
0.2
10
0.1
Adapter
VPT-Shellow
VPT-Depp
LoRA
ARCat
ARC
Figure 6: The parameter size comparison with lightweight adaptation methods with a different number of inserted layers. The X-axis represents different adaptation methods, while the Y-axis represents the parameter size in Million (M).
Thanks to our adaptation parameter sharing strategy, the ARC method avoids linear increase in the number of learnable parameters as the number of layers grows. We employ ViT-B as the backbone and integrate adapters into different layers. As shown 6 in contrast to other adaptation methods, both our ARCs and VPT-Shallowsi effectively manage parameter growth the number of inserted layers increases, but only our methods achieve promising performance without significant cost escalation. This highlights the scalability and effectiveness advantages of our ARCs.
17

---

D Experimental details on larger-scale and hierarchical VIT backbones
Table T0 and 12 respectively display the comprehensive results of the comparison conducted in Section among Vir-Large, ViT-Huge, and Swin-Base models.
Table 10: This table extended from Table Sa in Section 4.2 and describes the detailed experimental results of the performance comparison VTAB- using VTT- ViT-Largge pre- pre-trained ImageNet-21k as the backbone.
Specialed
Caleshiii
Fl
Canotoa
EUNOAAT
650 LAN
60000-5
Meun Rotel
BTD
SYNH
Mias
Clavs
DNLL
Mea
Mathed
$4.3
41.2
publies
63.6
Ailpeer
N
15.3
$4.3
54.5
97.4
64.3
31.3
52.9
68.6
15.8
85.1
03.4
65.5
73.5
35.4
34.1
90.8
47.1
30.4
23.4
10.5
15.8
70.0
52.9
3.58
81.4
91.3
99.5
51.2
11.3
WPSDeep
10.8
09.8
96.6
83.9
46.1
47.4
54.1
Laita [
15.8
89.8
13.6
09.1
96.8
81.2
67.5
81.4
16.0
91.
834
75.4
39.0
28.1
00.
16.7
81.6
76.7
51.5
78.0
15-4
57.3
7201
0.74
ARCun
72.2
56.
13.4
88.0
46.2
Table 11: This table extended from Table 3b Section 4.2 and describes the detailed experimental
results of the performance comparison VTAB-1K using Vi Huge pre-trained on ImageNet-21k as the backbone.
Calebii
Restasperto
Orent
KETTI DH
ANORDA Arte
ANORI-
Mlaa load
DTD
Prb
SYNU
Maan
Clow
2000
Full fanetasing
56.7
96.5
55.0
96.5
70.7
67.5
32.5
70.9
63.1
$5.5
91.9
13.6
13.6
47.6
53.9
37.8
09.9
53.6
45.6
30.3
25.8
46.3
63.1
630.9
Adquer
34.4
02.7
14.2
35.8
65.1
72.5
70.8
11.1
38.0
27.5
10.0
18.5
31.3
5.78
Bie 17
65.7
94.3
59.9
96-6
90.6
60.1
41.9
70.3
79.7
92.8
71.5
71.6
76.9
52.3
90.4
31.2
57.7
63.9
39.7
16.7
20.2
41.7
60.1
0.52
VIT-Statow
85.1
248
62.8
0.18
99.4
15.2
71.1
12.2
33.1
14.2
48.
65.1
26.4
10.
18.3
69.3
1.21
ABCun
65.5
90.1
00.5
15.0
92.5
10.1
45.6
25.3
53.4
94.5
945
14.4
15.2
06.6
45.6
76.2
18.5
51.2
32.1
37.6
57.6
76,
0.17
ABC
85.1
Table 12: This table is extended from Table 4 in Section 4.2 and describes the detailed experimental results of the performance comparison VTAB-Ik using Swin-Base pre-trained ImageNet-21k as the backbone.
Natural
Specialied
Strectored
Calichii
DTD
SVNIE
Mans
Reshed5
Lleees Cauns
Cles
KITR ANA
JNORB-AA AAd
ANORR-DO
Moue
Pamaus Mi
Methed
grahing
14.8
W1.2
15.9
23.5
12.11
39.1
35.9
6511
27.6
0.09
MI.-4
51.5
67.4
71.4
99.5
99.1
30.7
52.5
70.6
50.5
90.5
76.8
74.4
50.7
608
38.5
40.3
66.5
9.4
31.1
14.
35.8
31.2
57.7
4.54
VPT.Shalow
75.0
91.3
12.3
59.4
90.4
66.4
70.
50.1
43.0
72.7
40.8
43.9
63.2
28.4
21.5
26.3
638
066
VerEDwop
766
90.8
76.0
08.5
91.6
16.5
11.7
763
68.9
96.7
850
72.0
61.1
67A
98.
901
76.1
78.
536
757
36.7
13.4
677
0.33
71.9
71.9
52.4
E Experimental details on ablation studies
Table
B
and 16 display the complete results of the ablation studies in Section
4.3
18

---

Table 13: This table extended from Table 5a Section 4.3 and describes the detailed experimental content of the performance comparison among different bottleneck dimensionality.
Natural
SpeckaReed
Strecturd
CI UM
Canndoon
Clove ConEa
Moan Braal
SYNHL
E.00
Datuet
LF1
Dinossien
723
1122
91.1
34.7
1009
94.7
93.6
15.8
15.6
90.1
63.9
46.3
MIL5
13.5
46.3
19A
12,
5007
71.3
730
75.4
39.1
200
705
725
59.8
51.
61.
84.9
05.
75.7
830
67.8
15.9
76.8
50.5
51.3
3911
56.
71.4
Table 14: This table is extended from Table 5b Section 4.3 and describes the detailed experimental content of the performance comparison among different adapter positioning.
JN0 Kr
CHAR-JW
KITT- 861
Moan Total
Chev-
DTD
SVNH
45p
Laratien
Befnc KIRA
20.1
98.5
70.5
95.2
00.5
15.5
53.6
10.0
$4.5
05.5
16.5
73.5
85.
79,9
05.6
18.5
91.
15.1
48.7
20.1
20.6
58,4
12.2
0.08
Befoee FRN
20.8
369
858
Altes
66.
86.2
08.6
95.6
00.2
$2.5
78.4
$3.6
55.3
75.5
17.9
44.3
76.7
57.9
47.0
22.6
52.
(9.0
Reine MITA 0.7.A.1111 PRM
72.2
00.1
72.7
91.0
91.8
54.
$1.6
$1.8
95.7
86.7
75.
88.8
88.7
67.1
46.7
81.6
79.3
51.8
M.4
389
00.
7N4
0.13
Afe MILA FUN
705
189
91.4
66.9
86.
76.11
919
66.
78.4
714
Table 15: This table is extended from Table 5e in Section
4.3 and describes the detailed experimental
content of the performance comparison among different parameter sharing strategy.
Natural
Specialied
Stractoed
Caltehhln
Showens
Cler--iie
45ge On
Mexi Toma
Datnoct
CIPAR
D7D
PMA
Monn
DM
non intr
91.1
215
904
91.9
11.6
$1.3
94.8
$5.4
85.5
81.1
78.8
35.8
08.5
7544
0.98
cern e imur"
72.9
19
72.1
96.
91.
W1.7
35.6
$1.4
15.3
95.5
15.7
75.6
85.8
101
66.5
796
7701
N6.
704
960
99.
12.8
0.00
58.
918
364
91A
$439
98.7
86.7
85.5
M017
66.7
91A
31.4
800
134
Table 16: This table is extended from Table 5d in Section 4.3 and describes the detailed experimental
content of the performance comparison among different adapter insertion.
ANOWDD Mat
13P4X Jie
KITT1001
Dataet
D13
Musi
Oler
DML
Monn
separetd
80.
51.
52.2
343
$43
21.0
336
12 susesial
72.2
90.1
72.7
99.0
91.0
91.
544
51.6
849
95.7
967
75.
$5.
96.7
67.1
48.
81.
792
31.0
314
608
73.4
0L33
1- 12 penta
702
982
315
013
91.1
16.
33.
33.5
23.1
13.6
75.4
$15
76.6
64.1
45.8
163
8211
40.11
25.3
37.2
563
71.4
14333
F Expanded experiments with self-supervised pre-training
also conduct experiments with self-supervised pre-training approaches: MAE (2) and Moco V3 (23]. Specifically, We utilize MAE 2 and Moco V3 (23) self-supervised pre-trained VIT-B as the backbone and evaluate the performance of our ARC on VTAB-1k The results of MAE and Moco V3 self- supervised models are presented in Table 17 and Table [8] respectively. We observe that our ARC still exhibits competitive performance on two sup-eiueeed ViTs. In addition, our ARC method outperforms other adaptation methods: Adapter 7 and LoRA 24 on the majority of downstream tasks. Surprisingly, the ARC.tt with smaller learnable parameters even surpasses the ARC across different self-super pre-trained models. A possible explanation could be that ARCats contains fewer parameters, which allows it to effectively prevent overfitting.
In addition the models pre-trained with supervised objectives in Section
19

---

Table 17: Performance comparison on VTAB- Ik using MAE self-supervised pre- pre-treined ViT-Bast as backbone.
Natuná
Speckdkzed
Struclurd
Called
Canaltnn
Cheve-Connt
450 4W
SNORD- AAdi
Presnis OM
Dutuset
DTD
SYNI
Menn
Olev
Mathee
Linca
41.5
20.6
19.2
11.3
22.3
15.9
76.5
65.6
16.
53.2
53.1
13.6
32.5
25.0
51.1
13.0
6.5
17.9
23.7
26.2
0.64
21.4
52.6
4067
62.1
8660
24.3
54.6
15.7
00.1
12.6
659
51.
69.1
11.8
31.8
21.5
30.7
41.7
36.1
0.84
Adiprr
76.2
41.3
18.1
300
14.11
35.9
12.7
4013
64.7
00.3
11.8
18.3
412
0094
Vi'SDeep B
82
55.2
58.0
39.3
45.2
19.4
21.9
25.3
72.9
91.8
43.4
13.6
12.0
390
4019
30.5
530
21.0
12.1
11.01
14.9
21.9
380
0088
1x0tA [4
15.4
85.3
65.9
64.2
76.2
853
81.9
31.
67.
0.30
ARCan
ABC
89.3
01.2
83.1
91.6
66.2
90.4
78.
85.1
64.
83.8
83.8
53.5
36.3
19.7
48.
0.11
Table 18: Performance comparison on VTAB- Ik using Moco V3 self self-supervised pre-trained ViT- Vis-Base as backbone.
Natural
Specialiod
Stracturel
Calechii
Canbboo
BundAY
Redinpulky
Clav-ial
45pe-Or
ANORB-BL 8lc
DTD
SYNR
Mesn
Miun
DNLL
48p-
Mode
Fall lic susing
57.6
21.0
64.6
91.5
19.9
89.8
20.1
12.0
S5.1
96.4
63.1
14,3
$4.7
55.1
56.9
46.7
71.9
63.8
40.0
31.5
36.9
52.0
66.2
E5.59
Lisea
55.1
34.
24.7
67.9
$3.1
87-7
80.0
77.
70.8
98.0
19.8
70.2
49.0
47.8
514
Adquer 0
73.8
96.2
09:3
04.1
91.4
69.9
40.9
14.2
92.4
93.4
81.5
14.3
13.7
55.6
.6.1
390
71.8
00.5
60.2
198
37.1
47.7
668
0196
VPT-Station
73.8
568
34.
LoRA [3]
58.5
90.8
05.0
91.5
15.1
87.6
40.6
74.8
56.4
05.3
83.4
153
55
13.0
64.
51.
$1.9
832
47.5
31.4
47.
614
71.
ABC.n
59.3
900
67.7
99.6
892
90.5
87.1
85.4
15.5
150
64.9
525
88.2
53.4
33.0
46.2
632
726
G Broader impacts
Efficient usability. Unlike previous approaches, method incorporates parameter sharing scheme across different layers of the model, resulting in a significant reduction the number of parameters that need to be fine-tuned. This approach allows us maintain competitive performance while achieving parameter efficiency. By maximizing the utilization of large-scale pre-trained models, ARC methods offer enhanced usability and practicality in various applications.
Environmental-friendd consumption. In addition to the reduction in computational overheads, another significant benefit of our method is the positive impact carbon emissions reduction and environmental protection. By optimizing the computational efficiency of the model, we minimize the energy consumption required during the training and deployment of the model. This reduction in energy consumption leads to a decrease in carbon emissions, contributing to environmental sustainability. Our method only delivers improved performance and efficiency but also aligns with the larger goal of mitigating the environmente impact of Al technologies.
Ethical Considerations. Our model focuses utilizing the representation and generalization capacity obtained from large-scale pre-trained datasets and models. However, it is crucial to acknowl- edge that if the pre-training datasets contain bias or illegal information, there is a risk of inheriting such issues into our model.
In order to address this concern, it becomes imperative explore research directions that aim to identify and prevent privacy leakage and correct model bias. This involves developing robust mechanisms detect and mitigate bias in training data, well implementing techniques to safeguard sensitive information.
20


Reviews:
[{"review_id": "WIdvYFbnuQ", "reviewer": "Reviewer_7yx9", "summary": "This paper introduces a novel parameter-efficient fine-tuning method called Adapter Re-Composing (ARC). ARC effectively reuses parameters across different layers, resulting in remarkable improvements in performance across 24 image classification datasets while utilizing fewer learnable parameters.\nThe experimental evaluation conducted on various downstream datasets provides compelling evidence of ARC's superiority. It outperforms existing methods and establishes a new benchmark for parameter-efficient fine-tuning techniques.\n", "questions": "Please see Weaknesses.", "limitations": "Please see Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed ARC method is simple yet effective, achieving superior performance on multiple downstream datasets while utilizing fewer trainable parameters.\n\n2. The experiments conducted provide compelling evidence, as they encompass various datasets, attention-based architectures, and ablations, ensuring the robustness and reliability of the findings.\n", "weaknesses": "1. In Table 3, it is observed that the performance of ViT-Huge is lower than ViT-Large. It would be beneficial if the authors could provide an explanation for this disparity.\n\n2. The utilization of symmetric matrices ($W_{up} = W_{down}^T$) in the bottleneck design helps reduce the number of learnable parameters. However, it would be interesting to explore whether further improvements in performance can be achieved by making the downsampling and upsampling matrices independent. It would be valuable if the authors could provide a comparison of performance and parameter statistics to address this potential enhancement.\n\n3. The experiments conducted have demonstrated the effectiveness of the proposed method. However, it would greatly enhance the strength of this paper if the authors could supplement these empirical results with theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognition of our work strength. We extend our sincere appreciation for the invaluable guidance you have provided. We respond to your concerns as follows.\n\n**1. Regarding the performance of ViT-Huge is lower than ViT-Large.**\n**Re:** We wish to highlight that upon a thorough comparison between Table 3(a) and Table 3(b) in our paper, it becomes evident that a number of methods, including Full Fine-tuning, Adapter, VPT-Deep, and LoRA, exhibit inferior performance on ViT-Huge compared to ViT-Large. It is noteworthy that these methods possess trainable parameters that increase in correspondence with the number of layers, potentially contributing to overfitting stemming from model expansion. This phenomenon aligns with the observations delineated in the Visual Prompt Tuning (VPT) paper and harmonizes with the outcomes depicted in Fig. 4 of the VPT paper. Conversely, methods such as VPT-Shallow, which feature learnable parameters independent of layer count, or methods with a modest count of learnable parameters (e.g., Bias), tend to display less pronounced manifestations of this phenomenon.\n\n**2. Regarding making the downsampling and upsampling matrices independent.**\n**Re:** To offer clarity on the matter of making the symmetric matrices in the bottleneck structure independent of each other, we direct attention to Table 5(c) within our paper. In the last row of the aforementioned table, we elucidate the outcomes stemming from this adjustment, which regrettably fail to yield discernible enhancements in performance, despite introducing an elevation in parameter count. It is important to underscore that our rationale for adopting a symmetric structure in the design of adapters is expounded upon in Joint Response 1.\n\n**3. About theoretical analysis about the method.**\n**Re:** The key innovation underpinning our paper resides in our departure from learning low-rank adapters in isolation for each layer, and instead, reconstituting these adapters by recalibrating shared down- and up-projection matrices. To fortify the rationale for adaptation matrix sharing we present a visual depiction of the interlayer correlation among adaptation matrices in Fig. 1 of the rebuttal_pdf. Precisely, we apply the Singular Value Decomposition (SVD) technique to down- and up-projection matrices within adapters of every layer, gauging the alignment of right singular vectors in down-projection matrices and left singular vectors in up-projection matrices across layers. From the discernible patterns in Fig. 1(a) and Fig. 1(b) of the rebuttal_pdf, a conspicuous correlation is evident among the low-rank adapters spanning layers, substantiating the principle of shared projection matrices within our method. Furthermore, we expound the rationale for employing transpose down-projection matrices as up-projection matrices, as demonstrated in Fig. 1(c) of the rebuttal_pdf, where the strong correlation between the right and left singular vectors within each layer underscores the justification for this approach."}, {"review_id": "nCvsfVs2D6", "reviewer": "Reviewer_zRDU", "summary": "This paper proposes to further reduce the parameters of the adapter by introducing a weight-sharing scheme between different layers. To accommodate the variations across different layers, re-scaling coefficients are learned to re-compose the layer-adaptive adaptation matrices. Experiments are conducted on 24 downstream image classification tasks using various Vision Transformer variants.", "questions": "Please see the weaknesses", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The observation that “learned adaptation matrices naturally exhibit low-rank characteristics” as shown in Fig. 3 is quite interesting, making the motivation for the weight-sharing design compelling. \n+ This paper is well-written and easy to follow. The figures are well-prepared and illustrate the core idea clearly. \n+ The experiments are extensive. \n", "weaknesses": "- The comparisons to SSF shown in Tables 1 and 2 are re-implemented by the authors and data augmentations are removed. Their original performances of SSF are much higher than the proposed approach. It would be better if the authors reported the performance of the proposed approach with these advanced data augmentations.\n- Fig. 3 shows the singular value distribution of the original MHA and FFN adapter. It would be better to show how the proposed approach alleviates such a problem by visualizing the value distribution after using the proposed approach.\n- Though the observation of “learned adaptation matrices naturally exhibit low-rank characteristics” and the proposed weight-sharing scheme is very interesting, the motivation to further reduce the parameters of the adapter is not very compelling as the parameters of the adapter are already relatively small. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your diligent review of our paper and your acknowledgment of the strengths within our work. We have taken your feedback seriously and have formulated a detailed response to the specific issues you raised, which we present as follows:\n\n**1. Regarding incorporating advanced data augmentations in SSF.**\n**Re:** In response to your insightful suggestion, we have rigorously re-trained our ARC method while implementing the data augmentations featured in SSF [9]. The results of this endeavor have been presented in Table 1 of the provided rebuttal_pdf, and a comprehensive explanation can be found in Joint Response 2. Notably, our ARC method consistently exhibits improved performance compared to SSF [9] when subjected to the same set of augmentations, reaffirming the efficacy and generalizability of our approach.\n\n**2. Regarding how the proposed approach alleviates a problem shown in Fig. 3.** \n**Re:** We greatly appreciate your attention to this aspect. In Fig. 3, we offer a compelling insight into the remarkable low-rank attributes exhibited by the adaptation matrices. This distinctive observation underscores the feasibility of employing a shared basis for the reconstruction of these matrices. Drawing from this pivotal discovery, we introduce our ARC method, which not only shares down- and up-projection matrices across all layers but also learns low-dimensional re-calibration coefficients to reconstruct the adapters. This approach contributes to enhanced parameter efficiency. To further substantiate the rationale behind our parameter-sharing strategy, we provide a visual representation of the correlation between adaptation matrices across layers in Fig. 1 of the provided rebuttal_pdf. This visual analysis underscores a robust correlation among the adaptation matrices spanning layers, thus solidifying the justification for our parameter-sharing paradigm.\n\n**3. Regarding the motivation to further reduce the parameters of the adapter is not very compelling as the parameters of the adapter are already relatively small.**\n**Re:** Your insightful comment is greatly appreciated. We have provided a comprehensive response to this query in Joint Response 3. Thank you for raising this pertinent issue.\n\n"}, {"review_id": "DFpIJwlZD7", "reviewer": "Reviewer_4jX4", "summary": "The paper explores ARC, which is a novel parameter-efficient fine-tuning method which uses a similar architecture as adapters but introduces inter- and intra- layer weight sharing. Some down- and up- projection weights are shared but every adapter position uses an independent set of per-channel scaling factor on the channel-reduced intermediate features. The proposed method obtains competitive results on various vision transformer adaptation benchmarks in terms of recognition performance and number of parameters. Moreover it can be re-parameterized into adjacent fully-connected layers so no overhead is incurred during inference.", "questions": "* The paper cited [7] in multiple tables for visual adapter results but [7] seems to not include any results on computer vision tasks (actually the paper title is *Parameter-efficient transfer learning for **NLP***). Are those results reproduced by the authors using the same architecture as [7]? If so, could the authors point to a place where more detailed settings (e.g., the bottleneck dimension, the activation function, the training hyper-parameters) of this baseline can be found?\n\n* In Table 3, it is strange to me why the proposed methods obtain lower performance on ViT-Huge than ViT-Large. Is it possible that the proposed method or some baselines are over-fitting in extremely large backbones?\n\n* Following weakness 1, an understanding of why the proposed method works fairly well can also be provided by experiments: For example, are features from adjacent layers similar due to the identity connections? What if the projections are progressively shared in groups of increasing sizes among consecutive layers before reaching the global sharing setting as reported in the paper?", "limitations": "There are no unmentioned limitations in the paper to my mind.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "* The paper is generally well written. The figures are clear and the text is easy to follow.\n\n* The experiments are comprehensive and cover a wide range of visual recognition datasets and the results are competitive.", "weaknesses": "* The effectiveness of the proposed method is not well justified. Section 3.3 does not make much sense to me: Having long-tailed singular values only implies that the weight difference can be approximated well by a low-rank matrix and thus justifies the bottleneck design. Shared projections further requires the adapters to have largely overlapped kernel and image spaces, which, instead of the singular values themselves, are determined by the direction of the singular vectors corresponding to the top singular values. It also doesn't theoretically justify the intra-block sharing design (i.e., using the transpose of down-projection as up-projection).\n\n* In multiple places (e.g., caption of Table 1 and Line 229), the paper's claim of using simple augmentations on baselines for 'fair comparison' is questionable. Due to the vast differences in nature of different methods, it is expected that their optimal training configurations are different, and each method, including this paper's own, should have the right to choose its optimal training configuration as long as it does not violate some principal rules of machine learning (e.g., leak of test data): Conversely, it is also improper to request that the paper's method being run under SSF's data augmentation for 'fair comparison'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive comments. We address the concerns as follows:\n\n**1. Regarding the motivation and justification of shared adaptation matrices in weakness 1**.\n**Re:** We appreciate your feedback. In response, we have incorporated the suggested analysis to provide a robust rationale for the motivation behind our proposed adaptation matrix sharing strategy. For a comprehensive understanding of this matter, we invite you to refer to Joint Response 1, where you will find a detailed elaboration on this issue. \n\n**2. Regarding run the paper's method under SSF's data augmentation for 'fair comparison' in weakness 2**.\n**Re:** In response to your insightful suggestion, we have rigorously re-trained our ARC method while implementing the data augmentations featured in SSF [9]. The results of this endeavor have been presented in Table 1 of the provided rebuttal_pdf, and a comprehensive explanation can be found in Joint Response 2. Notably, our ARC method consistently exhibits improved performance compared to SSF [9] when subjected to the same set of augmentations, reaffirming the efficacy and generalizability of our approach.\n\n**3. Regarding the results of [7] in question 1**.\n**Re:** We wish to clarify that we did not independently replicate the results from the literature [7] in the realm of computer vision. Instead, we directly adopted the experimental outcomes presented in the VPT paper [6]. For precise details pertaining to the settings employed, we kindly direct your attention to the VPT paper [6].\n\n**4. Regarding inferior performance of ViT-Huge comparing to ViT-Large**.\n**Re:** We wish to highlight that upon a thorough comparison between Table 3(a) and Table 3(b) in our paper, it becomes evident that a number of methods, including Full Fine-tuning, Adapter, VPT-Deep, and LoRA, exhibit inferior performance on ViT-Huge compared to ViT-Large. It is noteworthy that these methods possess trainable parameters that increase in correspondence with the number of layers, potentially contributing to overfitting stemming from model expansion. This phenomenon aligns with the observations delineated in the Visual Prompt Tuning (VPT) paper and harmonizes with the outcomes depicted in Fig. 4 of the VPT paper. Conversely, methods such as VPT-Shallow, which feature learnable parameters independent of layer count, or methods with a modest count of learnable parameters (e.g., Bias), tend to display less pronounced manifestations of this phenomenon.\n\n**5. Regarding question 3**.\n**Re:** We appreciate your suggestions. In response, we conducted a thorough investigation into the input features of adjacent layers within the network. Our findings indicate that there are no noteworthy similarities in this regard. It's worth noting that the visualization and analysis elucidating the correlation between adaptation matrices across layers, as presented in Joint Response 1, sufficiently addresses the concerns surrounding the justification of adaptation matrix sharing. "}, {"review_id": "VZZOJOFIFN", "reviewer": "Reviewer_4kbb", "summary": "This paper introduces a parameter-efficient transfer learning method named Adapter Re-Composing (ARC), which mainly focuses on investigating the reusability of adapted parameters. The authors propose to apply a shared adapter to all the layers (blocks) of the pre-trained model, and they use different Re-Scaling Coefficients (diagonal matrices) in different layers to ensure the diversity of parameters in different layers.\nThe motivation behind this design is the adapter module's low-rank property, shown in Fig.3 in the main text.\nMoreover, the authors conduct extensive experiments to demonstrate the effectiveness of their designs, where they train fewer parameters and achieve competitive or even better performance compared to prior arts.", "questions": "See the weakness.", "limitations": "The authors disscuss the limitations and boarder impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: This paper is well-written and well-organized.\n\n2: The ARC design in the paper is well-motivated.\n\n3: Although the technic is simple and easy to implement, the performance is impressive.\n\n4: The experiments are comprehensive.", "weaknesses": "1: The ARC leverages learnable re-scaling coefficients in different layers to maintain diversity. However, the authors didn't disscuss the numerical difference among the re-scaling coefficients across different layers. If they are also similar, a share-weight adapter can replace the ARC. \n\n2: In Tab. 1 and Tab. 2, the SSF* applies additional techniques (e.g., data augmentation) during training and gains non-trivial improvements. Why did the authors not use those techniques to improve the performance of ARC further? Will these techniques further benefit the ARC?\n\n3: I wonder about the computational overhead of the ARC, e.g., the GPU memory usage during training, because the number of learnable parameters may not necessarily be positively correlated with the GPU memory usage. Could authors provide the additional GPU memory usage comparison between ARC and other related works?\n\n4: How to use ARC in Hierarchical Vision Transformers such as Swin Transformer is unclear in Lines 277-279. Adding more details here may be helpful.\n\nOverall, my major concern is weakness 1. Happy to raise my rating if my concers are well adressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the positive feedback you have shared and address the raised concerns as follows:\n\n**1. Regarding the diversity of re-scaling coefficients.**\n**Re:** We appreciate the thoughtful insights you've provided. To comprehensively address this matter, we conducted an extensive analysis of the re-scaling coefficients across various layers. The outcomes, specific to the DTD dataset within the VTAB-1k dataset, have been visually captured in Fig. 2 of the provided rebuttal_pdf. This analysis reveals substantial dissimilarity among the scaling factors across different layers. This outcome unequivocally underscores the distinctiveness of our proposed ARC structure, making it non-trivial to substitute with a shared adapter. Furthermore, we conducted the experiment you suggested by sharing adaptation matrices while excluding re-scaling parameters. The corresponding results are presented as \"Sharing-Adapter\" in Table 1 of the provided rebuttal_pdf. The considerable decline in performance observed in this scenario serves as compelling evidence attesting to the indispensable role played by the re-scaling coefficients. This exploration reaffirms the crucial significance attributed to the re-scaling mechanism in our approach.\n\n**2. Regarding run the paper's method under SSF's data augmentation**.\n**Re:** In response to your insightful suggestion, we have rigorously re-trained our ARC method while implementing the data augmentations featured in SSF [9]. The results of this endeavor have been presented in Table 1 of the provided rebuttal_pdf, and a comprehensive explanation can be found in Joint Response 2. Notably, our ARC method consistently exhibits improved performance compared to SSF [9] when subjected to the same set of augmentations, reaffirming the efficacy and generalizability of our approach.\n\n**3. Regarding GPU usage**.\n**Re:** The utilization of GPU resources during model training is influenced by multiple factors, including the model size, intermediate variables generated during the forward process, and gradient information in back propagation. Notably, intermediate variables tend to consume a substantial portion of GPU resources, surpassing the model size in significance. In alignment with the LoRA and Adapter designs, our method fine-tunes the low-rank bottleneck structure within each layer. Consequently, the accumulation of additional intermediate variables during training is akin to these methods, resulting in similar GPU memory usage for our ARC approach. In the case of SSF, a concise breakdown of the GPU usage for intermediate variables can be approximated as O(L\\*m\\*N\\*D), where L signifies the number of layers, m denotes the count of SSF modules embedded in each layer, N represents the token count, and D stands for feature dimensions. On the contrary, the GPU usage associated with intermediate variables produced by our ARC adheres to O(L\\*N\\*D), markedly smaller than the SSF approach. To validate this analysis, we conducted experiments, training different models under identical hardware conditions and gauging GPU memory consumption. The outcomes confirm that the GPU utilization of our method closely aligns with the Adapter and LoRA methods, notably outperforming the SSF method in terms of resource efficiency.\n\n**4. Regarding clarification of how to use ARC in Hierarchical Vision Transformers such as Swin Transformer**.\n**Re**: We regret any confusion caused by our previous description. To provide further clarity, it's important to note that the Swin Transformer comprises distinct stages, each characterized by varying feature dimensions. Achieving global sharing across these stages is unfeasible. In Swin Transformer, each stage consists of multiple transformer blocks with uniform feature dimensions. In light of this architecture, we've introduced shared adaptation matrices between blocks within the same stage. \n"}, {"review_id": "5HvrWY4WfD", "reviewer": "Reviewer_jreR", "summary": "This work presents an efficient adapter design for transfer learning of large retrained models. The key idea is to enable adapter to be shared across layers and imposing low-rank constraint, so that the overall trainable parameters can be reduced. Given the re-composed linear adapter, existing reparameterization methods can be further used to avoid extra computation cost. ", "questions": "See the weaknesses above.", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Parameter efficient transfer learning is an important approach for making large pertained model useful in many applications.\n\n- The paper is written to be easy to read.", "weaknesses": "First of all, the motivation of this work is weak -- Existing adapters are already lightweight, e.g., less than 0.5% of the pretrained model, and the scope for further reducing the size of adapter is marginal. That being said, this adapter efficiency problem is not significant. \n\nIn introduction, it is hard to read out the novelty of this method since more related works such LoRA is not discussed and compared at all.\n\nLimited novelty with the proposed method: Low-rank constraint is not new, as already used in LoRA [24] although not in the same way, and parameter sharing across layers is also not novel idea. This work combines the two in a single place, which could be considered as being not significant.\n\nLimited performance gain: \n- In most cases, the proposed method cannot achieve a good margin over previous methods (e.g., SSF). \n\n- Another important metric, FLOPs, would be useful to report.\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your dedicated efforts in reviewing our work. While we value your contribution, we would like to address a few factual errors that have come to our attention. Your understanding and collaboration in rectifying these inaccuracies would be immensely valuable to us.\n\n**1. Regarding this adapter efficiency problem is not significant.**\n**Re:** Our response to this particular concern is available in Joint Response 1. To facilitate access to our reply, we are reproducing it here. We acknowledge the valid point regarding the limitations on reducing the absolute count of adaptation parameters. In light of this, we are revising our tone to convey a more balanced perspective on parameter reduction. It is essential, however, to underscore that our approach demonstrates improved adaptation performance even when operating with a reduced parameter count. It is important to note that our contribution extends beyond this efficiency aspect, as we introduce innovative insights into the domain of low-rank adaptation strategies.\n\n**2. Regarding it is hard to read out the novelty of this method since more related works such LoRA is not discussed and compared at all.**\n**Re:** We respectfully disagree with this assertion. Within our introduction, we thoroughly reviewed and analyzed pertinent parameter-efficient pre-trained adaptation methods, as demonstrated in lines 32 to 41. Notably, references [7,8] employed low-rank adapters within the context of vision applications. Additionally, we accentuate the key distinctions that set our approach apart from these existing solutions, elaborated upon from lines 42 to 48. Moreover, to provide a more comprehensive understanding of these distinctions, we offer a visual summary of typical parameter-efficient pre-trained model adaptation techniques in Fig. 1 of the original paper. This illustration is complemented by an in-depth analysis of existing methodologies. Given these comprehensive efforts, we respectfully contend that the criticism regarding the perceived novelty of our work is unfounded.\n\n**3. Regarding concerns about limited novelty with the proposed method: Low-rank constraint is not new, as already used in LoRA [24] although not in the same way**.\n**Re:** We want to clarify that our assertion has never been to claim the low-rank design of adapters as our contribution. In fact, we explicitly state our approach's novelty from line 43 to line 48 of the Introduction, where we outline, \"We adopt a low-rank design for the adapter using a bottleneck operation but propose a novel approach. Unlike other methods that place the adapter in different layers and directly learn different parameters for each adapter to cater to layer-wise variation, we propose sharing the down/up projections in the low-rank adapter across different layers and simply learning low-dimensional re-scaling coefficients to re-compose the linear projections into layer-adaptive adapters.\" This clarifies that our innovation indeed lies in providing a fresh perspective within the domain of low-rank adaptation strategies. While the low-rank constraint itself might not be new, our distinctive approach and insights set our method apart from existing ones, such as LoRA [24].\n\n**4. Regarding concerns about limited novelty with the proposed method: parameter sharing across layers is also not novel idea**.\n**Re:** We respectfully disagree with the characterization that parameter sharing across layers lacks novelty. While the concept of parameter sharing is indeed a broad one, the crux lies in how this sharing is executed and whether it contributes fresh perspectives to the field. In our case, we delve into adaptation matrix sharing within the context of the low-rank design of adapters, specific to parameter-efficient pre-trained adaptation. As far as we are aware, our work is the inaugural exploration of this approach, imbuing the community with novel insights into the realm of low-rank-based adaptation methods. The uniqueness lies not in the general notion of parameter sharing, but in the inventive application and implications of this concept within our proposed framework.\n\n**5. Regarding limited gains: In most cases, the proposed method cannot achieve a good margin over previous methods (e.g., SSF).**\n**Re:** We appreciate your perspective on this matter. In the original paper, we executed our method with standard data augmentations and additionally re-implemented SSF under the same augmentation conditions. The findings, as detailed in Tables 1 and 2, reveal that our ARC method exhibits an improvement of approximately 3% over SSF – a substantial enhancement. Furthermore, in response to input from other reviewers, we proceeded to retrain our ARC method while integrating the data augmentation strategies outlined in the original SSF paper [9]. The outcomes, meticulously presented in Table 1 of the provided rebuttal_pdf, consistently underscore our method's superior performance compared to SSF across various datasets. These results affirm the efficacy of our approach and its propensity to yield consistent advancements over previous methodologies.\n\n**6. Regarding FLOPs**.\n**Re:** The majority of FLOPs emanate from our ARC modules, which are meticulously designed with a bottleneck structure. Consequently, the additional FLOPs introduced by our ARC approach align closely with those of LoRA and Adapter methods. Moreover, it's noteworthy that our supplementary adapter modules entail a linear mapping, enabling us to re-parameterize the ARC modules within the original pre-trained model framework without incurring additional FLOPs overhead. A detailed discussion on this aspect can be found in lines 170-176 of our paper. "}]

Author rebuttal:
**This concludes the the Joint Response to all reviewers.**

We appreciate the diligent efforts undertaken by both the reviewers and the ACs in thoroughly reviewing our manuscript. It is noteworthy that a consensus has been reached among four out of the five reviewers regarding the novelty and efficacy of our proposed methods. Within this context, we diligently address the shared concerns highlighted by the reviewers as **Joint Response**, and in addition, provide individualized responses to the specific inquiries raised by each reviewer.


**1. Regarding the motivation for adaptation matrix sharing across layers.**
**Re:** The key innovation underpinning our paper resides in our departure from learning low-rank adapters in isolation for each layer, and instead, reconstituting these adapters by recalibrating shared down- and up-projection matrices. This transformative approach is partially substantiated in Fig. 3 of the original paper, where the adaptation matrices exhibit pronouncedly low-rank characteristics, facilitating the use of a common basis to reconstruct the low-rank adapters. To fortify the rationale for adaptation matrix sharing, as suggested by Reviewer 4jX4 we present a visual depiction of the interlayer correlation among adaptation matrices in Fig. 1 of the rebuttal_pdf. Precisely, we apply the Singular Value Decomposition (SVD) technique to down- and up-projection matrices within adapters of every layer, gauging the alignment of right singular vectors in down-projection matrices and left singular vectors in up-projection matrices across layers. From the discernible patterns in Fig. 1 (a) and Fig. 1 (b) of the rebuttal_pdf, a conspicuous correlation is evident among the low-rank adapters spanning layers, substantiating the principle of shared projection matrices within our method. Furthermore, we expound the rationale for employing transpose down-projection matrices as up-projection matrices, as demonstrated in Fig. 1 (c) of the rebuttal_pdf, where the strong correlation between the right and left singular vectors within each layer underscores the justification for this approach.

**2. Applying the data augmentations in SSF [9] to the proposed ARC method.**
**Re:** In our original paper, we employed standard data augmentations for the proposed method. However, recognizing the advanced data augmentations used by SSF [9], we re-implemented SSF and presented results from both the original [9] and our re-implementation. In response to the insightful suggestions of Reviewers 4jX4 and 4kbb, we undertook re-training of the proposed ARC method using the same data augmentations as in SSF [9], and subsequently showcased the outcomes across the 19 datasets of VTAB-1k. Remarkably, our results indicated that with the integration of these advanced data augmentations, the proposed ARC method surpassed SSF [9] on 14 out of the 19 datasets. This noteworthy observation provides further compelling evidence underscoring the robust applicability and efficacy of our proposed method.

**3. Regarding the motivation to further reduce the parameters of the adapter is not very compelling as the parameters of the adapter are already relatively small.**
**Re:** We acknowledge the point that the scope for reducing the absolute count of adaption parameters is constrained. Therefore, we will adopt a more measured tone concerning parameter reduction. However, it's crucial to emphasize that our approach attains enhanced adaptation performance even with a reduced parameter count. Importantly, our contribution extends to providing novel insights into the realm of low-rank adaptation strategies.





Meta-review: This paper presents a new Adapter Re-Composing (ARC) method for parameter-efficient fine-tuning large models. The method is simple yet effective and achieves superior performance on multiple downstream datasets while utilizing fewer trainable parameters. After rebuttal, most of reviewers give positive scores. Reviewer jreR gives quite low score given short reviews. As authors rebut, the raised concerns mostly come from misunderstanding. The AC looks into paper details carefully, and agrees with the contribution this simple yet powerful ARC technique. Thus AC decides to accept it.
Decision: Accept (poster)